EnvAct：基于大语言模型与结构化环境表征的物联网自动化执行框架

胡明明; 滕储华; 李明辉; 孙丙坤; 沈立炜

doi:10.12263/DZXB.20250886

您当前的位置：

首页 >

文章列表页 >

EnvAct：基于大语言模型与结构化环境表征的物联网自动化执行框架

学术论文 | 更新时间：2026-10-16

- EnvAct：基于大语言模型与结构化环境表征的物联网自动化执行框架
- EnvAct: An IoT Automation Execution Framework Based on Large Language Models and Structured Environment Representation
- 电子学报 2026年54卷第6期页码：2657-2675
- 作者机构：
  
  1.复旦大学计算与智能创新学院，上海 200433
  2.山东省复杂网络软件自动构造技术重点实验室，山东济南 250101
- 作者简介：
  
  胡明明男，1997年3月出生于安徽省明光市。现为复旦大学计算与智能创新学院博士研究生。主要研究方向为人机物融合智能。E-mail: mmhu25@m.fudan.edu.cn
  滕储华男，2003年3月出生于江西省丰城市。现为复旦大学计算与智能创新学院硕士研究生。主要研究方向为人机物融合智能。E-mail: chteng25@m.fudan.edu.cn
  李明辉男，2002年11月出生于山东省临沂市。现为复旦大学计算与智能创新学院硕士研究生。主要研究方向为人机物融合智能。E-mail: limh2002@163.com
  孙丙坤男，1995年12月出生于陕西省西安市。现为复旦大学计算与智能创新学院博士研究生。主要研究方向为agent系统与大语言模型后训练算法。E-mail: sunbingkun@m.fudan.edu.cn
  沈立炜男，1982年7月出生于上海市。现为复旦大学计算与智能创新学院副教授。主要研究方向为人机物融合系统软件、泛在计算与泛在操作系统等。E-mail: shenliwei@fudan.edu.cn
- 基金信息：
  
  山东省重点研发计划项目(2025CXPT073)
- DOI：10.12263/DZXB.20250886
  中图分类号： TP273;TP18
- 收稿：2026-04-27，
  
  录用：2026-06-02，
  
  纸质出版：2026-06-25
- 稿件说明：
移动端阅览
胡明明, 滕储华, 李明辉, 等. EnvAct：基于大语言模型与结构化环境表征的物联网自动化执行框架[J]. 电子学报, 2026, 54(06): 2657-2675.

HU Mingming, TENG Chuhua, LI Minghui, et al. EnvAct: An IoT Automation Execution Framework Based on Large Language Models and Structured Environment Representation[J]. Acta Electronica Sinica, 2026, 54(06): 2657-2675.
胡明明, 滕储华, 李明辉, 等. EnvAct：基于大语言模型与结构化环境表征的物联网自动化执行框架[J]. 电子学报, 2026, 54(06): 2657-2675. DOI：10.12263/DZXB.20250886

HU Mingming, TENG Chuhua, LI Minghui, et al. EnvAct: An IoT Automation Execution Framework Based on Large Language Models and Structured Environment Representation[J]. Acta Electronica Sinica, 2026, 54(06): 2657-2675. DOI：10.12263/DZXB.20250886

摘要

随着物联网设备在智能家居、智慧建筑等场景中广泛部署，智能系统需要在设备状态持续变化、用户需求表达模糊和外部环境动态扰动的情况下完成可靠控制。现有触发-动作规则方法多依赖预先设定的静态逻辑，难以处理设备操作的前置约束、状态依赖和运行时环境变化等问题；而基于大语言模型（Large Language Model， LLM）的物联网代理虽然增强了自然语言理解和任务规划能力，但通常缺乏统一的结构化环境表征，容易产生环境依赖遗漏、设备选择错误或操作顺序不合理等问题。尤其在多设备协同、持续监测和异常反馈场景中，上述问题会进一步影响任务执行的稳定性。为解决上述问题，本文提出EnvAct（Environment-aware Action framework），一种基于LLM与结构化环境表征的物联网自动化执行框架。EnvAct将空间、设备、状态属性、操作约束和状态依赖统一建模，构建由约束表和依赖表组成的环境表征机制。约束表描述设备操作执行前需要满足的前置状态、操作效果及状态生效延迟；依赖表建立环境状态到候选操作的逆向映射，使系统能够在前置条件不满足时自动补全必要操作。为降低冷启动建模成本，本文从Home Assistant社区蓝图脚本中抽取事件-条件-动作规则，并利用LLM结合当前设备、可观测状态和物理常识过滤个性化或不适配规则，生成可持续更新的环境表征。系统由规划器、合规器、执行器、反思器和环境表征层组成。规划器融合用户需求、实时环境、设备接口和历史偏好生成初始操作序列；合规器基于约束表和依赖表进行合法性校验、依赖补全和偏序图构建；执行器按照偏序关系并行调度独立操作，并通过重试、反馈和状态验证实现闭环执行；反思器根据执行轨迹和用户反馈修正环境表征与偏好信息。实验结果表明，在六类物联网任务中，EnvAct在qwen-plus和GPT-4o驱动下的总体成功率分别达到88.3%和90.0%，优于Sasha和SAGE等基准方法；在动态干扰实验中，系统平均执行可靠性达到81.25%；在真实沙盘实验中，系统成功率保持在90%以上，平均操作响应时间低于30 s。结果表明，EnvAct能够提升动态物联网场景下自动化任务执行的可靠性、适应性和工程可用性。

Abstract

With the widespread deployment of internet of things (IoT) devices in smart homes

smart buildings

and other application scenarios

intelligent systems are increasingly expected to provide reliable automation control in environments where device states keep changing

user requirements are often ambiguous

and external disturbances may occur during execution. Existing trigger-action rule methods mainly rely on predefined static logic. Therefore

they have limited ability to represent the preconditions of device operations

state dependencies among devices and environmental attributes

and runtime environmental changes. Large language model (LLM)-based IoT agents improve natural language understanding and task planning

but they usually lack a unified structured environmental representation. As a result

such agents may miss important environmental dependencies

select inappropriate devices

or generate operation sequences with unreasonable execution orders. These limitations become more serious in scenarios involving multi-device collaboration

continuous monitoring

and abnormal execution feedback

where they may directly affect the stability of task execution. To address these problems

this paper proposes environment-aware action framework (EnvAct)

an IoT automation execution framework based on LLMs and structured environmental representation. EnvAct uniformly models spaces

devices

state attributes

operational constraints

and state dependencies

and constructs an environmental representation mechanism composed of a constraint table and a dependency table. The constraint table describes the preconditions that must be satisfied before a device operation is executed

the expected effects of the operation

and the latency for the corresponding state effects to take place. The dependency table establishes an inverse mapping from environmental states to candidate operations

so that the system can automatically supplement necessary operations when some preconditions are not satisfied. To reduce the modeling cost in the cold-start stage

this paper extracts event-condition-action rules from Home Assistant community blueprint scripts. Based on the current devices

observable states

and physical commonsense

LLMs are used to filter personalized or environment-incompatible rules and generate an environmental representation that can be continuously updated. The EnvAct system consists of a planner

a compliance checker

an executor

a reflector

and an environmental representation layer. The planner integrates user requirements

real-time environmental information

device interfaces

and historical preferences to generate an initial operation sequence. The compliance checker performs legality verification

dependency completion

and partial-order graph construction according to the constraint table and the dependency table

thereby transforming the initial sequence into an executable plan that satisfies environmental constraints. The executor schedules independent operations in parallel according to partial-order relations

and realizes closed-loop execution through retries

execution feedback

and state verification. The reflector revises environmental representation and preference information based on execution traces and user feedback

enabling the system to improve its behavior in subsequent tasks. Experimental results show that

across six types of IoT tasks

EnvAct achieves overall success rates of 88.3% and 90.0% when driven by qwen-plus and GPT-4o

respectively

outperforming baseline methods such as Sasha and SAGE. In dynamic disturbance experiments

the average execution reliability of the system reaches 81.25%. In real sandbox experiments

the system maintains a success rate above 90%

with an average operation response time below 30 s. These results demonstrate that EnvAct can improve the reliability

adaptability

and engineering usability of automation task execution in dynamic IoT scenarios.

关键词

Keywords

references

Chataut R , Phoummalayvane A , Akl R . Unleashing the power of IoT: A comprehensive review of IoT applications and future prospects in healthcare, agriculture, smart homes, smart cities, and industry 4.0 [J ] . Sensors , 2023 , 23 ( 16 ): 7194 . DOI: 10.3390/s23167194 http://dx.doi.org/10.3390/s23167194

Ur B , McManus E , Pak Yong Ho M , et al . Practical trigger-action programming in the smart home [C ] // Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . New York : ACM , 2014 : 803 - 812 . DOI: 10.1145/2556288.2557420 http://dx.doi.org/10.1145/2556288.2557420

Wang Qi , Datta P , Yang Wei , et al . Charting the attack surface of trigger-action IoT platforms [C ] // Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security . New York : ACM , 2019 : 1439 - 1453 . DOI: 10.1145/3319535 http://dx.doi.org/10.1145/3319535

Huang J , Cakmak M . Supporting mental model accuracy in trigger-action programming [C ] // Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing . New York : ACM , 2015 : 215 - 225 . DOI: 10.1145/2750858.2805830 http://dx.doi.org/10.1145/2750858.2805830

Brown T B , Mann B , Ryder N , et al . Language models are few-shot learners [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . New York : ACM , 2020 : 1877 - 1901 . DOI: 10.18653/v1/2021.emnlp-main.734 http://dx.doi.org/10.18653/v1/2021.emnlp-main.734

Cui Hongwei , Du Yuyang , Yang Qun , et al . LLMind: Orchestrating AI and IoT with LLM for complex task execution [J ] . IEEE Communications Magazine , 2025 , 63 ( 4 ): 214 - 220 . DOI: 10.1109/mcom.002.2400106 http://dx.doi.org/10.1109/mcom.002.2400106

Kök İ , Demirci O , Özdemir S . When IoT meet LLMs: Applications and challenges [C ] // 2024 IEEE International Conference on Big Data . Piscataway : IEEE , 2024 : 7075 - 7084 . DOI: 10.1109/bigdata62323.2024.10825187 http://dx.doi.org/10.1109/bigdata62323.2024.10825187

An Tuo , Zhou Yunjiao , Zou Han , et al . IoT-LLM: A framework for enhancing large language model reasoning from real-world sensor data [J ] . Patterns , 2026 , 7 ( 1 ): 101429 . DOI: 10.1016/j.patter.2025.101429 http://dx.doi.org/10.1016/j.patter.2025.101429 .

Rivkin D , Hogan F , Feriani A , et al . SAGE: Smart home agent with grounded execution [PP/OL ] . V2. arXiv ( 2024-01-19 )[ 2026-04-27 ] . https://doi.org/10.48550/arXiv.2311.00772 https://doi.org/10.48550/arXiv.2311.00772 .

Cheng Ye , Xu Minghui , Zhang Yue , et al . AutoIoT: Automated IoT platform using large language models [J ] . IEEE Internet of Things Journal , 2025 , 12 ( 10 ): 13644 - 13656 . DOI: 10.1109/JIOT.2024.3523907 http://dx.doi.org/10.1109/JIOT.2024.3523907 .

Liu Kaiwei , Yang Bufang , Xu Lilin , et al . TaskSense: A translation-like approach for tasking heterogeneous sensor systems with LLMs [C ] // Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems . New York : ACM , 2025 : 213 - 225 . DOI: 10.1145/3715014.3722070 http://dx.doi.org/10.1145/3715014.3722070

Ji Zhenlan , Wu Daoyuan , Ma Pingchuan , et al . Testing and understanding erroneous planning in LLM agents through synthesized user inputs [PP/OL ] . V1. arXiv ( 2024-04-27 )[ 2026-04-27 ] . https://doi.org/10.48550/arXiv.2404.17833 https://doi.org/10.48550/arXiv.2404.17833 .

King E , Yu Haoxiang , Lee S , et al . Sasha: Creative goal-oriented reasoning in smart homes with large language models [J ] . Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 2024 , 8 ( 1 ): 1 - 38 . DOI: 10.1145/3643505 http://dx.doi.org/10.1145/3643505

Soares D , Dias J P , Restivo A , et al . Programming IoT-spaces: A user-survey on home automation rules [C ] // Computational Science - ICCS 2021 . Cham : Springer International Publishing , 2021 : 512 - 525 . DOI: 10.1007/978-3-030-77970-2_39 http://dx.doi.org/10.1007/978-3-030-77970-2_39

Mi Xianghang , Qian Feng , Zhang Ying , et al . An empirical characterization of IFTTT: Ecosystem, usage, and performance [C ] // Proceedings of the 2017 Internet Measurement Conference . New York : ACM , 2017 : 398 - 404 . DOI: 10.1145/3131365.3131369 http://dx.doi.org/10.1145/3131365.3131369

Aavild A P , Rosenkrantz de Lasson A , Moesgaard Andersen C , et al . Distributed home automation with home assistant [C ] // Proceedings of the 14th International Conference on the Internet of Things . New York : ACM , 2025 : 206 - 212 . DOI: 10.1145/3703790.3703828 http://dx.doi.org/10.1145/3703790.3703828

Yu Haoxiang , Hua Jie , Julien C . Analysis of IFTTT recipes to study how humans use Internet-of-things (IoT) devices [C ] // Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems . New York : ACM , 2021 : 537 - 541 . DOI: 10.1145/3485730.3494115 http://dx.doi.org/10.1145/3485730.3494115

Cabitza F , Fogli D , Lanzilotti R , et al . End-user development in ambient intelligence: A user study [C ] // Proceedings of the 11th Biannual Conference of the Italian SIGCHI Chapter . New York : ACM , 2015 : 146 - 153 . DOI: 10.1145/2808435.2808446 http://dx.doi.org/10.1145/2808435.2808446

Hsu K H , Chiang Y H , Hsiao H C . SafeChain: Securing trigger-action programming from attack chains [J ] . IEEE Transactions on Information Forensics and Security , 2019 , 14 ( 10 ): 2607 - 2622 .

Minder B , Wolf P , Baldauf M , et al . Voice assistants in private households: A conceptual framework for future research in an interdisciplinary field [J ] . Humanities and Social Sciences Communications , 2023 , 10 : 173 . DOI: 10.1057/s41599-023-01615-z http://dx.doi.org/10.1057/s41599-023-01615-z

Tamanna I , Lima S . An IoT based smart home automation using Google assistant [J ] . International Journal of Research and Innovation in Applied Science , 2023 , VIII: 112- 119 . DOI: 10.51584/ijrias.2023.8913 http://dx.doi.org/10.51584/ijrias.2023.8913

Bogdan R , Tatu A , Crisan-Vida M M , et al . A practical experience on the Amazon alexa integration in smart offices [J ] . Sensors , 2021 , 21 ( 3 ): 734 . DOI: 10.3390/s21030734 http://dx.doi.org/10.3390/s21030734

Celebre A M D , Dubouzet A Z D , Medina I B A , et al . Home automation using raspberry Pi through Siri enabled mobile devices [C ] // 2015 International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management . Piscataway : IEEE , 2015 : 1 - 6 . DOI: 10.1109/hnicem.2015.7393270 http://dx.doi.org/10.1109/hnicem.2015.7393270

Venkatraman S , Overmars A , Thong M . Smart home automation: Use cases of a secure and integrated voice-control system [J ] . Systems , 2021 , 9 ( 4 ): 77 . DOI: 10.3390/systems9040077 http://dx.doi.org/10.3390/systems9040077

Hwang A , Oza N , Callison-Burch C , et al . Rewriting the script: Adapting text instructions for voice interaction [C ] // Proceedings of the 2023 ACM Designing Interactive Systems Conference . New York : ACM , 2023 : 2233 - 2248 . DOI: 10.1145/3563657.3596059 http://dx.doi.org/10.1145/3563657.3596059

Khan I U , Afzal S , Lee J W . Human activity recognition via hybrid deep learning based model [J ] . Sensors , 2022 , 22 ( 1 ): 323 . DOI: 10.3390/s22010323 http://dx.doi.org/10.3390/s22010323

Gao Yi , Xiao Kaijie , Li Fu , et al . ChatIoT: Zero-code generation of trigger-action based IoT programs [J ] . Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 2024 , 8 ( 3 ): 1 - 29 . DOI: 10.1145/3678585 http://dx.doi.org/10.1145/3678585

Wu Qinyue , Shen Beijun , Chen Yuting . Learning to recommend trigger-action rules for end-user development [C ] // Reuse in Emerging Software Engineering Practices . Cham : Springer International Publishing , 2020 : 190 - 207 . DOI: 10.1007/978-3-030-64694-3_12 http://dx.doi.org/10.1007/978-3-030-64694-3_12

Yusuf I N B , Jiang Lingxiao , Lo D . Accurate generation of trigger-action programs with domain-adapted sequence-to-sequence learning [C ] // Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension . New York : ACM , 2022 : 99 - 110 . DOI: 10.1145/3524610.3527922 http://dx.doi.org/10.1145/3524610.3527922

Felfernig A , Polat-Erdeniz S , Uran C , et al . An overview of recommender systems in the Internet of Things [J ] . Journal of Intelligent Information Systems , 2019 , 52 ( 2 ): 285 - 309 . DOI: 10.1007/s10844-018-0530-7 http://dx.doi.org/10.1007/s10844-018-0530-7

Yao Yuhang , Kamani M M , Cheng Zhongwei , et al . FedRule: Federated rule recommendation system with graph neural networks [C ] // Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation . New York : ACM , 2023 : 197 - 208 . DOI: 10.1145/3576842.3582328 http://dx.doi.org/10.1145/3576842.3582328

Wang Beidou , Guo Xin , Ester M , et al . Device-aware rule recommendation for the Internet of Things [C ] // Proceedings of the 27th ACM International Conference on Information and Knowledge Management . New York : ACM , 2018 : 2037 - 2045 . DOI: 10.1145/3269206.3272009 http://dx.doi.org/10.1145/3269206.3272009

Rivkin D , Hogan F , Feriani A , et al . AIoT smart home via autonomous LLM agents [J ] . IEEE Internet of Things Journal , 2025 , 12 ( 3 ): 2458 - 2472 . DOI: 10.1109/JIOT.2024.3471904 http://dx.doi.org/10.1109/JIOT.2024.3471904 .

Song K , Jayarajan A , Ding Yaoyao , et al . Aegis: Taxonomy and optimizations for overcoming agent-environment failures in LLM agents [PP/OL ] . V1. arXiv ( 2025-08-27 )[ 2026-04-27 ] . https://doi.org/10.48550/arXiv.2508.19504 https://doi.org/10.48550/arXiv.2508.19504 .

Aghzal M , Plaku E , Stein G J , et al . A survey on large language models for automated planning [PP/OL ] . V1. arXiv ( 2025-02-18 )[ 2026-04-27 ] . https://doi.org/10.48550/arXiv.2502.12435 https://doi.org/10.48550/arXiv.2502.12435 .

Wei J , Wang Xuezhi , Schuurmans D , et al . Chain-of-thought prompting elicits reasoning in large language models [C ] // Proceedings of the 36th International Conference on Neural Information Processing Systems . New York : ACM , 2022 : 24824 - 24837 . DOI: 10.52202/068431-1800 http://dx.doi.org/10.52202/068431-1800

Yao Shunyu , Zhao J , Yu Dian , et al . ReAct: Synergizing reasoning and acting in language models [C/OL ] // International Conference on Learning Representations (ICLR) . National Science Foundation , 2023 . https://openreview.net/forum?id=WE_vluYUL-X https://openreview.net/forum?id=WE_vluYUL-X .

Aghzal M , Plaku E , Yao Ziyu . Can large language models be good path planners? A benchmark and investigation on spatial-temporal reasoning[PP/OL ] . V3. arXiv ( 2025-02-24 )[ 2026-04-27 ] . https://doi.org/10.48550/arXiv.2310.03249 https://doi.org/10.48550/arXiv.2310.03249 .

Jiao Kechen , Fang Zhirui , Liu Jiahao , et al . InteractiveCOT: Aligning dynamic chain-of-thought planning for embodied decision-making [PP/OL ] . OpenReview ( 2024-09-24 )[ 2026-04-27 ] . https://openreview.net/forum id=Y4iaDU4yMi https://openreview.net/forumid=Y4iaDU4yMi . DOI: 10.18653/v1/2025.emnlp-main.484 http://dx.doi.org/10.18653/v1/2025.emnlp-main.484

Singhal K , Azizi S , Tu Tao , et al . Large language models encode clinical knowledge [J ] . Nature , 2023 , 620 ( 7972 ): 172 - 180 . DOI: 10.1038/s41586-023-06291-2 http://dx.doi.org/10.1038/s41586-023-06291-2

Wang Xinglin , Feng Shaoxiong , Li Yiwei , et al . Make every penny count: Difficulty-adaptive self-consistency for cost-efficient reasoning [PP/OL ] . V3. arXiv ( 2025-02-12 )[ 2026-04-27 ] . https://doi.org/10.48550/arXiv.2408.13457 https://doi.org/10.48550/arXiv.2408.13457 .

浏览量

133

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

大模型赋能的二维真时延智能超表面天线近场波束聚焦

面向边缘物联网的双教师蒸馏式轻量化入侵检测

面向大语言模型协作感知的海域通信抗干扰

基于信息融合的区块链系统隐匿安全补丁识别及迁移技术

基于全局初检与局部列表精检的OTFS两阶段检测方法