Directional Interaction for Mobile Augmented Reality Based on Visible Light Backscatter Communication

LI Jing-yu; CHEN Tuo-chao; LI Ming-zhe; XU Xu-hai; ZHANG Cheng; XU Zi-chen; LIU Xuan-zhe; HUANG Gang; FENG Yun; XU Chen-ren

doi:10.12263/DZXB.20220677

您当前的位置：

首页 >

文章列表页 >

Directional Interaction for Mobile Augmented Reality Based on Visible Light Backscatter Communication

PAPERS | 更新时间：2025-12-11

- Directional Interaction for Mobile Augmented Reality Based on Visible Light Backscatter Communication
- ACTA ELECTRONICA SINICA Vol. 52, Issue 11, Pages: 3643-3656(2024)
- 作者机构：
  
  1.北京大学计算机学院，北京 100871
  2.华盛顿大学计算机科学与技术学院，美国西雅图 98195
  3.卡内基梅隆大学人机交互研究所，美国匹兹堡 15213
  4.康奈尔大学信息科学学院，美国伊萨卡14853
  5.南昌大学数学与计算机学院，江西南昌 330031
  6.北京大学第三医院，北京 100191
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62022005;62061146001)
- DOI：10.12263/DZXB.20220677
  CLC： TP399;TN929
- Received：12 June 2022，
  
  Revised：2022-08-15，
  
  Published：25 November 2024
- 稿件说明：
移动端阅览
李景聿, 陈拓潮, 李明哲, 等. 基于可见光逆反射通信的移动增强现实定向交互[J]. 电子学报, 2024, 52(11): 3643-3656.

LI Jing-yu, CHEN Tuo-chao, LI Ming-zhe, et al. Directional Interaction for Mobile Augmented Reality Based on Visible Light Backscatter Communication[J]. Acta Electronica Sinica, 2024, 52(11): 3643-3656.
李景聿, 陈拓潮, 李明哲, 等. 基于可见光逆反射通信的移动增强现实定向交互[J]. 电子学报, 2024, 52(11): 3643-3656. DOI：10.12263/DZXB.20220677

LI Jing-yu, CHEN Tuo-chao, LI Ming-zhe, et al. Directional Interaction for Mobile Augmented Reality Based on Visible Light Backscatter Communication[J]. Acta Electronica Sinica, 2024, 52(11): 3643-3656. DOI：10.12263/DZXB.20220677

摘要

移动增强现实应用中，用户频繁地与环境中不同的智能物体交互完成任务，其完成效率和用户体验由所采用的定向交互技术决定.然而，从交互手段上来说，现阶段定向交互依赖于Wi-Fi、BLE等无线技术，其信号全向传播的特点使其不能利用用户的空间环境（即位置和方向）来缩短互动时间，带来了不必要的时间成本；从交互界面上来说，现阶段基于视觉的界面存在可靠性低、扩展性低等问题，进一步限制了系统的通用性.本文介绍了 RetroAR——基于可见光逆反射通信的定向交互系统.RetroAR利用了光的定向传播特性来保留用户的空间环境，并依靠逆反射通信实现用户与目标设备之间无连接的快速定向交互.系统实验表明，RetroAR最远可在4 m距离支持100°视角的交互，同时实现厘米级的六自由度（6-Degrees Of Freedom，6-DoF）三维跟踪.用户研究表明，与基于Wi-Fi的解决方案相比，RetroAR将非接触式控制的交互时间减少了2倍，并且具有更好的用户体验.RetroAR借助可见光逆反射通信来利用用户空间环境，保持交互过程中的直观性.用户可以“所指即所控”的方式与多个目标进行互动，实现类似自然交互的快速定向交互.

Abstract

In mobile augmented reality applications

users interact with smart objects in proximity to finish collaboration or interaction tasks

whose efficiency and user experience are determined by the underlying directional interaction technology. However

current directional interaction technologies at this stage are inefficient. For interaction means

they rely on wireless technologies such as Wi-Fi and BLE

which propagates omni-directionally and thus cannot use the user's spatial context (i.e.

location and direction) to shorten the interaction time

bringing unnecessary effort. In terms of interaction interface

current vision-based interfaces suffer from low reliability and low scalability

which further limits the adaptability and efficiency of the system. To address this issue

we developed RetroAR: an optical-sensing solution that leverages visible light backscatter communication to serve for directional interaction with intelligent objects on commodity smartphones. RetroAR exploits the directional propagation property of light to preserve the user's spatial context

which enables fast connection-free directional interaction between the user and the target devices. RetroAR instruments objects with custom retro-reflective markers called ViTags. When users interact with these smart objects

these ViTags are used to communicate with the camera on the mobile reader by backscattering the flashlight beams. We first conducted a system evaluation which showed that RetroAR could work reliably at a distance up to 4 meters and a view angle up to 100 degrees

and is able to achieve 6-DoF 3D tracking with an error as low as 1 cm on translation and 4.7 degrees on rotation. To evaluate how our system performs in terms of interaction

we then conducted a user study with 12 participants

which demonstrated that RetroAR improves the interaction time of MAR contactless control by at least two times compared to Wi-Fi-based solutions. RetroAR utilizes the user's spatial environment with visible light backscatter communication to maintain the intuitiveness of the interaction process. Users can interact with multiple targets in a point-and-control manner

which reduces interaction costs and brings a natural and intuitive interaction experience.

关键词

Keywords

references

林倞 , 杨珂 , 王涌天 , 等 . 移动增强现实系统的关键技术研究 [J ] . 中国图象图形学报 , 2009 , 14 ( 3 ): 560 - 564 .

LIN L , YANG K , WANG Y T , et al . Key issues study for mobile augmented reality system [J ] . Journal of Image and Graphics , 2009 , 14 ( 3 ): 560 - 564 . (in Chinese)

陈靖 , 王涌天 , 郭俊伟 , 等 . 增强现实技术在智能手机上的应用 [J ] . 电子科技大学学报 , 2010 , 39 ( S1 ): 80 - 84 .

CHEN J , WANG Y T , GUO J W , et al . Application of augmented reality technology in smart phone [J ] . Journal of University of Electronic Science and Technology of China , 2010 , 39 ( S1 ): 80 - 84 . (in Chinese)

王同聚 . 虚拟和增强现实(VR/AR)技术在教学中的应用与前景展望 [J ] . 数字教育 , 2017 , 3 ( 1 ): 1 - 10 .

WANG T J . Application and prospect forecast of VR/AR technology in education [J ] . Digital Education , 2017 , 3 ( 1 ): 1 - 10 . (in Chinese)

汤晓颖 , 区雪兰 , 谢丹妮 . 基于AR技术的广府家具"纸上博物馆" 设计应用与研究 [J ] . 包装工程 , 2018 , 39 ( 4 ): 115 - 122 .

TANG X Y , OU X L , XIE D N . Design application and research of "museum on paper" of Cantonese furniture based on AR technology [J ] . Packaging Engineering , 2018 , 39 ( 4 ): 115 - 122 . (in Chinese)

PERLI S D , AHMED N , KATABI D . PixNet: Interference-free wireless links using LCD-camera pairs [C ] // Proceedings of the Sixteenth Annual International Conference on Mobile Computing and Networking . New York : ACM , 2010 : 137 - 148 .

DE FREITAS A A , NEBELING M , CHEN X ' , et al . Snap-to-it: A user-inspired platform for opportunistic device interactions [C ] // Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems . New York : ACM , 2016 : 5909 - 5920 .

DE FREITAS A A , NEBELING M , RANITHANGAM A S K K , et al . Bluewave: Enabling opportunistic context sharing via bluetooth device names [C ] // Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems . New York : ACM , 2016 : 38 - 49 .

LI J T , LIU A L , SHEN G B , et al . Retro-VLC: Enabling battery-free duplex visible light communication for mobile and IoT applications [C ] // Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications . New York : ACM , 2015 : 21 - 26 .

XU X Y , SHEN Y , YANG J R , et al . PassiveVLC: Enabling practical visible light backscatter communication for battery-free IoT applications [C ] // Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking . New York : ACM , 2017 : 180 - 192 .

HEUN V , KASAHARA S , MAES P . Smarter objects: Using AR technology to program physical objects and their interactions [C ] // CHI '13 Extended Abstracts on Human Factors in Computing Systems . New York : ACM , 2013 : 961 - 966 .

TANG L Z W , ANG K S , AMIRUL M , et al . Augmented reality control home (ARCH) for disabled and elderlies [C ] // 2015 IEEE Tenth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP) . Piscataway : IEEE , 2015 : 1 - 2 .

JO D , KIM G J . ARIoT: Scalable augmented reality framework for interacting with Internet of Things appliances everywhere [J ] . IEEE Transactions on Consumer Electronics , 2016 , 62 ( 3 ): 334 - 340 .

HUO K , CAO Y Z , YOON S H , et al . Scenariot: Spatially mapping smart things within augmented reality scenes [C ] // Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems . New York : ACM , 2018 : 1 - 13 .

韩玉仁 , 李铁军 , 杨冬 . 增强现实中三维跟踪注册技术概述 [J ] . 计算机工程与应用 , 2019 , 55 ( 21 ): 26 - 35 .

HAN Y R , LI T J , YANG D . Overview of 3D tracking registration technology in augmented reality [J ] . Computer Engineering and Applications , 2019 , 55 ( 21 ): 26 - 35 . (in Chinese)

REKIMOTO J . Matrix: A realtime object identification and registration method for augmented reality [C ] // Proceedings . 3rd Asia Pacific Computer Human Interaction . Piscataway : IEEE , 1998 : 63 - 68 .

CHOY C B , STARK M , CORBETT-DAVIES S , et al . Enriching object detection with 2D-3D registration and continuous viewpoint estimation [C ] // 2015 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2015 : 2512 - 2520 .

GAO Q H , WAN T R , TANG W , et al . Object registration in semi-cluttered and partial-occluded scenes for augmented reality [J ] . Multimedia Tools and Applications , 2019 , 78 ( 11 ): 15079 - 15099 .

王笛 , 胡辽林 . 基于双目视觉的改进特征立体匹配方法 [J ] . 电子学报 , 2022 , 50 ( 1 ): 157 - 166 .

WANG D , HU L L . Improved feature stereo matching method based on binocular vision [J ] . Acta Electronica Sinica , 2022 , 50 ( 1 ): 157 - 166 . (in Chinese)

GAO Q H , WAN T R , TANG W , et al . A stable and accurate marker-less augmented reality registration method [C ] // 2017 International Conference on Cyberworlds . Piscataway : IEEE , 2017 : 41 - 47 .

MICROSOFT . Microsoft hololens | mixed reality technology for business [EB/OL ] . ( 2020 )[2022 ] . https://www.microsoft.com/en-us/hololens/ https://www.microsoft.com/en-us/hololens/ .

GOOGLE . ARCore - google developers [EB/OL ] . ( 2020 )[2022 ] . https://developers.google.com/ar https://developers.google.com/ar .

GARRIDO-JURADO S , MUÑOZ-SALINAS R , MADRID-CUEVAS F J , et al . Automatic generation and detection of highly reliable fiducial markers under occlusion [J ] . Pattern Recognition , 2014 , 47 ( 6 ): 2280 - 2292 .

KAWSAR F , RUKZIO E , KORTUEM G . An explorative comparison of magic lens and personal projection for interacting with smart objects [C ] // Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services . New York : ACM , 2010 : 157 - 160 .

JUNG J D A , VOGEL D . Methods for intentional encoding of high capacity human-designable visual markers [C ] // Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems . New York : ACM , 2018 : 1 - 10 .

COSTANZA E , HUANG J . Designable visual markers [C ] // Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . New York : ACM , 2009 : 1879 - 1888 .

AHUJA K , PAREDDY S , XIAO R , et al . LightAnchors: Appropriating point lights for spatially-anchored augmented reality interfaces [C ] // Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology . New York : ACM , 2019 : 189 - 196 .

YANG J , LANDAY J A . InfoLED: Augmenting LED indicator lights for device positioning and communication [C ] // Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology . New York : ACM , 2019 : 175 - 187 .

SUN Y B , ARMENGOL-URPI A , REDDY KANTAREDDY S N , et al . Magic hand: Interact with IOT devices in augmented reality environment [C ] // 2019 IEEE Conference on Virtual Reality and 3D User Interfaces . Piscataway : IEEE , 2019 : 1738 - 1743 .

卢霆威 , 王泽平 , 刘梦 , 等 . 基于可见光通信技术的全双工以太网通信系统设计 [J ] . 电子学报 , 2022 , 50 ( 1 ): 45 - 53 .

LU T W , WANG Z P , LIU M , et al . Full-duplex indoor ethernet communication system based on visible light communication [J ] . Acta Electronica Sinica , 2022 , 50 ( 1 ): 45 - 53 . (in Chinese)

GUMMADI R , WETHERALL D , GREENSTEIN B , et al . Understanding and mitigating the impact of RF interference on 802.11 networks [J ] . ACM SIGCOMM Computer Communication Review , 2007 , 37 ( 4 ): 385 - 396 .

YANG J J . Apparatus and Method for Suppressing Interference Caused by Coexistence of WiMAX and Wi-Fi [M ] . New York : Google Patents , 2013 .

SHAO S H , KHREISHAH A , ELGALA H . Pixelated VLC-backscattering for self-charging indoor IOT devices [J ] . IEEE Photonics Technology Letters , 2017 , 29 ( 2 ): 177 - 180 .

AMIRI I S , ALI J . Data signal processing via Manchester coding-decoding method using chaotic signals generated by PANDA ring resonator [J ] . Chinese Optics Letters , 2013 , 11 ( 4 ): 41901 - 41904 .

TAO Q , ZHONG C J , LIN H , et al . Symbol detection of ambient backscatter systems with Manchester coding [J ] . IEEE Transactions on Wireless Communications , 2018 , 17 ( 6 ): 4028 - 4038 .

CANNY J . A Computational approach to edge detection [M ] // Readings in computer Vision . Amsterdam : Elsevier , 1987 : 184 - 203 .

SUZUKI S , BE K . Topological structural analysis of digitized binary images by border following [J ] . Computer Vision, Graphics, and Image Processing , 1985 , 30 ( 1 ): 32 - 46 .

DUDA R O , HART P E . Use of the Hough transformation to detect lines and curves in pictures [J ] . Communications of the ACM , 1972 , 15 ( 1 ): 11 - 15 .

GRAHAM R L . An efficient algorith for determining the convex hull of a finite planar set [J ] . Information Processing Letters , 1972 , 1 ( 4 ): 132 - 133 .

LEPETIT V , MORENO-NOGUER F , FUA P . EPnP: An accurate O(n) solution to the PnP problem [J ] . International Journal of Computer Vision , 2009 , 81 ( 2 ): 155 - 166 .

PERSSON M , NORDBERG K . Lambda twist: An accurate fast robust perspective three point (P3P) solver [C ] // Proceedings of the European Conference on Computer Vision . New York : ACM , 2018 : 318 - 332 .

WILLSON R G , SHAFER S A . Perspective projection camera model for zoom lenses [EB/OL ] . ( 2022 )[2022 ] . https://www.spiedigitallibrary.org/conference-proceedings -of-spie/2252/0000/Perspective-projection-camera-model-for-zoom-lenses/10.1117/12.169831.full https://www.spiedigitallibrary.org/conference-proceedings-of-spie/2252/0000/Perspective-projection-camera-model-for-zoom-lenses/10.1117/12.169831.full .

YANG X F , HUANG Y M , GAO F . A simple camera calibration method based on sub-pixel corner extraction of the chessboard image [C ] // 2010 IEEE International Conference on Intelligent Computing and Intelligent Systems . Piscataway : IEEE , 2010 : 688 - 692 .

LEPETIT V , MORENO-NOGUER F , FUA P . EPnP: An accurate O(n) solution to the PnP problem [EB/OL ] . ( 2022 )[2022 ] . https://link.springer.com/article/10.1007/s11263-008-0152- https://link.springer.com/article/10.1007/s11263-008-0152- .

OPENCV . Detection of ArUco markers [EB/OL ] . ( 2022 )[2022 ] . https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_ detection.html https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_detection.html .

MAHMOUD M S , MOHAMAD A A H . A study of efficient power consumption wireless communication techniques/modules for Internet of Things (IoT) applications [J ] . Advances in Internet of Things , 2016 , 6 ( 2 ): 19 - 29 .

Views

178

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

No data

Related Author

CHEN Tuo-chao

LI Ming-zhe

XU Xu-hai

ZHANG Cheng

LIU Xuan-zhe

HUANG Gang

FENG Yun

XU Chen-ren

Related Institution

School of Computer Science, Peking University, Beijing

G. Allen School of Computer Science and Engineering, University of Washington

Human-Computer Interaction Institute, Carnegie Mellon University

Department of Information Science, Cornell University, Ithaca

Peking University Third Hospital, Beijing

⁰