Temporal-Guided Cross-View Feature Fusion Network for Multi-Drone Multi-Object Tracking

WU Han; SUN Hao; JI Ke-feng; KUANG Gang-yao

doi:10.12263/DZXB.20240727

您当前的位置：

首页 >

文章列表页 >

Temporal-Guided Cross-View Feature Fusion Network for Multi-Drone Multi-Object Tracking

Intelligent Vision Algorithms for Unmanned Systems | 更新时间：2025-12-08

- Temporal-Guided Cross-View Feature Fusion Network for Multi-Drone Multi-Object Tracking
- ACTA ELECTRONICA SINICA Vol. 53, Issue 3, Pages: 728-743(2025)
- 作者机构：
  
  国防科技大学电子科学学院电子信息系统复杂电磁环境效应国家重点实验室，湖南长沙 410073
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20240727
  CLC： TP391.41;
- Received：01 August 2024，
  
  Revised：2024-10-30，
  
  Published：25 March 2025
- 稿件说明：
移动端阅览
伍瀚, 孙浩, 计科峰, 等. 时序信息引导跨视角特征融合的多无人机多目标跟踪方法[J]. 电子学报, 2025, 53(03): 728-743.

WU Han, SUN Hao, JI Ke-feng, et al. Temporal-Guided Cross-View Feature Fusion Network for Multi-Drone Multi-Object Tracking[J]. Acta Electronica Sinica, 2025, 53(03): 728-743.
伍瀚, 孙浩, 计科峰, 等. 时序信息引导跨视角特征融合的多无人机多目标跟踪方法[J]. 电子学报, 2025, 53(03): 728-743. DOI：10.12263/DZXB.20240727

WU Han, SUN Hao, JI Ke-feng, et al. Temporal-Guided Cross-View Feature Fusion Network for Multi-Drone Multi-Object Tracking[J]. Acta Electronica Sinica, 2025, 53(03): 728-743. DOI：10.12263/DZXB.20240727

摘要

多无人机多目标跟踪旨在从多架无人机同时捕获的视频中预测所有目标的轨迹和身份标识，以解决单个无人机视频受遮挡和杂乱背景等干扰时跟踪性能衰退的问题.然而，不同无人机捕获的图像视角和尺度差异通常较大，导致对齐和融合不同无人机图像特征困难.针对该问题，本文提出一种通过时序信息引导跨视角特征融合的跟踪算法——TCFNet.该算法首先设计一种目标感知的对齐网络（Object-aware Alignment Network，OAN），利用跟踪过程中的目标轨迹先验估计先前时刻不同视角无人机视频帧间的转换关系.其次，构建一种时序感知的对齐网络（Temporal-aware Alignment Network，TAN），探索前后时刻同一架无人机捕获图像的信息对不同视角图像的转换关系进行精调.最后，基于OAN和TAN估计的不同无人机图像间的转换关系，设计一个跨机特征融合网络（Cross-drone Feature Fusion Network，CFFN）对不同无人机捕获的视觉信息进行融合，解决复杂场景下模型跟踪性能衰退的问题.在MDMT数据集上的实验结果表明，所提出的TCFNet相比其他主流的跟踪方法更具竞争力，在跟踪准确率、识别F1值和多机目标关联分数上超出当前的先进算法2.23、1.67和2.15个百分点.

Abstract

Multi-drone multi-object tracking aims to predict the tracklets and identities of all targets from videos simultaneously captured by multiple drones

which alleviates the tracking performance degradation when individual drone videos suffer from challenges such as occlusion and cluttered backgrounds. However

the differences in viewpoints and scales of images captured by different drones are usually large

resulting in significant difficulties for aligning and fusing cross-drone features. To address this problem

we propose a novels tracker based on cross-view feature fusion guided by temporal information. It first designs an object-aware alignment network (OAN) that utilizes the tracklet prior during tracking to estimate the transformation relationships between cross-drone frames at previous moments. Then

a temporal-aware alignment network (TAN) is constructed to explore the information of single-drone images in the before-and-after moments to fine-tune the transformation relationship across the images. Finally

based on the cross-drone image transformation relationship estimated by OAN and TAN

this paper presents a cross-drone feature fusion network (CFFN) to fuse the visual information captured by multiple drones

which mitigates the tracking performance degradation in complex scenes. Experimental results on the MDMT dataset show that the proposed TCFNet is more competitive than existing mainstream trackers

exceeding current state-of-the-art model by 2.23

1.67

and 2.15 percentage points in terms of tracking accuracy

identification F1 score

and multi-device association score.

关键词

Keywords

references

陈阳 , 皮德常 , 代成龙 , 等 . 多无人机协同陆地设施辅助移动边缘计算的系统能耗最小化方法 [J ] . 电子学报 , 2023 , 51 ( 4 ): 984 - 992 .

CHEN Y , PI D C , DAI C L , et al . Energy minimization for multi-UAVs cooperative ground access points assisted mobile edge computing [J ] . Acta Electronica Sinica , 2023 , 51 ( 4 ): 984 - 992 . (in Chinese)

任双 , 周洁 , 高嵩 , 等 . 基于注意力机制的无人机集群协同分群控制算法 [J ] . 电子学报 , 2023 , 51 ( 7 ): 1898 - 1905 .

REN S , ZHOU J , GAO S , et al . Cooperative fission control algorithm of UAV swarm based on attention mechanism [J ] . Acta Electronica Sinica , 2023 , 51 ( 7 ): 1898 - 1905 . (in Chinese)

LIU Z H , SHANG Y Y , LI T M , et al . Robust multi-drone multi-target tracking to resolve target occlusion: A benchmark [J ] . IEEE Transactions on Multimedia , 2023 , 25 : 1462 - 1476 .

LIU R S , LIU Z , LIU J Y , et al . A task-guided, implicitly-searched and meta-initialized deep model for image fusion [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024 , 46 ( 10 ): 6594 - 6609 .

乔通 , 陈彧星 , 谢世闯 , 等 . 多色彩通道特征融合的GAN合成图像检测方法 [J ] . 电子学报 , 2024 , 52 ( 3 ): 924 - 936 .

QIAO T , CHEN Y X , XIE S C , et al . GAN synthetic image detection using fused features in the multi-color channels [J ] . Acta Electronica Sinica , 2024 , 52 ( 3 ): 924 - 936 . (in Chinese)

ZHAO Z X , BAI H W , ZHANG J S , et al . Equivariant multi-modality image fusion [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2024 : 25912 - 25921 .

WANG Z D , ZHENG L , LIU Y X , et al . Towards real-time multi-object tracking [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2020 : 107 - 122 .

VOIGTLAENDER P , KRAUSE M , OSEP A , et al . MOTS: Multi-object tracking and segmentation [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2019 : 7934 - 7943 .

LIANG C , ZHANG Z P , ZHOU X , et al . Rethinking the competition between detection and ReID in multiobject tracking [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 3182 - 3196 .

WOJKE N , BEWLEY A , PAULUS D . Simple online and realtime tracking with a deep association metric [C ] // 2017 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2017 : 3645 - 3649 .

SUN S J , AKHTAR N , SONG H S , et al . Deep affinity network for multiple object tracking [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021 , 43 ( 1 ): 104 - 119 .

GUO S , WANG J Y , WANG X C , et al . Online multiple object tracking with cross-task synergy [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 8132 - 8141 .

REN H , HAN S D , DING H L , et al . Focus on details: Online multi-object tracking with diverse fine-grained representation [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 11289 - 11298 .

PANG J M , QIU L L , LI X , et al . Quasi-dense similarity learning for multiple object tracking [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 164 - 173 .

YU E , LI Z L , HAN S D . Towards discriminative representation: Multi-view trajectory contrastive learning for online multi-object tracking [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 8824 - 8833 .

KIM C , LI F X , ALOTAIBI M , et al . Discriminative appearance modeling with multi-track pooling for real-time multi-object tracking [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 9548 - 9557 .

YOU S S , YAO H T , BAO B K , et al . UTM: A unified multiple object tracking model with identity-aware feature enhancement [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 21876 - 21886 .

BEWLEY A , GE Z Y , OTT L , et al . Simple online and realtime tracking [C ] // 2016 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2016 : 3464 - 3468 .

CAO J K , PANG J M , WENG X S , et al . Observation-centric sort: Rethinking sort for robust multi-object tracking [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 9686 - 9696 .

QIN Z , ZHOU S P , WANG L , et al . MotionTrack: Learning robust short-term and long-term motions for multi-object tracking [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 17939 - 17948 .

CHEN L , AI H Z , ZHUANG Z J , et al . Real-time multiple people tracking with deeply learned candidate selection and person re-identification [C ] // 2018 IEEE International Conference on Multimedia and Expo (ICME) . Piscataway : IEEE , 2018 : 1 - 6 .

ZHANG Y F , SUN P Z , JIANG Y , et al . Bytetrack: Multi-object tracking by associating every detection box [M ] // Lecture Notes in Computer Science . Cham : Springer Nature Switzerland , 2022 : 1 - 21 .

HUANG Z Y , FU C H , LI Y M , et al . Learning aberrance repressed correlation filters for real-time UAV tracking [C ] // 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2019 : 2891 - 2900 .

LIN F L , FU C H , HE Y J , et al . BiCF: Learning bidirectional incongruity-aware correlation filter for efficient UAV object tracking [C ] // 2020 IEEE International Conference on Robotics and Automation (ICRA) . Piscataway : IEEE , 2020 : 2365 - 2371 .

CHEN J J , XU T F , HUANG B , et al . ARTracker: Compute a more accurate and robust correlation filter for UAV tracking [J ] . IEEE Geoscience and Remote Sensing Letters , 2022 , 19 : 6514605 .

CAO Z A , FU C H , YE J J , et al . SiamAPN: Siamese attentional aggregation network for real-time UAV tracking [C ] // 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . Piscataway : IEEE , 2021 : 3086 - 3092 .

YE J J , FU C H , CAO Z A , et al . Tracker meets night: A transformer enhancer for UAV tracking [J ] . IEEE Robotics and Automation Letters , 2022 , 7 ( 2 ): 3866 - 3873 .

XING D T , EVANGELIOU N , TSOUKALAS A , et al . Siamese transformer pyramid networks for real-time UAV tracking [C ] // 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) . Piscataway : IEEE , 2022 : 1898 - 1907 .

YU H Y , LI G R , SU L , et al . Conditional GAN based individual and global motion fusion for multiple object tracking in UAV videos [J ] . Pattern Recognition Letters , 2020 , 131 : 219 - 226 .

LIU S , LI X , LU H C , et al . Multi-object tracking meets moving UAV [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 8866 - 8875 .

WU H , NIE J H , HE Z W , et al . One-shot multiple object tracking in UAV videos using task-specific fine-grained features [J ] . Remote Sensing , 2022 , 14 ( 16 ): 3853 - 3872 .

CHENG S , YAO M B , XIAO X M . DC-MOT: Motion deblurring and compensation for multi-object tracking in UAV videos [C ] // 2023 IEEE International Conference on Robotics and Automation (ICRA) . Piscataway : IEEE , 2023 : 789 - 795 .

SHI L K , ZHANG Q R , PAN B , et al . Global-local and occlusion awareness network for object tracking in UAVs [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2023 , 16 : 8834 - 8844 .

CHEN G L , ZHU P F , CAO B , et al . Cross-drone transformer network for robust single object tracking [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2023 , 33 ( 9 ): 4552 - 4563 .

LOWE D G . Distinctive image features from scale-invariant keypoints [J ] . International Journal of Computer Vision , 2004 , 60 ( 2 ): 91 - 110 .

FISCHLER M A , BOLLES R C . Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography [M ] // Readings in Computer Vision . Amsterdam : Elsevier , 1987 : 726 - 740 .

JADERBERG M , SIMONYAN K , ZISSERMAN A , et al . Spatial transformer network [C ] // NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 . Massachusetts, MIT Press , 2015 : 2017 - 2025 .

REN S Q , HE K M , GIRSHICK R , et al . Faster R-CNN: Towards real-time object detection with region proposal networks [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 6 ): 1137 - 1149 .

CHEN K , WANG J Q , PANG J M , et al . MMDetection: Open MMLab detection toolbox and benchmark [EB/OL ] . ( 2019-06-17 )[ 2024-03-25 ] . https://arxiv.org/abs/1906.07155v1 https://arxiv.org/abs/1906.07155v1 .

LINDENBERGER P , SARLIN P E , POLLEFEYS M . LightGlue: Local feature matching at light speed [C ] // 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2023 : 17581 - 17592 .

EDSTEDT J , SUN Q Y , BÖKMAN G , et al . RoMa: Robust dense feature matching [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2024 : 19790 - 19800 .

ZHU Y H , SUN X Y , WANG M , et al . Multi-modal feature pyramid transformer for RGB-infrared object detection [J ] . IEEE Transactions on Intelligent Transportation Systems , 2023 , 24 ( 9 ): 9984 - 9995 .

ZHAO Z X , BAI H W , ZHANG J S , et al . CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 5906 - 5916 .

FENG C J , ZHONG Y J , GAO Y , et al . TOOD: Task-aligned one-stage object detection [C ] // 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2021 : 3490 - 3499 .

ZHU B J , WANG J F , JIANG Z K , et al . AutoAssign: Differentiable label assignment for dense object detection [EB/OL ] . ( 2020-07-07 )[ 2024-03-25 ] . https://arxiv.org/abs/2007.03496v3 https://arxiv.org/abs/2007.03496v3 .

WANG J Q , CHEN K , XU R , et al . CARAFE: Content-aware reassembly of features [C ] // 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2019 : 3007 - 3016 .

VU T , JANG H , PHAM T X , et al . Cascade RPN: Delving into high-quality region proposal network with adaptive convolution [EB/OL ] . ( 2019-09-15 )[ 2024-03-25 ] . https://arxiv.org/abs/1909.06720v2 https://arxiv.org/abs/1909.06720v2 .

HE L X , LIAO X Y , LIU W , et al . FastReID: A pytorch toolbox for general instance re-identification [C ] // Proceedings of the 31st ACM International Conference on Multimedia . New York : ACM , 2023 : 9664 - 9667 .

CHAVDAROVA T , BAQUÉ P , BOUQUET S , et al . WILDTRACK: A multi-camera HD dataset for dense unscripted pedestrian detection [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 5030 - 5039 .

FENG W , WANG F F , HAN R Z , et al . Unveiling the power of self-supervision for multi-view multi-human association and tracking [EB/OL ] . ( 2024-01-31 )[ 2024-03-25 ] . https://arxiv.org/abs/2401.17617v1 https://arxiv.org/abs/2401.17617v1 .

ONG J , VO B T , VO B N , et al . A bayesian filter for multi-view 3D multi-object tracking with occlusion handling [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022 , 44 ( 5 ): 2246 - 2263 .

YOU Q Z , JIANG H . Real-time 3D deep multi-camera tracking [EB/OL ] . ( 2020-03-26 )[ 2024-03-25 ] . https://arxiv.org/abs/2003.11753v1 https://arxiv.org/abs/2003.11753v1 .

CHENG C C , QIU M X , CHIANG C K , et al . ReST: A reconfigurable spatial-temporal graph model for multi-camera multi-object tracking [C ] // 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2023 : 10017 - 10026 .

BERGMANN P , MEINHARDT T , LEAL-TAIXÉ L . Tracking without bells and whistles [C ] // 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2019 : 941 - 951 .

ZHOU X Y , KOLTUN V , KRÄHENBÜHL P . Tracking objects as points [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2020 : 474 - 490 .

WU J L , CAO J L , SONG L C , et al . Track to detect and segment: An online multi-object tracker [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 12347 - 12356 .

MEINHARDT T , KIRILLOV A , LEAL-TAIXÉ L , et al . Trackformer: Multi-object tracking with transformers [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 8834 - 8844 .

RISTANI E , TOMASI C . Features for multi-target multi-camera tracking and re-identification [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 6036 - 6046 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research of Moving Object Detection Based on Deep Frame Difference Convolution Neural Network

Related Author

OU Xian-feng

YAN Peng-cheng

WANG Han-pu

TU Bing

HE Wei

ZHANG Guo-yun

XU Zhi

Related Institution

湖南理工学院信息科学与工程学院机器视觉与人工智能研究中心

桂林电子科技大学广西图像图形智能处理重点实验室

⁰