医学图像关键点检测深度学习方法研究与挑战

doi:10.12263/DZXB.20200725

电子学报 ›› 2022, Vol. 50 ›› Issue (1): 226-237.DOI: 10.12263/DZXB.20200725

医学图像关键点检测深度学习方法研究与挑战

李居朋¹, 王颖慧², 李刚²

^1.北京交通大学电子信息工程学院，北京 100044
^2.北京大学口腔医学院，北京 100081

收稿日期:2020-07-16 修回日期:2020-12-17 出版日期:2022-01-25 发布日期:2022-01-25
作者简介:李居朋男，1981年8月出生于江苏省沛县.2009年获得北京交通大学大学博士学位.2019年赴北卡罗来纳大学教堂山分校研修，现为北京交通大学副教授、研究生导师，主要研究方向为医学图像分割及多模态配准等. E-mail:lijupeng@bjtu.edu.cn
王颖慧女，1992年1月出生于河北省保定市.2018年毕业于首都医科大学长学制（本硕连读）口腔医学专业.现为北京大学口腔医学院口腔颌面医学影像学在读博士研究生，主要研究方向为数字化口腔颌面医学影像.
李刚男，1970年2月出生于黑龙江省哈尔滨市.2004年于瑞典卡罗林斯卡医学院获博士学位.现为北京大学教授、博士生导师，中国电子学会会员，中华口腔医学会口腔颌面放射专委会副主任委员，口腔医学院医学影像科副主任，主要研究方向为数字化口腔医学影像的基础和临床研究等.E-mail:kqgang@bjmu.edu.cn
基金资助:
国家自然科学基金(81671034);军队后勤科研重大项目(AWS17J011);国家留学基金(201907095037)

Research and Challenges of Medical Image Landmark Detection Based on Deep Learning

LI Ju-peng¹, WANG Ying-hui², LI Gang²

^1.School of Electronic and Information Engineering，Beijing Jiaotong University，Beijing 100044，China
^2.Peking University School of Stomatology，Beijing 100081，China

Received:2020-07-16 Revised:2020-12-17 Online:2022-01-25 Published:2022-01-25

摘要/Abstract

摘要：

作为众多医学图像处理的前提和关键，医学图像关键点检测具有重要的理论研究和应用价值.由于个体间差异性和个体内歧义性的影响，以及更高的临床应用定位精度的要求，医学解剖关键点检测面临着巨大的挑战.鉴于深度学习技术在医学图像关键点检测乃至整个医学图像处理领域都表现出了强大的实力，本文全面检索发表于顶级医学期刊和会议论文集中的医学图像关键点研究成果并进行了详细的梳理和综述.从计算机视觉任务角度简述医学图像关键点检测及其存在的难点；总结了深度学习技术在医学图像关键点检测中的基本框架，详细论述了医学图像关键点检测的分类问题和回归分析两种不同类型的解决思路；最后探讨了医学图像关键点检测深度学习方法面临的挑战、主要应对策略和开放的研究方向.

关键词: 医学图像处理, 关键点检测, 深度学习, 卷积神经网络

Abstract:

As an entrance and challenge for many medical images processing, it is clinically essential to study on the medical image landmark detection and localization. Due to inter-individual variability and intra-individual ambiguity, as well as higher accuracy requirements of clinical application, the detection of medical anatomical landmarks was facing enormous challenges. In view of strength of deep learning of medical image landmark detection and the entire medical image processing field, we comprehensively retrieves relevant papers published in the top medical journals and conference proceedings to conduct a detailed review of these research findings. First of all, we briefly introduce difficulties in medical image landmark detection from the view of computer vision tasks. Secondly, we describe basic framework in medical image landmark detection, and discuss two different categories: classification and regression landmark detection solutions. Finally, we discuss the challenges and practicable strategies in deep learning for medical image landmark detection, as well as open research.

Key words: medical images processing, landmark detection, deep learning, convolutional neural networks

中图分类号:

TP391.4

李居朋, 王颖慧, 李刚. 医学图像关键点检测深度学习方法研究与挑战[J]. 电子学报, 2022, 50(1): 226-237.

LI Ju-peng, WANG Ying-hui, LI Gang. Research and Challenges of Medical Image Landmark Detection Based on Deep Learning[J]. Acta Electronica Sinica, 2022, 50(1): 226-237.

图/表 9

参考文献 58

1	HANAOKA S, SHIMIZU A, NEMOTO M, et al. Automatic detection of over 100 anatomical landmarks in medical CT images: A framework with independent detectors and combinatorial optimization[J]. Medical Image Analysis, 2017, 35: 192-214.
2	GE J W, SAEIDI H, OPFERMANN J D, et al. Landmark-guided deformable image registration for supervised autonomous robotic tumor resection[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2019.Cham, GER: Springer, 2019: 320-328.
3	PETERLÍK I, COURTECUISSE H, ROHLING R, et al. Fast elastic registration of soft tissues under large deformations[J]. Medical Image Analysis, 2018, 45: 24-40.
4	MADER A O, BERG J, FABRITZ A, et al. Localization and labeling of posterior ribs in chest radiographs using a CRF-regularized FCN with local refinement[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2018. Cham, GER: Springer, 2018: 562-570.
5	TOROSDAGLI N, LIBERTON D K, VERMA P, et al. Deep geodesic learning for segmentation and anatomical landmarking[J]. IEEE Transactions on Medical Imaging, 2019, 38(4): 919-931.
6	ZHANG J, LIU M, WANG L, et al. Joint craniomaxillofacial bone segmentation and landmark digitization by context-guided fully convolutional networks[J]. Medical Image Computing and Computer-Assisted Intervention, 2017, 10434: 720-728.
7	ZHANG J, LIU M X, WANG L, et al. Context-guided fully convolutional networks for joint craniomaxillofacial bone segmentation and landmark digitization[J]. Medical Image Analysis, 2020, 60: 101621.
8	ZHANG J, SAHA A, ZHU Z, et al. Hierarchical convolutional neural networks for segmentation of breast tumors in MRI with application to radiogenomics[J]. IEEE Transactions on Medical Imaging, 2019, 38(2): 435-447.
9	LIU C B, XIE H T, ZHANG S C, et al. Misshapen pelvis landmark detection by spatial local correlation mining for diagnosing developmental dysplasia of the hip[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2019. Cham, GER: Springer, 2019: 441-449.
10	ZHENG Y F, LIU D, GEORGESCU B, et al. 3D deep learning for efficient and robust landmark detection in volumetric data[C]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. Cham, GER: Springer, 2015: 565-572.
11	AUBERT B, VAZQUEZ C, CRESSON T, et al. Toward automated 3D spine reconstruction from biplanar radiographs using CNN for statistical spine model fitting[J]. IEEE Transactions on Medical Imaging, 2019, 38(12): 2796-2806.
12	NOOTHOUT J M H, VOS B D D, WOLTERINK J M, et al. CNN-based landmark detection in cardiac CTA scans[EB/OL]. [2018]. .
13	GREWAL M, DEIST T M, WIERSMA J, et al. An end-to-end deep learning approach for landmark detection and matching in medical images[EB/OL]. [2020]. .
14	ZHANG J, LIU M X, SHEN D G. Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks[J]. IEEE Transactions on Image Processing, 2017, 26(10): 4753-4764.
15	DUAN J M, BELLO G, SCHLEMPER J, et al. Automatic 3D Bi-ventricular segmentation of cardiac images by a shape-refined multi- task deep learning approach[J]. IEEE Transactions on Medical Imaging, 2019, 38(9): 2151-2164.
16	TIULPIN A, MELEKHOW I, SAARAKKALA S. Knee anatomical landmark localization using hourglass networks[EB/OL]. [2019]. .
17	IBRAGIMOV B, LIKAR B, PEMUS F, et al. Automatic cephalometric x-ray landmark detection by applying game theory and random forests[C]//Proceedings of ISBI International Symposium on Biomedical Imaging. Cham, GER: Springer, 2014: 1-8.
18	LINDNER C, COOTES T F. Fully automatic cephalometric evaluation using random forest regression-voting[C]//Proceedings of ISBI International Symposium on Biomedical Imaging. Cham, GER: Springer, 2015: 1-8.
19	WANG C W, HHANG C T, HSIEH M C, et al. Evaluation and comparison of anatomical landmark detection methods for cephalometric X-ray images: A grand challenge[J]. IEEE Transactions on Medical Imaging, 2015, 34(9): 1890-1900.
20	WANG C W, HUANG C T, LEE J H, et al. A benchmark for comparison of dental radiography analysis algorithms[J]. Medical Image Analysis, 2016, 31: 63-76.
21	BIER B, GOLDMANN F, ZAECH J N, et al. Learning to detect anatomical landmarks of the pelvis in X-rays from arbitrary views[J]. International Journal of Computer Assisted Radiology and Surgery, 2019, 14(9): 1463-1473.
22	YANG D, XIONG T, XU D G, et al. Automatic vertebra labeling in large-scale 3D CT using deep image-to-image network with message passing and sparsity regularization[C]//Information Processing in Medical Imaging. Cham, GER: Springer, 2017: 633-644.
23	PROBST T, MANINIS K K, CHHATKULI A, et al. Automatic tool landmark detection for stereo vision in robot-assisted retinal surgery[J]. IEEE Robotics and Automation Letters, 2018, 3(1): 612-619.
24	TEIXEIRA B, TAMERSOY B, SINGH V, et al. Adaloss: Adaptive loss function for landmark localization[EB/OL]. [2019]..
25	ZHANG D Q, WANG J N, NOBLE J H, et al. HeadLocNet: Deep convolutional neural networks for accurate classification and multi-landmark localization of head CTs[J]. Medical Image Analysis, 2020, 61: 101659.
26	URSCHLER M, EBNER T, ŠTERN D. Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization[J]. Medical Image Analysis, 2018, 43: 23-36.
27	CHEN R N, MA Y X, CHEN N L, et al. Cephalometric landmark detection by attentive feature pyramid fusion and regression-voting[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2019. Cham, GER: Springer, 2019: 873-881.
28	LÉVY-MANDEL A D, VENETSANOPOULOS A N, TSOTSOS J K. Knowledge-based landmarking of cephalograms[J]. Computers and Biomedical Research, 1986, 19(3): 282-309.
29	EL-FEGHI I, SID-AHMED M A, AHMADI M. Automatic localization of craniofacial landmarks for assisted cephalometry[J]. Pattern Recognition, 2004, 37(3): 609-621.
30	CARDILLO J, SID-AHMED M A. An image processing system for locating craniofacial landmarks[J]. IEEE Transactions on Medical Imaging, 1994, 13(2): 275-289.
31	COOTES T F, TAYLOR C J, COOPER D H, et al. Active shape models-their training and application[J]. Computer Vision and Image Understanding, 1995, 61(1): 38-59.
32	COOTES T F, EDWARDS G J, TAYLOR C J. Active appearance models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 681-685.
33	YUE W, YIN D, LI C, et al. Automated 2-D cephalometric analysis on X-ray images by a model-based approach[J]. IEEE Transactions on Biomedical Engineering, 2006, 53(8): 1615-1623.
34	LINDNER C, WANG C W, HUANG C T, et al. Fully automatic system for accurate localisation and analysis of cephalometric landmarks in lateral cephalograms[J]. Scientific Reports, 2016, 6: 33581.
35	ARlK S Ö, IBRAGIMOV B, XING L. Fully automated quantitative cephalometry using convolutional neural networks[J]. Journal of Medical Imaging (Bellingham, Wash.), 2017, 4(1): 014501.
36	DROSTE R, CAI Y F, SHARMA H, et al. Ultrasound image representation learning by modeling sonographer visual attention[C]//Information Processing in Medical Imaging. Cham, GER: Springer, 2019: 592-604.
37	LITJENS G, KOOI T, BEJNORDI B E, et al. A survey on deep learning in medical image analysis[J]. Medical Image Analysis, 2017, 42: 60-88.
38	田娟秀, 刘国才, 谷珊珊, 等. 医学图像分析深度学习方法研究与挑战[J]. 自动化学报, 2018, 44(3): 401-424.
	TIAN J X, LIU G C, GU S S, et al. Deep learning in medical image analysis and its challenges[J]. Acta Automatica Sinica, 2018, 44(3): 401-424. (in Chinese)
39	WELLS W M III. Medical Image Analysis - past, present, and future[J]. Medical Image Analysis, 2016, 33: 4-6.
40	SHEN D, WU G, SUK H I. Deep learning in medical image analysis[J]. Annual Review of Biomedical Engineering, 2017, 19: 221-248.
41	BOOKSTEIN F L. Morphometric Tools for Landmark Data[M]. Cambridge, UK: Cambridge University Press, 1992.
42	TOMPSON J, JAIN A, LECUN Y, et al. Joint training of a convolutional network and a graphical model for human pose estimation[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. New York, USA: ACM, 2014: 1799-1807.
43	WU Y, JI Q. Facial landmark detection: A literature survey[J]. International Journal of Computer Vision, 2019, 127(2): 115-142.
44	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
45	DANG Q, YIN J Q, WANG B, et al. Deep learning based 2D human pose estimation: A survey[J]. Tsinghua Science and Technology, 2019, 24(6): 663-676.
46	LI R, LIU Z Y, TAN J R. A survey on 3D hand pose estimation: Cameras, methods, and datasets[J]. Pattern Recognition, 2019, 93: 251-272.
47	CHENG W H, SONG S J, CHEN C Y, et al. Fashion meets computer vision[J]. ACM Computing Surveys, 2021, 54(4): 1-41.
48	TUYSUZOGLU A, TAN J, EISSA K, et al. Deep adversarial context-aware landmark detection for ultrasound imaging[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2018. Cham, GER: Springer, 2018: 151-158.
49	ANDERMATT S, PEZOLD S, AMANN M, et al. Multi-dimensional gated recurrent units for automated anatomical landmark localization[EB/OL].[2017]..
50	PAYER C, ŠTERN D, BISCHOF H, et al. Regressing heatmaps for multiple landmark localization using CNNs[C]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2016. Cham, GER: Springer, 2016: 230-238.
51	PAYER C, ŠTERN D, BISCHOF H, et al. Integrating spatial configuration into heatmap regression based CNNs for landmark localization[J]. Medical Image Analysis, 2019, 54: 207-219.
52	ZHONG Z S, LI J, ZHANG Z X, et al. An attention-guided deep regression model for landmark detection in cephalograms[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2019. Cham, GER: Springer, 2019: 540-548.
53	ESLAMI M, NEUSCHAEFER-RUBE C, Serrurier A. Automatic vocal tract landmark localization from midsagittal MRI data[EB/OL]. [2020]. .
54	XU Z B, HUO Y K, PARK J, et al. Less is more: Simultaneous view classification and landmark detection for abdominal ultrasound images[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2018. Cham, GER: Springer, 2018: 711-719.
55	LI Y W, ALANSARY A, CERROLAZA J J, et al. Fast multiple landmark localisation using a patch-based iterative network[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2018. Cham, GER: Springer, 2018: 563-571.
56	GREENSPAN H, GINNEKEN B VAN, SUMMERS R M. Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique[J]. IEEE Transactions on Medical Imaging, 2016, 35(5): 1153-1159.
57	SHORTEN C, KHOSHGOFTAAR T M. A survey on image data augmentation for deep learning[J]. Journal of Big Data, 2019, 6(1): 1-48.
58	DROSTE R, CHATELAIN P, DRUKKER L, et al. Discovering salient anatomical landmarks by predicting human gaze[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City, IA, USA: IEEE, 2020: 1711-1714.

文献编号	网络模型	数据类型	应用对象	方法特点描述
［25］	3D U-Net	CT	耳蜗	由FCN回归左右内耳关键点热度图，结合假阳性抑制以正确检测左、右内耳关键点
［4］	U-Net	X-Ray	肋骨	利用FCN回归16个关键点热度图，经条件随机场评估关键点间的空间信息完成局部细化
［52］	U-Net	X-Ray	颅骨	由粗略到精细两个阶段回归19个关键点的热度图，利用阶段间的Attention-Guide机制指导推理的扩展探索策略
［27］	VGG-19	X-Ray	颅骨	结合VGG-19和特征金字塔融合模块回归19个关键点热度图，经通过逐像素回归投票得出坐标
［53］	Flat-Net	MRI矢状位断层图像	声道	设计独特的Flat-Net网络结构用于回归MRI矢状位断层2D图像中21个关键点的热度图
［22］	DI2IN	CT	脊柱	结合传统Encoder-Decoder网络和监督网络设计DI2IN网络回归12个关键点热度图，经稀疏表示优化定位精度
［23］	SHN	高清手术图像	手术器具	采用堆叠沙漏网络回归高清图像中手术器具的3个关键点热度图，经关键点定位用于机器人手术控制
［24］	U-Net	X-Ray，手术图像	血管、手术钳	采用U-Net回归关键点热度图，结合新颖的自适应损失函数设计，根据训练统计信息更新目标精度
［54］	FCN	超声	腹部	采用Skip-Connection全卷积网络回归14个关键点热度图，与视图分类组成多任务学习模型提高检测精度

文献编号	网络模型	数据类型	应用对象	方法特点描述
［25］	3D U-Net	CT	耳蜗	由FCN回归左右内耳关键点热度图，结合假阳性抑制以正确检测左、右内耳关键点
［4］	U-Net	X-Ray	肋骨	利用FCN回归16个关键点热度图，经条件随机场评估关键点间的空间信息完成局部细化
［52］	U-Net	X-Ray	颅骨	由粗略到精细两个阶段回归19个关键点的热度图，利用阶段间的Attention-Guide机制指导推理的扩展探索策略
［27］	VGG-19	X-Ray	颅骨	结合VGG-19和特征金字塔融合模块回归19个关键点热度图，经通过逐像素回归投票得出坐标
［53］	Flat-Net	MRI矢状位断层图像	声道	设计独特的Flat-Net网络结构用于回归MRI矢状位断层2D图像中21个关键点的热度图
［22］	DI2IN	CT	脊柱	结合传统Encoder-Decoder网络和监督网络设计DI2IN网络回归12个关键点热度图，经稀疏表示优化定位精度
［23］	SHN	高清手术图像	手术器具	采用堆叠沙漏网络回归高清图像中手术器具的3个关键点热度图，经关键点定位用于机器人手术控制
［24］	U-Net	X-Ray，手术图像	血管、手术钳	采用U-Net回归关键点热度图，结合新颖的自适应损失函数设计，根据训练统计信息更新目标精度
［54］	FCN	超声	腹部	采用Skip-Connection全卷积网络回归14个关键点热度图，与视图分类组成多任务学习模型提高检测精度

数据集	类型	规模	链接
Digital Hand Atlas Dataset	X-Ray	895	https：//ipilab.usc.edu/research/baaweb/
Cephalometric Dataset	X-Ray	400	http：//www-o.ntust.edu.tw/~cweiwang/
Pathological Spine CT Dataset	CT	242	https：//biomedia.doc.ic.ac.uk/data/spine/
DDH Dataset	X-Ray	9813	https：//github.com/liuboss1992/FR-DDH
Indiana Chest X-ray Dataset	X-Ray	7470	https：//openi.nlm.nih.gov/
Osteoarthritis Initiative Dataset	X-Ray	748	https：//oai.epi-ucsf.org/datarelease/
Instrument Tracking Dataset	可见光	860	https：//endovissub-instrument.grand-challenge.org/

数据集	类型	规模	链接
Digital Hand Atlas Dataset	X-Ray	895	https：//ipilab.usc.edu/research/baaweb/
Cephalometric Dataset	X-Ray	400	http：//www-o.ntust.edu.tw/~cweiwang/
Pathological Spine CT Dataset	CT	242	https：//biomedia.doc.ic.ac.uk/data/spine/
DDH Dataset	X-Ray	9813	https：//github.com/liuboss1992/FR-DDH
Indiana Chest X-ray Dataset	X-Ray	7470	https：//openi.nlm.nih.gov/
Osteoarthritis Initiative Dataset	X-Ray	748	https：//oai.epi-ucsf.org/datarelease/
Instrument Tracking Dataset	可见光	860	https：//endovissub-instrument.grand-challenge.org/

[1]	张波, 陆云杰, 秦东明, 邹国建. 一种卷积自编码深度学习的空气污染多站点联合预测模型[J]. 电子学报, 2022, 50(6): 1410-1427.
[2]	丁毅, 沈薇, 李海生, 钟琼慧, 田明宇, 李洁. 面向CNN的区块链可信隐私服务计算模型[J]. 电子学报, 2022, 50(6): 1399-1409.
[3]	杨伟超, 杜宇, 文伟, 侯舒维, 徐常志, 张建华. 基于多重分形谱智能分析的卫星信号调制识别研究[J]. 电子学报, 2022, 50(6): 1336-1343.
[4]	彭闯, 王伦文, 胡炜林. 融合深度特征的电磁频谱异常检测算法[J]. 电子学报, 2022, 50(6): 1359-1369.
[5]	李政伟, 李佳树, 尤著宏, 聂茹, 赵欢, 钟堂波. 基于异质图注意力网络的miRNA与疾病关联预测算法[J]. 电子学报, 2022, 50(6): 1428-1435.
[6]	张云, 化青龙, 姜义成, 徐丹. 基于混合型复数域卷积神经网络的三维转动舰船目标识别[J]. 电子学报, 2022, 50(5): 1042-1049.
[7]	冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242.
[8]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.
[9]	潘敏婷, 王韫博, 朱祥明, 高思宇, 龙明盛, 杨小康. 基于无标签视频数据的深度预测学习方法综述[J]. 电子学报, 2022, 50(4): 869-886.
[10]	杨曦, 张鑫, 郭浩远, 王楠楠, 高新波. 基于不变特征的多源遥感图像舰船目标检测算法[J]. 电子学报, 2022, 50(4): 887-899.
[11]	崔亚奇, 何友, 唐田田, 熊伟. 一种深度学习航迹关联方法[J]. 电子学报, 2022, 50(3): 759-763.
[12]	张浩, 胡昌华, 杜党波, 裴洪, 张建勋. 多状态影响下基于Bi‑LSTM网络的锂电池剩余寿命预测方法[J]. 电子学报, 2022, 50(3): 619-624.
[13]	伍邦谷, 张苏林, 石红, 朱鹏飞, 王旗龙, 胡清华. 基于多分支结构的不确定性局部通道注意力机制[J]. 电子学报, 2022, 50(2): 374-382.
[14]	骈纬国, 吴映波, 陈蒙, 蔡俊鹏. 一种基于时空动态图注意力网络的共享出行需求预测方法[J]. 电子学报, 2022, 50(2): 432-439.
[15]	张聿远, 张立民, 闫文君. 基于深度多级残差网络的低信噪比下空频分组码识别方法[J]. 电子学报, 2022, 50(1): 79-88.

医学图像关键点检测深度学习方法研究与挑战

Research and Challenges of Medical Image Landmark Detection Based on Deep Learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 58

相关文章 15

编辑推荐

Metrics