A Semantic Segmentation Method of Embryo Image Based on Curriculum Learning

TANG Hong-zhong; WANG Wei; WANG Tao; LU Wang-da; HUANG Xiang-hong; ZHANG Jin

doi:10.12263/DZXB.20230097

您当前的位置：

首页 >

文章列表页 >

A Semantic Segmentation Method of Embryo Image Based on Curriculum Learning

PAPERS | 更新时间：2025-12-08

- A Semantic Segmentation Method of Embryo Image Based on Curriculum Learning
- ACTA ELECTRONICA SINICA Vol. 51, Issue 11, Pages: 3365-3376(2023)
- 作者机构：
  
  1.湘潭大学自动化与电子信息学院,湖南湘潭 411105
  2.湘潭大学智能计算与信息处理教育部重点实验室,湖南湘潭 411105
  3.湖南大学电气与信息工程学院,湖南长沙 410082
  4.湘潭市中心医院,湖南湘潭 411199
- 作者简介：
- 基金信息：
  
  Major Project of Cooperative Prevention and Control of Birth Defects in Hunan Province(2019SK1010);Joint Fund of Natural Science of Hunan province(2023JJ50242);Joint Fund for Regional Innovation and Development of National Natural Science Foundation in China(U19A2083);Key Projects of Education Department of Hunan Province(22A0115);Open Project of Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education, Xiangtan University(2020ICIP06)
- DOI：10.12263/DZXB.20230097
  CLC： TP301.6;TP391.41
- Received：01 February 2023，
  
  Revised：2023-09-07，
  
  Published：25 November 2023
- 稿件说明：
移动端阅览
汤红忠,王蔚,王涛等.一种基于课程学习的胚胎图像语义分割方法[J].电子学报,2023,51(11):3365-3376.

TANG Hong-zhong,WANG Wei,WANG Tao,et al.A Semantic Segmentation Method of Embryo Image Based on Curriculum Learning[J].ACTA ELECTRONICA SINICA,2023,51(11):3365-3376.
汤红忠,王蔚,王涛等.一种基于课程学习的胚胎图像语义分割方法[J].电子学报,2023,51(11):3365-3376. DOI： 10.12263/DZXB.20230097.

TANG Hong-zhong,WANG Wei,WANG Tao,et al.A Semantic Segmentation Method of Embryo Image Based on Curriculum Learning[J].ACTA ELECTRONICA SINICA,2023,51(11):3365-3376. DOI： 10.12263/DZXB.20230097.

摘要

胚胎植入前形态学特征是人类体外受精胚胎质量评估的重要依据.目前，胚胎学家主要利用胚胎时差成像（Time-Lapse Imaging，TLI）技术观察胚胎图像形态变化，从而筛选最有发育潜能的胚胎进行移植或冷冻保存.然而，人工评估不仅费时费力，且需要较强的专业知识，并存在一定的主观性等.针对这一问题，该文提出一种基于课程学习的胚胎图像语义分割方法，实现了胚胎细胞、细胞质与雌雄原核的分割，为后续胚胎质量评估提供定量的形态特征参数.首先，利用评估语义分割算法性能的IoU指标（Intersection over Union，IoU）构建课程学习的难度评分函数（Scoring Function，SF），根据SF评分将所有样本从易到难进行排序；再结合SF评分与目标类别数定义课程学习的步调函数（Pacing Functions，PF），构建了难易程度递增的样本子集；最后，设计多阶段渐进式U-net语义分割（Multi-Stage Progressive U-net，MSPU）模型，根据课程难度顺序依次训练不同阶段的网络，从而实现胚胎图像的语义分割.相关实验结果表明，本文提出的MSPU模型在胚胎图像语义分割任务上获得较好的性能，与基准模型相比，IoU值提高1.4%；特别是在较易与较难的分割任务上具有不错的表现，如细胞与雌雄原核的分割IoU值分别提升了4.6%与1.2%.

Abstract

Morphological features of embryo play an important role to evaluate the quality of human embryos in vitro fertilization. Embryologists mainly use time-lapse imaging (TLI) technology to observe the morphological variation of embryo images and select the most potential embryos for subsequent transfer or cryopreservation. However

manual evaluation is not only time-consuming and laborious

but it also demands specialty-oriented skills and has some subjectivities. To solve this problem

we propose a semantic segmentation method of embryo images based on curriculum learning to segment embryo cell

cytoplasm and pronucleus

which can provide the quantitative parameters of morphological feature for the subsequent evaluation of embryo quality. First

IoU (Intersection over Union) index that is usually used to evaluate the performance of semantic segmentation algorithms is adopted to construct a difficulty scoring function (SF) of curriculum learning. All samples are sorted from easy to difficult according to SF value. Second

we define a pace function (PF) by combining the SF and the number of target categories

and sample subsets is established with increasing sequence of difficulty. Lastly

we design a multi-stage progressive U-net (MSPU) model to segment embryo cell

cytoplasm and pronucleus embryo images at pronuclear-stage

in which the network at different stages are trained using the sample subsets with increasing sequence of difficulty. Experimental results demonstrate that our proposed MSPU model obtains a satisfactory performance on the semantic segmentation of embryo images

and IoU is improved by 1.4% compared with Vanilla Baseline. Our proposed MSPU model shows a pleasing consistency on the easy and difficult segmentation tasks

for example the improvement of 4.6% and 1.2% can be obtained for the segmentation of embryonic cells and pronucleus

respectively.

关键词

Keywords

references

HANEVIK H I , HESSEN D O . IVF and human evolution [J ] . Human Reproduction Update , 2022 , 28 ( 4 ): 457 - 479 .

PUISSANT F , VAN R M , BARLOW P , et al . Embryo scoring as a prognostic tool in IVF treatment [J ] . Human Reproduction , 1987 , 2 ( 8 ): 705 - 708 .

耿蒙慧 , 张璨 , 邢阿英 , 等 . 胚胎质量评估方法的研究进展 [J ] . 中华妇幼临床医学杂志 (电子版), 2018 , 14 ( 5 ): 612 - 616 .

GENG M H , ZHANG C , XING A Y , et al . Research progress of methods on the assessment of embryonic quality [J ] . Chinese Journal of Obstetrics & Gynecology and Pediatrics (Electronic Edition) , 2018 , 14 ( 5 ): 612 - 616 . (in Chinese)

MENG Q X , XU Y Y , ZHENG A Y , et al . Noninvasive embryo evaluation and selection by time-lapse monitoring vs. conventional morphologic assessment in women undergoing in vitro fertilization/intracytoplasmic sperm injection: A single-center randomized controlled study [J ] . Fertility and Sterility , 2022 , 117 ( 6 ): 1203 - 1212 .

ARMSTRONG S , ARROLL N , CREE L M , et al . Time-lapse systems for embryo incubation and assessment in assisted reproduction [J ] . Cochrane Database of Systematic Reviews , 2014 ( 5 ): CD011320 .

黄国宁 . Time-lapse是否能预测胚胎染色体整倍性? [J ] . 生殖医学杂志 , 2019 , 28 ( 11 ): 1249 - 1250 .

HUANG G N . Could time-lapse predict euploidy embryos? [J ] . Journal of Reproductive Medicine , 2019 , 28 ( 11 ): 1249 - 1250 . (in Chinese)

ADAMSON G D , ABUSIEF M E , PALAO L , et al . Improved implantation rates of day 3 embryo transfers with the use of an automated time-lapse-enabled test to aid in embryo selection [J ] . Fertility and Sterility , 2016 , 105 ( 2 ): 369 - 375 .

WATANABE H , FUKUNAGA N , SANAMI S , et al . Development of an automatic pronuclear detection system for human embryos using deep learning technology [J ] . Fertility and Sterility , 2019 , 112 ( 3 ): e429 - e430 .

LIAO Q Y , ZHANG Q , FENG X E , et al . Development of deep learning algorithms for predicting blastocyst formation and quality by time-lapse monitoring [J ] . Communications Biology , 2021 , 4 ( 1 ): 415 .

BORMANN C L , KANAKASABAPATHY M K , THIRUMALARAJU P , et al . Performance of a deep learning based neural network in the selection of human blastocysts for implantation [J ] . eLife , 2020 , 9 : 55301 .

LEAHY B D , JANG W D , YANG H Y , et al . Automated measurements of key morphological features of human embryos for IVF [C ] // Medical Image Computing and Computer Assisted Intervention — MICCAI 2020 . Cham : Springer International Publishing , 2020 : 25 - 35 .

ZHAO M P , XU M R , LI H H , et al . Application of convolutional neural network on early human embryo segmentation during in vitro fertilization [J ] . Journal of Cellular and Molecular Medicine , 2021 , 25 ( 5 ): 2633 - 2644 .

SAWADA Y , SATO T , NAGAYA M , et al . Evaluation of artificial intelligence using time-lapse images of IVF embryos to predict live birth [J ] . Reproductive Biomedicine Online , 2021 , 43 ( 5 ): 843 - 852 .

KRAGH M F , RIMESTAD J , BERNTSEN J , et al . Automatic grading of human blastocysts from time-lapse imaging [J ] . Computers in Biology and Medicine , 2019 , 115 : 103494 .

GARGEE V , SHREYA C , RUCHI G , et al . Time series prediction of viable embryo and automatic grading in IVF using deep learning [J ] . The Open Biomedical Engineering Journal , 2021 , 15(Suppl-2): 190- 203 .

VERMILYEA M , HALL J M M , DIAKIW S M , et al . Development of an artificial intelligence-based assessment model for prediction of embryo viability using static images captured by optical light microscopy during IVF [J ] . Human Reproduction (Oxford, England) , 2020 , 35 ( 4 ): 770 - 784 .

THIRUMALARAJU P , KANAKASABAPATHY M K , BORMANN C L , et al . Evaluation of deep convolutional neural networks in classifying human embryo images based on their morphological quality [J ] . Heliyon , 2021 , 7 ( 2 ): e06298 .

BENGIO Y , LOURADOUR J , COLLOBERT R , et al . Curriculum learning [C ] // Proceedings of the 26th Annual International Conference on Machine Learning . New York : ACM , 2009 : 41 - 48 .

WANG X , CHEN Y D , ZHU W W . A survey on curriculum learning [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022 , 44 ( 9 ): 4555 - 4576 .

SOVIANY P , IONESCU R T , ROTA P , et al . Curriculum learning: A survey [J ] . International Journal of Computer Vision , 2022 , 130 ( 6 ): 1526 - 1565 .

PLATANIOS E A , STRETCU O , NEUBIG G , et al . Competence-based curriculum learning for neural machine translation [C ] // Proceedings of the 2019 Conference of the North . Stroudsburg : Association for Computational Linguistics , 2019 : 1162 - 1172 .

SPITKOVSKY V I , ALSHAWI H , JURAFSKY D . From baby steps to Leapfrog: How “Less is more” in unsupervised dependency parsing [C ] // HLT 10: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics . New York : ACM , 2010 : 751 - 759 .

WEI Y C , LIANG X D , CHEN Y P , et al . STC: A simple to complex framework for weakly-supervised semantic segmentation [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 11 ): 2314 - 2320 .

KOCMI T , BOJAR O . Curriculum learning and minibatch bucketing in neural machine translation [C ] // RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning . Incoma Ltd . Shoumen, Bulgaria , 2017 : 379 - 386 .

RANJAN S , HANSEN J H L . Curriculum learning based approaches for noise robust speaker recognition [J ] . ACM Transactions on Audio, Speech, and Language Processing , 2018 , 26 ( 1 ): 197 - 210 .

WEI J , SURIAWINATA A , REN B , et al . Learn like a pathologist: Curriculum learning by annotator agreement for histopathology image classification [C ] // 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) . Piscataway : IEEE , 2021 : 2473 - 2483 .

GRAVES A , BELLEMARE M G , MENICK J , et al . Automated curriculum learning for neural networks [C ] // Proceedings of the 34th International Conference on Machine Learning - Volume 70 . New York : ACM , 2017 : 1311 - 1320 .

KUMAR M P , PACKER B , KOLLER D . Self-paced learning for latent variable models [C ] // The 23rd International Conference on Neural Information Processing Systems . New York : ACM , 2010 : 1189 - 1197 .

TULLIS J G , BENJAMIN A S . On the effectiveness of self-paced learning [J ] . Journal of Memory and Language , 2011 , 64 ( 2 ): 109 - 118 .

JIANG L , MENG D , ZHAO Q , et al . Self-paced curriculum learning [C ] // The 29th AAAI Conference on Artificial Intelligence (AAAI) . Menlo Park : AAAI Press , 2015 : 2694 - 2700 .

MATIISEN T , OLIVER A , COHEN T , et al . Teacher-student curriculum learning [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2020 , 31 ( 9 ): 3732 - 3740 .

FAN Y , TIAN F , QIN T , et al . Learning to teach [C ] // The 6th International Conference on Learning Representations (ICLR) . Vancouver : ICLR , 2018 : 1 - 16 .

SHRIVASTAVA A , GUPTA A , GIRSHICK R . Training region-based object detectors with online hard example mining [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2016 : 761 - 769 .

JIMÉNEZ-SÁNCHEZ A , MATEUS D , KIRCHHOFF S , et al . Medical-based deep curriculum learning for improved fracture classification [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2019 : 694 - 702 .

ZHAO R C , CHEN X L , CHEN Z L , et al . Diagnosing glaucoma on imbalanced data with self-ensemble dual-curriculum learning [J ] . Medical Image Analysis , 2022 , 75 : 102295 .

ZHANG Y , DAVID P , GONG B Q . Curriculum domain adaptation for semantic segmentation of urban scenes [C ] // 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2017 : 2020 - 2030 .

DAI D X , SAKARIDIS C , HECKER S , et al . Curriculum model adaptation with synthetic and real data for semantic foggy scene understanding [J ] . International Journal of Computer Vision , 2020 , 128 ( 5 ): 1182 - 1204 .

文载道 , 王佳蕊 , 王小旭 , 等 . 解耦表征学习综述 [J ] . 自动化学报 , 2022 , 48 ( 2 ): 351 - 374 .

WEN Z D , WANG J R , WANG X X , et al . A review of disentangled representation learning [J ] . Acta Automatica Sinica , 2022 , 48 ( 2 ): 351 - 374 . (in Chinese)

任莎莎 , 刘琼 . 小目标特征增强图像分割算法 [J ] . 电子学报 , 2022 , 50 ( 8 ): 1894 - 1904 .

REN S S , LIU Q . A tiny target feature enhancement algorithm for semantic segmentation [J ] . Acta Electronica Sinica , 2022 , 50 ( 8 ): 1894 - 1904 . (in Chinese)

MA J , CHEN J N , NG M , et al . Loss odyssey in medical image segmentation [J ] . Medical Image Analysis , 2021 , 71 : 102035 .

RONNEBERGER O , FISCHER P , BROX T . U-net: Convolutional networks for biomedical image segmentation [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2015 : 234 - 241 .

LONG J , SHELHAMER E , DARRELL T . Fully convolutional networks for semantic segmentation [C ] // 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2015 : 3431 - 3440 .

BADRINARAYANAN V , KENDALL A , CIPOLLA R . SegNet: A deep convolutional encoder-decoder architecture for image segmentation [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 12 ): 2481 - 2495 .

CHEN L C , ZHU Y K , PAPANDREOU G , et al . Encoder-decoder with atrous separable convolution for semantic image segmentation [C ] // Computer Vision — ECCV 2018 . Cham : Springer International Publishing , 2018 : 833 - 851 .

LAMBERT Z , PETITJEAN C , DUBRAY B , et al . SegTHOR: Segmentation of thoracic organs at risk in CT images [C ] // 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA) . Piscataway : IEEE , 2020 : 1 - 6 .

ISENSEE F , JAEGER P F , KOHL S A A , et al . NNU-Net: A self-configuring method for deep learning-based biomedical image segmentation [J ] . Nature Methods , 2021 , 18 ( 2 ): 203 - 211 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

DFRNet: A Semantic Segmentation Method Inspired with Physical Mechanism of Diffusion-Focus

Remote Sensing Image Segmentation of Around Plateau Lakes Based on Multi-Attention Fusion

Semantic Segmentation Network Based on Semantic and Morphological Feature Fusion

Related Author

FANG Xiao-ping

HUANG Yi-sha

JIANG Lin

GUAN Ya-fei

ZHANG Ya-sha

LIANG Xin

ZENG Wei-hao

LI Su-min

Related Institution

School of Mathematics and Statistics, Hunan University of Technology and Business

School of Intelligent Engineering and Manufacturing, Hunan University of Technology and Business

Xiangjiang Laboratory

School of AI and Advanced Computing, Hunan University of Technology and Business

College of Mechanical and Electrical Engineering， Kunming University of Science and Technology

⁰