面向机器学习模型安全的测试与修复

张笑宇; 沈超; 蔺琛皓; 李前; 王骞; 李琦; 管晓宏

doi:10.12263/DZXB.20220821

您当前的位置：

首页 >

文章列表页 >

面向机器学习模型安全的测试与修复

《电子学报》创刊60周年专栏 | 更新时间：2025-12-11

- 面向机器学习模型安全的测试与修复
- The Testing and Repairing Methods for Machine Learning Model Security
- 电子学报 2022年50卷第12期页码：2884-2918
- 作者机构：
  
  1.西安交通大学电子与信息学部网络空间安全学院，陕西西安 710049
  2.智能网络与网络安全教育部重点实验室（西安交通大学），陕西西安 710049
  3.武汉大学国家网络安全学院，湖北武汉 430072
  4.清华大学网络科学与网络空间研究院，北京 100084
  5.中关村实验室，北京 100094
- 作者简介：
  
  [ "张笑宇男，出生于1999年，河南郑州人.西安交通大学网络空间安全学院博士研究生.主要研究领域为人工智能软件测试.E-mail: zxy0927@stu.xjtu.edu.cn" ]
  [ "沈超男，出生于1985年，重庆人.博士，西安交通大学教授、博士生导师.主要研究领域为可信人工智能、人工智能安全和信息物理系统安全." ]
  [ "蔺琛皓男，出生于1989年，陕西西安人.博士，西安交通大学特聘研究员、博士生导师.主要研究领域为人工智能安全、对抗机器学习、智能身份认证.E-mail: linchenhao@xjtu.edu.cn" ]
  [ "李前男，出生于1992年，陕西宝鸡人.博士，西安交通大学助理教授.主要研究领域为人工智能安全、对抗机器学习.E-mail: qianlix@xjtu.edu.cn" ]
  [ "王骞男，出生于1980年，湖北武汉人.博士，武汉大学教授、博士生导师.主要研究领域为人工智能安全、云计算安全与隐私、无线系统安全、应用密码学.E-mail: qianwang@whu.edu.cn" ]
  [ "李琦男，出生于1979年，浙江临安人.博士，清华大学副教授、博士生导师.主要研究领域为互联网和云安全、移动安全、机器学习与安全、大数据安全、区块链与安全.E-mail: qli01@tsinghua.edu.cn" ]
  [ "管晓宏男，出生于1955年，四川泸州人.博士，西安交通大学教授、博士生导师，中国科学院院士.主要研究领域为网络信息安全、网络化系统、电力系统优化调度.E-mail: xhguan@xjtu.edu.cn" ]
- 基金信息：
  
  科技创新2030——“新一代人工智能”重大项目(2020AAA0107702);国家自然科学基金(62161160337;U21B2018;U20A20177;62132011;62006181;U20B2049);陕西重点研发计划项目(2021ZD LGY01-02)
- DOI：10.12263/DZXB.20220821
  中图分类号： TP391;
- 收稿：2022-07-14，
  
  修回：2022-10-20，
  
  纸质出版：2022-12-25
- 稿件说明：
移动端阅览
张笑宇,沈超,蔺琛皓等.面向机器学习模型安全的测试与修复[J].电子学报,2022,50(12):2884-2918.

ZHANG Xiao-yu,SHEN Chao,LIN Chen-hao,et al.The Testing and Repairing Methods for Machine Learning Model Security[J].ACTA ELECTRONICA SINICA,2022,50(12):2884-2918.
张笑宇,沈超,蔺琛皓等.面向机器学习模型安全的测试与修复[J].电子学报,2022,50(12):2884-2918. DOI： 10.12263/DZXB.20220821.

ZHANG Xiao-yu,SHEN Chao,LIN Chen-hao,et al.The Testing and Repairing Methods for Machine Learning Model Security[J].ACTA ELECTRONICA SINICA,2022,50(12):2884-2918. DOI： 10.12263/DZXB.20220821.

摘要

近年来，以机器学习算法为代表的人工智能技术在计算机视觉、自然语言处理、语音识别等领域取得了广泛的应用，各式各样的机器学习模型为人们的生活带来了巨大的便利.机器学习模型的工作流程可以分为三个阶段.首先，模型接收人工收集或算法生成的原始数据作为输入，并通过预处理算法（如数据增强和特征提取）对数据进行预处理.随后，模型定义神经元或层的架构，并通过运算符（例如卷积和池）构建计算图.最后，模型调用机器学习框架的函数功能实现计算图并执行计算，根据模型神经元的权重计算输入数据的预测结果.在这个过程中，模型中单个神经元输出的轻微波动可能会导致完全不同的模型输出，从而带来巨大的安全风险.然而，由于对机器学习模型的固有脆弱性及其黑箱特征行为的理解不足，研究人员很难提前识别或定位这些潜在的安全风险，这为个人生命财产安全乃至国家安全带来了诸多风险和隐患.研究机器学习模型安全的相关测试与修复方法，对深刻理解模型内部风险与脆弱性、全面保障机器学习系统安全性以及促进人工智能技术的广泛应用有着重要意义.本文从不同安全测试属性出发，详细介绍了现有的机器学习模型安全测试和修复技术，总结和分析了现有研究中的不足，探讨针对机器学习模型安全的测试与修复的技术进展和未来挑战，为模型的安全应用提供了指导和参考.本文首先介绍了机器学习模型的结构组成和主要安全测试属性，随后从机器学习模型的三个组成部分即数据、算法和实现，六种模型安全相关测试属性即正确性、鲁棒性、公平性、效率、可解释性和隐私性，分析、归纳和总结了相关的测试与修复方法及技术，并探讨了现有方法的局限.最后本文讨论和展望了机器学习模型安全的测试与修复方法的主要技术挑战和发展趋势.

Abstract

In recent years

artificial intelligence technology led by machine learning algorithms has been widely used in many fields

such as computer vision

natural language processing

speech recognition

etc. A variety of machine learning models have greatly facilitated people's lives. The workflow of a machine learning model consists of three stages. First

the model receives the raw data which is collected or generated by the developers as the model input and preprocesses the data through preprocessing algorithms

such as data augmentation and feature extraction. Subsequently

the model defines the architecture of neurons or layers in the model and constructs a computational graph through operators(e.g.

convolution and pooling). Finally

the model calls the machine learning framework function to implement the operators and calculates the prediction result of the input data according to the weights of model neurons. In this process

slight fluctuations in the output of individual neurons in the model may lead to an entirely different model output

which can bring huge security risks. However

due to the insufficient understanding of the inherent vulnerability of machine learning models and their black box characteristic behaviors

it is difficult for researchers to identify or locate these potential security risks in advance. This brings many risks and hidden dangers to personal property safety and even national security. There is great significance to studying the testing and repairing methods for machine learning model security

which can help deeply understand the internal risks and vulnerabilities of models

comprehensively guarantee the security of machine learning systems

and widely apply artificial intelligence technology. The existing testing research for the machine learning model security has mainly focused on the correctness

robustness

and other testing properties of the model

and this research has achieved certain results. This paper intends to start from different security testing attributes

introduces the existing machine learning model security testing and repair technology in detail

summarizes and analyzes the deficiencies in the existing research

and discusses the technical progress and challenges of machine learning model security testing and repairing

providing guidance and reference for the safe application of the model. In this paper

we first introduce the structural composition and main testing properties of the machine learning model security. Afterwards

we systematically summarize and analyze the existing work from the three components of the machine learning model—data

algorithm

and implementation

and six model security-related testing properties-correctness

robustness

fairness

efficiency

interpretability

and privacy. We also discuss the effectiveness and limitations of the existing testing and repairing methods. Finally

we discuss several technical challenges and potential development directions of the testing and repairing methods for machine learning model security in the future.

关键词

Keywords

references

WORTSMAN M , ILHARCO G , GADRE S Y , et al . Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time [EB/OL ] . ( 2022-03-10 )[ 2022-07 ] . https://arxiv.org/abs/2203.05482 https://arxiv.org/abs/2203.05482 .

BAO H B , DONG L , PIAO S H , et al . BEiT: BERT pre-training of image transformers [EB/OL ] . ( 2021-06-15 )[ 2022-07 ] . https://arxiv.org/abs/2106.08254 https://arxiv.org/abs/2106.08254 .

TAN M X , LE Q . Efficientnet: Rethinking model scaling for convolutional neural networks [C ] // International Conference on Machine Learning . New Orleans : PMLR.org , 2019 : 6105 - 6114 .

BROWN T B , MANN B , RYDER N , et al . Language models are few-shot learners [C ] // 34th International Conference on Neural Information Processing Systems . Vancouver : Curran Associates Inc. , 2020 : 1877 - 1901 .

MELIS G , KOČISKÝ T , BLUNSOM P . Mogrifier LSTM [EB/OL ] . ( 2019-09-04 ) [ 2022-07 ] . https://arxiv.org/abs/1909.01792 https://arxiv.org/abs/1909.01792 .

YAMADA I , ASAI A , SHINDO H , et al . LUKE: Deep contextualized entity representations with entity-aware self-attention [EB/OL ] . ( 2020-10-02 )[ 2022-07 ] . https://arxiv.org/abs/2010.01057 https://arxiv.org/abs/2010.01057 .

KOLOBOV R , OKHAPKINA O , OMELCHISHINA O , et al . MediaSpeech: Multilanguage ASR benchmark and dataset [EB/OL ] . ( 2021-03-30 )[ 2022-07 ] . https://arxiv.org/abs/2103.16193 https://arxiv.org/abs/2103.16193 .

PARK D S , ZHANG Y , JIA Y , et al . Improved noisy student training for automatic speech recognition [EB/OL ] . ( 2020-05-19 )[ 2022-07 ] . https://arxiv.org/abs/2005.09629 https://arxiv.org/abs/2005.09629 .

XU Q T , BAEVSKI A , LIKHOMANENKO T , et al . Self-training and pre-training are complementary for speech recognition [C ] // 2021 IEEE International Conference on Acoustics, Speech and Signal Processing . Toronto : IEEE , 2021 : 3030 - 3034 .

JHA D , RIEGLER M A , JOHANSEN D , et al . DoubleU-net: A deep convolutional neural network for medical image segmentation [C ] // 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems . Rochester : IEEE , 2020 : 558 - 564 .

SRIVASTAVA A , JHA D , CHANDA S , et al . MSRF-net: A multi-scale residual fusion network for biomedical image segmentation [EB/OL ] . ( 2021-05-16 )[ 2022-07 ] . https://arxiv.org/abs/2105.07451 https://arxiv.org/abs/2105.07451 .

WANG J F , HUANG Q M , TANG F L , et al . Stepwise feature fusion: Local guides global [EB/OL ] . ( 2022-03-07 )[ 2022-07 ] . https://arxiv.org/abs/2203.03635 https://arxiv.org/abs/2203.03635 .

STOICA I , SONG D , POPA R A , et al . A Berkeley view of systems challenges for AI [EB/OL ] . ( 2017-12-15 )[ 2022-07 ] . https://arxiv.org/abs/1712.05855 https://arxiv.org/abs/1712.05855 .

Research and Market . Edge AI Market – Forecasts from 2021 to 2026 [EB/OL ] . ( 2021-03 )[ 2022-07 ] . https://www.researchandmarkets.com/reports/5308992/edge-ai-market-forecasts-from-2021-to-2026 https://www.researchandmarkets.com/reports/5308992/edge-ai-market-forecasts-from-2021-to-2026 .

ABADI M . TensorFlow: Learning functions at scale [J ] . ACM SIGPLAN Notices , 2016 , 51 ( 9 ): 1 .

马艳军 , 于佃海 , 吴甜 , 等 , 飞桨 : 源于产业实践的开源深度学习平台 [J ] . 数据与计算发展前沿 , 2019 , 1 ( 1 ): 105 - 115 .

MA Y , YU D , WU T , et al . PaddlePaddle: An open-source deep learning platform from industrial practice [J ] . Frontiers of Data and Domputing , 2019 , 1 ( 1 ): 105 - 115 . (in Chinese)

CHEN T Q , LI M , LI Y T , et al . MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems [EB/OL ] . ( 2015-12-03 )[ 2022-07 ] . https://arxiv.org/abs/1512.01274 https://arxiv.org/abs/1512.01274 .

PASZKE A , GROSS S , MASSA F , et al . Pytorch: An imperative style, high-performance deep learning library [C ] // 33rd International Conference on Neural Information Processing Systems . Vancouver : Curran Associates, Inc. , 2019 : 8026 - 8037 .

Google . AI and machine learning products [EB/OL ] . ( 2022 )[ 2022-07-11 ] . https://cloud.google.com/products/ai/ https://cloud.google.com/products/ai/ .

Baidu . Baidu Al open platform [EB/OL ] . ( 2021 )[ 2022-07-11 ] . http://ai.baidu.com/ http://ai.baidu.com/ .

JULIA A , JEFF L , SURYA M , et al . Machine Bias [R/OL ] . ( 2016-05-23 )[ 2022-07-11 ] . https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing .

WAKABAYASHI D . Self-driving uber car kills pedestrian in Arizona,where robots roam [EB/OL ] . ( 2018-03-19 )[ 2022-07-11 ] . https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html .

ELSOM J . Moment an Amazon Alexa tells a terrified mother, 29 , to “stab yourself in the heart for the greater good” while reading from rogue Wikipedia text [EB/OL ] . ( 2019-12-19 )[ 2022-07-11 ] . https://www.dailymail.co.uk/news/article-7809269/Amazon-Alexa-told-terrified-mother-29-stab-heart-greater-good.html https://www.dailymail.co.uk/news/article-7809269/Amazon-Alexa-told-terrified-mother-29-stab-heart-greater-good.html .

XIE X F , MA L , JUEFEI-XU F , et al . DeepHunter: A coverage-guided fuzz testing framework for deep neural networks [C ] // Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis . Beijing : ACM , 2019 : 146 - 157 .

ZHANG X Y , ZHAI J , MA S Q , et al . AUTOTRAINER: An automatic DNN training problem detection and repair system [C ] // 2021 IEEE/ACM 43rd International Conference on Software Engineering . Madrid : IEEE , 2021 : 359 - 371 .

ODENA A , OLSSON C , ANDERSEN D , et al . Tensorfuzz: Debugging neural networks with coverage-guided fuzzing [C ] // Proceedings of the 36th International Conference on Machine Learning . Virtual Conference : PMLR.org , 2019 : 4901 - 4911 .

GAO X Q , ZHAI J , MA S Q , et al . FairNeuron: Improving deep neural network fairness with adversary games on selective neurons [EB/OL ] . ( 2022-04-06 )[ 2022-07-11 ] . https://arxiv.org/abs/2204.02567 https://arxiv.org/abs/2204.02567 .

中华人民共和国工业和信息化部 . 工业和信息化部关于印发《促进新一代人工智能产业发展三年行动计划(2018—2020年)》的通知 [EB/OL ] . ( 2017-12-13 )[ 2022-07-11 ] . https://www.miit.gov.cn/jgsj/kjs/wjfb/art/2020/art_08d153ee9e9d4676aa69d0aa12676ca1.html https://www.miit.gov.cn/jgsj/kjs/wjfb/art/2020/art_08d153ee9e9d4676aa69d0aa12676ca1.html .

The White House Office Of Science And Technology Policy . American AI Initiative One Year Annua Report [R/OL ] . 2020 . https://www.nitrd.gov/nitrdgroups/images/c/c1/American-AI-Initiative-One-Year-Annual-Report.pdf https://www.nitrd.gov/nitrdgroups/images/c/c1/American-AI-Initiative-One-Year-Annual-Report.pdf .

纪守领 , 杜天宇 , 李进锋 , 等 . 机器学习模型安全与隐私研究综述 [J ] . 软件学报 , 2021 , 32 ( 1 ): 41 - 67 .

JI S L , DU T Y , LI J F , et al . Security and privacy of machine learning models: A survey [J ] . Journal of Software , 2021 , 32 ( 1 ): 41 - 67 . (in Chinese)

HUANG X W , KROENING D , RUAN W J , et al . A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability [J ] . Computer Science Review , 2020 , 37 : 100270 .

ZHANG J M , HARMAN M , MA L , et al . Machine learning testing: Survey, landscapes and horizons [J ] . IEEE Transactions on Software Engineering , 2022 , 48 ( 1 ): 1 - 36 .

BRAIEK H B , KHOMH F . On testing machine learning programs [J ] . Journal of Systems and Software , 2020 , 164 : 110542 .

MEHRABI N , MORSTATTER F , SAXENA N , et al . A survey on bias and fairness in machine learning [J ] . ACM Computing Surveys , 2021 , 54 ( 6 ): 1 - 35 .

AMERSHI S , BEGEL A , BIRD C , et al . Software engineering for machine learning: A case study [C ] // 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice . Montreal : IEEE , 2019 : 291 - 300 .

JESMEEN M Z H , HOSSEN J , SAYEED S , et al . A survey on cleaning dirty data using machine learning paradigm for big data analytics [J ] . Indonesian Journal of Electrical Engineering and Computer Science , 2018 , 10 ( 3 ): 1234 - 1243 .

KHALID S , KHALIL T , NASREEN S . A survey of feature selection and feature extraction techniques in machine learning [C ] // 2014 Science and Information Conference . London : IEEE , 2014 : 372 - 378 .

ROH Y , HEO G , WHANG S E . A survey on data collection for machine learning: A big data - AI integration perspective [J ] . IEEE Transactions on Knowledge and Data Engineering , 2021 , 33 ( 4 ): 1328 - 1347 .

REFAEILZADEH P , TANG L , LIU H . Cross-validation [M ] // Encyclopedia of Database Systems . Boston : Springer , 2009 : 532 - 538 .

SHAHROKNI A , FELDT R . A systematic review of software robustness [J ] . Information and Software Technology , 2013 , 55 ( 1 ): 1 - 17 .

IEEE . IEEE Standard Glossary of Software Engineering Terminology [A/OL ] . ( 1990-12-31 ) [ 2022-07-11 ] . https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=159342 https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=159342 .

纪守领 , 杜天宇 , 邓水光 , 等 . 深度学习模型鲁棒性研究综述 [J ] . 计算机学报 , 2022 , 45 ( 1 ): 190 - 206 .

JI S L , DU T Y , DENG S G , et al . Robustness certification research on deep learning models: A survey [J ] . Chinese Journal of Computers , 2022 , 45 ( 1 ): 190 - 206 . (in Chinese)

GAJANE P , PECHENIZKIY M . On formalizing fairness in prediction with machine learning [EB/OL ] . ( 2017-10-09 )[ 2022-07-11 ] . https://arxiv.org/abs/1710.03184 https://arxiv.org/abs/1710.03184 .

HARDT M , PRICE E , SREBRO N . Equality of opportunity in supervised learning [J ] . Advances in Neural Information Processing Systems. Barcelona: Curran Associates Inc. , 2016 : 29 .

ZAFAR M B , VALERA I , ROGRIGUEZ M G , et al . Fairness constraints: Mechanisms for fair classification [C ] // Proceedings of the 20th International Conference on Artificial Intelligence and Statistics . Fort Laud-erdale : PMLR , 2017 : 962 - 970 .

KUSNER M J , LOFTUS J , RUSSELL C , et al . Counterfactual fairness [J ] . Advances in Neural Information Processing Systems. Long Beach: Curran Associates Inc. , 2017 : 30 .

DWORK C , HARDT M , PITASSI T , et al . Fairness through awareness [C ] // Proceedings of the 3rd Innovations in Theoretical Computer Science Conference . Beijing : ACM , 2012 : 214 - 226 .

GUO Q Y , CHEN S , XIE X F , et al . An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms [C ] // 34th IEEE/ACM International Conference on Automated Software Engineering . San Diego : IEEE , 2019 : 810 - 822 .

GOODMAN B , FLAXMAN S . European union regulations on algorithmic decision-making and a “right to explanation” [J ] . AI Magazine , 2017 , 38 ( 3 ): 50 - 57 .

DWORK C . Differential privacy: A survey of results [C ] // International Conference on Theory and Applications of Models of Computation . Berlin : Springer , 2008 : 1 - 19 .

GUO Q Y , XIE X F , LI Y , et al . Audee: Automated testing for deep learning frameworks [C ] // 35th IEEE/ACM International Conference on Automated Software Engineering . Virtual Conference : ACM , 2020 : 486 - 498 .

PHAM H V , LUTELLIER T , QI W Z , et al . CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries [C ] // 2019 IEEE/ACM 41st International Conference on Software Engineering . Montreal : IEEE , 2019 : 1027 - 1038 .

WANG J N , LUTELLIER T , QIAN S S , et al . EAGLE: Creating equivalent graphs to test deep learning libraries [C ] // 2022 IEEE/ACM 44th International Conference on Software Engineering . Pittsburgh : IEEE , 2022 : 798 - 810 .

ZHANG X F , SUN N , FANG C R , et al . Predoo: precision testing of deep learning operators [C ] // Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis . Virtual Conference : ACM , 2021 : 400 - 412 .

SANTOS S H N , SILVEIRA B N C DA , ANDRADE S A , et al . An experimental study on applying metamorphic testing in machine learning applications [C ] // Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing . Natal : ACM , 2020 : 98 - 106 .

XIAO D W , LIU Z B , YUAN Y Y , et al . Metamorphic testing of deep learning compilers [J ] . Proceedings of the ACM on Measurement and Analysis of Computing Systems , 2022 , 6 ( 1 ): 1 - 28 .

GOODFELLOW I J , SHLENS J , SZEGEDY C . Explaining and harnessing adversarial examples [EB/OL ] . ( 2014-12-20 )[ 2022-07-11 ] . https://arxiv.org/abs/1412.6572 https://arxiv.org/abs/1412.6572 .

PAPERNOT N , FAGHRI F , CARLINI N , et al . Technical report on the CleverHans v2.1.0 adversarial examples library [EB/OL ] . ( 2016-10-03 )[ 2022-07-11 ] . https://arxiv.org/abs/1610.00768 https://arxiv.org/abs/1610.00768 .

XIE C H , WANG J Y , ZHANG Z S , et al . Mitigating adversarial effects through randomization [EB/OL ] . ( 2017-11-06 )[ 2022-07-11 ] . https://arxiv.org/abs/1711.01991 https://arxiv.org/abs/1711.01991 .

KOLBEINSSON A , KOSSAIFI J , PANAGAKIS Y , et al . Tensor dropout for robust learning [J ] . IEEE Journal of Selected Topics in Signal Processing , 2021 , 15 ( 3 ): 630 - 640 .

XU W L , EVANS D , QI Y J . Feature squeezing: Detecting adversarial examples in deep neural networks [EB/OL ] . ( 2017-04-04 )[ 2022-07-11 ] . https://arxiv.org/abs/1704.01155 https://arxiv.org/abs/1704.01155 .

XU W L , EVANS D , QI Y J . Feature squeezing mitigates and detects carlini/Wagner adversarial examples [EB/OL ] . ( 2017-05-30 )[ 2022-07-11 ] . https://arxiv.org/abs/1705.10686 https://arxiv.org/abs/1705.10686 .

WANG J Y , DONG G L , SUN J , et al . Adversarial sample detection for deep neural network through model mutation testing [C ] // 2019 IEEE/ACM 41st International Conference on Software Engineering . Montreal : IEEE , 2019 : 1245 - 1256 .

ZHAO Z , CHEN G K , WANG J Y , et al . Attack as defense: Characterizing adversarial examples using robustness [C ] // Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis . Virtual Conference : ACM . 2021 : 42 - 55 .

NGUYEN G H , BOUZERDOUM A , PHUNG S L . A supervised learning approach for imbalanced data sets [C ] // 2008 19th International Conference on Pattern Recognition . Tampa : IEEE , 2008 : 1 - 4 .

AMINI A , SOLEIMANY A P , SCHWARTING W , et al . Uncovering and mitigating algorithmic bias through learned latent structure [C ] // Proceedings of the 2019 AAAI/ACM Conference on AI , Ethics, and Society . Honolulu : ACM , 2019 : 289 - 295 .

MULLICK S S , DATTA S , DHEKANE S G , et al . Appropriateness of performance indices for imbalanced data classification: An analysis [J ] . Pattern Recognition , 2020 , 102 : 107197 .

KAMIRAN F , CALDERS T . Classifying without discriminating [C ] // 2009 2nd International Conference on Computer, Control and Communication . Karachi : IEEE , 2009 : 1 - 6 .

AMINI A , SCHWARTING W , ROSMAN G , et al . Variational autoencoder for end-to-end control of autonomous driving with novelty detection and training de-biasing [C ] // 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems . Madrid : IEEE , 2018 : 568 - 575 .

TOMALIN M , BYRNE B , CONCANNON S , et al . The practical ethics of bias reduction in machine translation: Why domain adaptation is better than data debiasing [J ] . Ethics and Information Technology , 2021 , 23 ( 3 ): 419 - 433 .

HOLLAND S , HOSNY A , NEWMAN S , et al . The dataset nutrition label: A framework to drive higher data quality standards [EB/OL ] . ( 2018-05-09 )[ 2022-07-11 ] . https://arxiv.org/abs/1805.03677 https://arxiv.org/abs/1805.03677 .

HYNES N , SCULLEY D , TERRY M . The data linter: Lightweight automated sanity checking for ML data sets [C ] // NIPS MLSys Workshop . Cambridge : MIT Press , 2017 : 1 .

KRISHNAN S , WU E . AlphaClean: Automatic generation of data cleaning pipelines [EB/OL ] . ( 2019-04-26 )[ 2022-07-11 ] . https://arxiv.org/abs/1904.11827 https://arxiv.org/abs/1904.11827 .

LAISHRAM R , PHOHA V V . Curie: A method for protecting SVM Classifier from Poisoning Attack [EB/OL ] . ( 2016-06-05 )[ 2022-07-11 ] . https://arxiv.org/abs/1606.01584 https://arxiv.org/abs/1606.01584 .

ZHANG W N , WANG D , TAN X Y . Robust class-specific autoencoder for data cleaning and classification in the presence of label noise [J ] . Neural Processing Letters , 2019 , 50 ( 2 ): 1845 - 1860 .

STEINHARDT J , KOH P W , LIANG P . Certified defenses for data poisoning attacks [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems . Long Beach : Curran Associates Inc. , 2017 : 3520 - 3532 .

SHOKRI R , STRONATI M , SONG C Z , et al . Membership inference attacks against machine learning models [C ] // 2017 IEEE Symposium on Security and Privacy . San Jose : IEEE , 2017 : 3 - 18 .

PAPERNOT N , ABADI M , ERLINGSSON Ú , et al . Semi-supervised knowledge transfer for deep learning from private training data [EB/OL ] . ( 2016-10-18 )[ 2022-07-11 ] . https://arxiv.org/abs/1610.05755 https://arxiv.org/abs/1610.05755 .

HUANG K , LIU X M , FU S J , et al . A lightweight privacy-preserving CNN feature extraction framework for mobile sensing [J ] . IEEE Transactions on Dependable and Secure Computing , 2021 , 18 ( 3 ): 1441 - 1455 .

BONAWITZ K , IVANOV V , KREUTER B , et al . Practical secure aggregation for privacy-preserving machine learning [C ] // Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . Dallas : ACM , 2017 : 1175 - 1191 .

SZEGEDY C , ZAREMBA W , SUTSKEVER I , et al . Intriguing properties of neural networks [EB/OL ] . ( 2013-12-31 )[ 2022-07-11 ] . https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199 .

CARLINI N , WAGNER D . Towards evaluating the robustness of neural networks [C ] // 2017 IEEE Symposium on Security and Privacy . San Jose : IEEE , 2017 : 39 - 57 .

MOOSAVI-DEZFOOLI S M , FAWZI A , FROSSARD P . DeepFool: A simple and accurate method to fool deep neural networks [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE , 2016 : 2574 - 2582 .

GOPINATH D , KATZ G , PASAREANU C S , et al . DeepSafe: A data-driven approach for assessing robustness of neural networks [C ] // International Symposium on Automated Technology for Verification and Analysis . Los Angeles : Springer , 2018 : 3 - 19 .

SHEN M , YU H , ZHU L H , et al . Effective and robust physical-world attacks on deep learning face recognition systems [J ] . IEEE Transactions on Information Forensics and Security , 2021 , 16 : 4063 - 4077 .

HAN S C , LIN C H , SHEN C , et al . Rethinking adversarial examples exploiting frequency-based analysis [C ] // International Conference on Information and Communications Security . Chongqing : Springer , 2021 : 73 - 89 .

MU J M , WANG B H , LI Q , et al . A hard label black-box adversarial attack against graph neural networks [C ] // Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security . Virtual Conference : ACM , 2021 : 108 - 125 .

MAHMOOD K , MAHMOOD R , VAN DIJK M . On the robustness of vision transformers to adversarial examples [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE , 2021 : 7818 - 7827 .

BALUJA S , FISCHER I . Learning to attack: Adversarial transformation networks [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Lousiana : AAAI Press , 2018 , 32 ( 1 ): 2687 - 2695 .

CARLINI N , WAGNER D . Audio adversarial examples: Targeted attacks on speech-to-text [C ] // 2018 IEEE Security and Privacy Workshops . San Francisco : IEEE , 2018 : 1 - 7 .

CISSE M , ADI Y , NEVEROVA N , et al . Houdini: Fooling deep structured prediction models [EB/OL ] . ( 2017-07-17 )[ 2022-07-11 ] . https://arxiv.org/abs/1707.05373 https://arxiv.org/abs/1707.05373 .

ZHENG B L , JIANG P P , WANG Q , et al . Black-box adversarial attacks on commercial speech platforms with minimal information [C ] // Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security . Virtual Conference : ACM , 2021 : 86 - 107 .

BROWN T B , MANÉ D , ROY A , et al . Adversarial patch [EB/OL ] . ( 2017-12-27 )[ 2022-07-11 ] . https://arxiv.org/abs/ 1712.09665 https://arxiv.org/abs/1712.09665 .

GOODFELLOW I J , PAPERNOT N , MCDANIEL P . Cleverhans V 0 . 1 : An adversarial machine learning library[EB/OL ] . ( 2016-10-03 )[ 2022-07-11 ] . https://arxiv.org/abs/ https://arxiv.org/abs/

1610 . 00768 v 1 .

RAUBER J , BRENDEL W , BETHGE M . Foolbox: A Python toolbox to benchmark the robustness of machine learning models [EB/OL ] . ( 2017-07-13 )[ 2022-07-11 ] . https://arxiv.org/abs/1707.04131 https://arxiv.org/abs/1707.04131 .

NICOLAE M I , SINN M , TRAN M N , et al . Adversarial robustness toolbox v 1 . 0 . 0 [EB/OL ] . ( 2018-07-03 )[ 2022-07-11 ] . https://arxiv.org/abs/1807.01069 https://arxiv.org/abs/1807.01069 .

任奎 , ZHENG Tianhang , 秦湛 , 等 . 深度学习中的对抗性攻击和防御 [J ] . Engineering , 2020 , 6 ( 3 ): 307 - 339 .

REN K , ZHEBG T , QIN Z , et al . Adversarial attacks and defenses in deep learning [J ] . Engineering , 2020 , 6 ( 3 ): 307 - 339 . (in Chinese)

LIU X Q , CHENG M H , ZHANG H , et al . Towards robust neural networks via random self-ensemble [C ] // European Conference on Computer Vision . Munich : Springer , 2018 : 381 - 397 .

GUO C , RANA M , CISSE M , et al . Countering adversarial images using input transformations [EB/OL ] . ( 2017-10-31 )[ 2022-07-11 ] . https://arxiv.org/abs/1711.00117 https://arxiv.org/abs/1711.00117 .

LUO T G , CAI T L , ZHANG M X , et al . RANDOM MASK: Towards robust convolutional neural networks [EB/OL ] . ( 2020-07-27 )[ 2022-07-11 ] . https://arxiv.org/abs/2007.14249 https://arxiv.org/abs/2007.14249 .

SHARMA Y , CHEN P Y . Bypassing feature squeezing by increasing adversary strength [EB/OL ] . ( 2018-03-27 )[ 2022-07-11 ] . https://arxiv.org/abs/1803.09868 https://arxiv.org/abs/1803.09868 .

SAMANGOUEI P , KABKAB M , CHELLAPPA R . Defense-GAN: Protecting classifiers against adversarial attacks using generative models [EB/OL ] . ( 2018-03-17 )[ 2022-07-11 ] . https://arxiv.org/abs/1805.06605 https://arxiv.org/abs/1805.06605 .

LIAO F Z , LIANG M , DONG Y P , et al . Defense against adversarial attacks using high-level representation guided denoiser [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 1778 - 1787 .

SHEN S W , JIN G Q , GAO K , et al . APE-GAN: Adversarial perturbation elimination with GAN [EB/OL ] . ( 2017-07-18 )[ 2022-07-11 ] . https://arxiv.org/abs/1707.05474 https://arxiv.org/abs/1707.05474 .

YANG R , CHEN X Q , CAO T J . APE-GAN++: An improved APE-GAN to eliminate adversarial perturbations [J ] . IAENG International Journal of Computer Science , 2021 , 48 ( 3 ): 827 - 844 .

KHERCHOUCHE A , FEZZA S A , HAMIDOUCHE W . Detect and defense against adversarial examples in deep learning using natural scene statistics and adaptive denoising [J ] . Neural Computing and Applications , 2022 , 34 ( 24 ): 21567 - 21582 .

ESMAEILPOUR M , CARDINAL P , KOERICH A L . Class-conditional defense GAN against end-to-end speech attacks [C ] // ICASSP 2021 – 2021 IEEE International Conference on Acoustics, Speech and Signal Processing . Toronto : IEEE , 2021 : 2565 - 2569 .

METZEN J H , GENEWEIN T , FISCHER V , et al . On detecting adversarial perturbations [EB/OL ] . ( 2017-02-14 )[ 2022-07-11 ] . https://arxiv.org/abs/1702.04267 https://arxiv.org/abs/1702.04267 .

CARLINI N , WAGNER D . Adversarial examples are not easily detected: Bypassing ten detection methods [C ] // Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security . Dallas : ACM , 2017 : 3 - 14 .

BRECK E , POLYZOTIS N , ROY S , et al . Data validation for machine learning [C ] // Proceedings of Machine Learning and Systems . Stanford : mlsys.org , 2019 : 334 - 347 .

GEBRU T , MORGENSTERN J , VECCHIONE B , et al . Datasheets for datasets [J ] . Communications of the ACM , 2021 , 64 ( 12 ): 86 - 92 .

BENDER E M , FRIEDMAN B . Data statements for natural language processing: Toward mitigating system bias and enabling better science [J ] . Transactions of the Association for Computational Linguistics , 2018 , 6 : 587 - 604 .

CHAKRABORTY J , XIA T P , FAHID F M , et al . Software engineering for fairness: A case study with hyperparameter optimization [EB/OL ] . ( 2019-05-14 )[ 2022-07-11 ] . https://arxiv.org/abs/1905.05786 https://arxiv.org/abs/1905.05786 .

KAMIRAN F , CALDERS T . Data preprocessing techniques for classification without discrimination [J ] . Knowledge and Information Systems , 2012 , 33 ( 1 ): 1 - 33 .

SATTIGERI P , HOFFMAN S C , CHENTHAMARAKSHAN V , et al . Fairness GAN [EB/OL ] . ( 2018-05-24 )[ 2022-07-11 ] . https://arxiv.org/abs/1805.09910 https://arxiv.org/abs/1805.09910 .

AÏVODJI U , BIDET F , GAMBS S , et al . Local data debiasing for fairness based on generative adversarial training [J ] . Algorithms , 2021 , 14 ( 3 ): 87 .

JALAL A , KARMALKAR S , HOFFMANN J , et al . Fairness for image generation with uncertain sensitive attributes [C ] // Proceedings of the 38th International Conference on Machine Learning . Virtual Conference : PMLR , 2021 : 4721 - 4732 .

KRISHNAN S , WANG J N , WU E , et al . ActiveClean: Interactive data cleaning for statistical modeling [J ] . Proceedings of the VLDB Endowment , 2016 , 9 ( 12 ): 948 - 959 .

KRISHNAN S , FRANKLIN M J , GOLDBERG K , et al . BoostClean: automated error detection and repair for machine learning [EB/OL ] . ( 2017-11-03 )[ 2022-07-11 ] . https://arxiv.org/abs/1711.01299 https://arxiv.org/abs/1711.01299 .

SONG J , HE Y Y . Auto-validate: Unsupervised data validation using data-domain patterns inferred from data lakes [C ] // Proceedings of the 2021 International Conference on Management of Data . Virtual Conference : ACM , 2021 : 1678 - 1691 .

RUBINSTEIN B I P , NELSON B , HUANG L , et al . ANTIDOTE: understanding and defending against poisoning of anomaly detectors [C ] // Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement . Chicago : ACM , 2009 : 1 - 14 .

RAHM E , DO H . Data cleaning: Problems and current approaches [J ] . IEEE Data Eng. Bull. , 2000 , 23 : 3 - 13 .

FREDRIKSON M , LANTZ E , JHA S , et al . Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing [C ] // Proceedings of the 23rd USENIX Security Symposium . Berkeley : USENIX Association , 2014 , 2014 : 17 - 32 .

HITAJ B , ATENIESE G , PEREZ-CRUZ F . Deep models under the GAN: Information leakage from collaborative deep learning [C ] // Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . Dallas : ACM , 2017 : 603 - 618 .

ATENIESE G , FELICI G , MANCINI L V , et al . Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers [EB/OL ] . ( 2013-06-19 )[ 2022-07-11 ] . https://arxiv.org/abs/1306.4447 https://arxiv.org/abs/1306.4447 .

ERLINGSSON Ú , PIHUR V , KOROLOVA A . RAPPOR: randomized aggregatable privacy-preserving ordinal response [C ] // Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security . Scottsdale : ACM , 2014 : 1054 - 1067 .

SALEM A , ZHANG Y , HUMBERT M , et al . ML-leaks: Model and data independent membership inference attacks and defenses on machine learning models [EB/OL ] . ( 2018-06-04 )[ 2022-07-11 ] . https://arxiv.org/abs/1806.01246 https://arxiv.org/abs/1806.01246 .

李强 , 颜浩 , 陈克非 . 安全多方计算协议的研究与应用 [J ] . 计算机科学 , 2003 , 30 ( 8 ): 52 - 55 .

LI Q , YAN H , CHEN K F . Research and application of secure multi-party computation protocols [J ] . Computer Science , 2003 , 30 ( 8 ): 52 - 55 . (in Chinese)

YAO A C . Protocols for secure computations [C ] // 23rd Annual Symposium on Foundations of Computer Science . Chicago : IEEE , 1982 : 160 - 164 .

GOLDREICH O , MICALI S , WIGDERSON A . How to play ANY mental game [C ] // Proceedings of the Nineteenth Annual ACM Symposium on Theory of Computing . New York : ACM , 1987 : 218 - 229 .

VAIDYA J , CLIFTON C . Privacy-preserving k-means clustering over vertically partitioned data [C ] // Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . Washington : ACM , 2003 : 206 - 215 .

MEHNAZ S , BELLALA G , BERTINO E . A secure sum protocol and its application to privacy-preserving multi-party analytics [C ] // Proceedings of the 22nd ACM on Symposium on Access Control Models and Technologies . Indianapolis : ACM , 2017 : 219 - 230 .

MOHASSEL P , ZHANG Y P . SecureML: A system for scalable privacy-preserving machine learning [C ] // 2017 IEEE Symposium on Security and Privacy . San Jose : IEEE , 2017 : 19 - 38 .

ROUHANI B D , RIAZI M S , KOUSHANFAR F . DeepSecure: scalable provably-secure deep learning [C ] // 55th ACM/ESDA/IEEE Design Automation Conference . San Francisco : IEEE , 2018 : 1 - 6 .

KONEČNÝ J , MCMAHAN H B , YU F X , et al . Federated learning: Strategies for improving communication efficiency [EB/OL ] . ( 2016-10-18 )[ 2022-07-11 ] . https://arxiv.org/abs/1610.05492 https://arxiv.org/abs/1610.05492 .

MCMAHAN H B , RAMAGE D , TALWAR K , et al . Learning differentially private recurrent language models [EB/OL ] . ( 2017-10-18 )[ 2022-07-11 ] . https://arxiv.org/abs/1710.06963 https://arxiv.org/abs/1710.06963 .

WENG J S , WENG J , ZHANG J L , et al . DeepChain: Auditable and privacy-preserving deep learning with blockchain-based incentive [J ] . IEEE Transactions on Dependable and Secure Computing , 2021 , 18 ( 5 ): 2438 - 2455 .

GOEL K , RAJANI N , VIG J , et al . Robustness gym: Unifying the NLP evaluation landscape [EB/OL ] . ( 2021-01-13 )[ 2022-07-11 ] . https://arxiv.org/abs/2101.04840 https://arxiv.org/abs/2101.04840 .

PAULI P , KOCH A , BERBERICH J , et al . Training robust neural networks using lipschitz bounds [J ] . IEEE Control Systems Letters , 2022 , 6 : 121 - 126 .

PEI K X , CAO Y Z , YANG J F , et al . DeepXplore: automated whitebox testing of deep learning systems [C ] // Proceedings of the 26th Symposium on Operating Systems Principles . Shanghai : ACM , 2017 : 1 - 18 .

GUO J M , JIANG Y , ZHAO Y , et al . DLFuzz: Differential fuzzing testing of deep learning systems [C ] // Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Lake Buena Vista : ACM , 2018 : 739 - 743 .

MURPHY C , KAISER G , HU L F , et al . Properties of machine learning applications for use in metamorphic testing [C ] // Proceedings of the Twentieth International Conference on Software Engineering & Knowledge Engineering . San Francisco : Knowledge Systems Institute Graduate School , 2008 : 867 - 872 .

XIE X Y , ZHANG Z Y , CHEN T Y , et al . METTLE: A METamorphic testing approach to assessing and validating unsupervised machine learning systems [J ] . IEEE Transactions on Reliability , 2020 , 69 ( 4 ): 1293 - 1322 .

JIANG M Y , CHEN T Y , WANG S . On the effectiveness of testing sentiment analysis systems with metamorphic testing [J ] . Information and Software Technology , 2022 , 150 : 106966 .

MA S Q , LIU Y Q , LEE W C , et al . MODE: Automated neural network model debugging via state differential analysis and input selection [C ] // Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Lake Buena Vista : ACM , 2018 : 175 - 186 .

YU B , QI H , GUO Q , et al . DeepRepair: Style-guided repairing for deep neural networks in the real-world operational environment [J ] . IEEE Transactions on Reliability , 2022 , 71 ( 4 ): 1401 - 1416 .

SUN Z Y , ZHANG J M , HARMAN M , et al . Automatic testing and improvement of machine translation [C ] // Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . Seoul : ACM , 2020 : 974 - 985 .

WARDAT M , LE W , RAJAN H . DeepLocalize: Fault localization for deep neural networks [C ] // 2021 IEEE/ACM 43rd International Conference on Software Engineering . Madrid : IEEE , 2021 : 251 - 262 .

TRAMÈR F , ATLIDAKIS V , GEAMBASU R , et al . FairTest: Discovering unwarranted associations in data-driven applications [C ] // 2017 IEEE European Symposium on Security and Privacy . Paris : IEEE , 2017 : 401 - 416 .

ANGELL R , JOHNSON B , BRUN Y , et al . Themis: automatically testing software for discrimination [C ] // Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Lake Buena Vista : ACM , 2018 : 871 - 875 .

UDESHI S , ARORA P , CHATTOPADHYAY S . Automated directed fairness testing [C ] // Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering . Montpellier : ACM , 2018 : 98 - 108 .

BLACK E , YEOM S , FREDRIKSON M . FlipTest: Fairness testing via optimal transport [C ] // Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . Barcelona : ACM , 2020 : 111 - 121 .

KAMIRAN F , MANSHA S , KARIM A , et al . Exploiting reject option in classification for social discrimination control [J ] . Information Sciences , 2018 , 425 : 18 - 33 .

YANG Z , JAIN H , SHI J K , et al . BiasHeal: On-the-fly black-box healing of bias in sentiment analysis systems [C ] // 2021 IEEE International Conference on Software Maintenance and Evolution . Luxembourg : IEEE , 2021 : 644 - 648 .

SLACK D , FRIEDLER S A , SCHEIDEGGER C , et al . Assessing the local interpretability of machine learning models [EB/OL ] . ( 2019-02-09 )[ 2022-07-11 ] . https://arxiv.org/abs/1902.03501 https://arxiv.org/abs/1902.03501 .

ZHOU Z Q , SUN L Q , CHEN T Y , et al . Metamorphic relations for enhancing system understanding and use [J ] . IEEE Transactions on Software Engineering , 2020 , 46 ( 10 ): 1120 - 1154 .

MOLNAR C . Interpretable Machine Learning [M ] . Morrisville : Lulu Press , 2019 .

CHEN H J , JI Y F . Learning variational word masks to improve the interpretability of neural text classifiers [EB/OL ] . ( 2020-10-01 )[ 2022-07-11 ] . https://arxiv.org/abs/2010.00667 https://arxiv.org/abs/2010.00667 .

DING Z Y , WANG Y X , WANG G H , et al . Detecting violations of differential privacy [C ] // Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security . Toronto : ACM , 2018 : 475 - 489 .

CARLINI N , JAGIELSKI M , MIRONOV I . Cryptanalytic extraction of neural network models [C ] // Annual International Cryptology Conference . Santa Barbara : Springer , 2020 : 189 - 218 .

LIU J , JUUTI M , LU Y , et al . Oblivious neural network predictions via MiniONN transformations [C ] // Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . Dallas : ACM , 2017 : 619 - 631 .

RUAN W J , WU M , SUN Y C , et al . Global robustness evaluation of deep neural networks with provable guarantees for the L0 norm [EB/OL ] . ( 2018-04-16 )[ 2022-07-11 ] . https://arxiv.org/abs/1804.05805 https://arxiv.org/abs/1804.05805 .

MANGAL R , NORI A V , ORSO A . Robustness of neural networks: A probabilistic and practical approach [C ] // 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results . Montreal : IEEE , 2019 : 93 - 96 .

LORENZ T , RUOSS A , BALUNOVIĆ M , et al . Robustness certification for point cloud models [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE , 2021 : 7588 - 7598 .

BHOJANAPALLI S , CHAKRABARTI A , GLASNER D , et al . Understanding robustness of transformers for image classification [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE , 2021 : 10211 - 10221 .

MADRY A , MAKELOV A , SCHMIDT L , et al . Towards deep learning models resistant to adversarial attacks [EB/OL ] . ( 2017-06-19 )[ 2022-07-11 ] . https://arxiv.org/abs/1706.06083 https://arxiv.org/abs/1706.06083 .

CARLINI N , KATZ G , BARRETT C , et al . Provably minimally-distorted adversarial examples [EB/OL ] . ( 2017-09-29 )[ 2022-07-11 ] . https://arxiv.org/abs/1709.10207 https://arxiv.org/abs/1709.10207 .

KURAKIN A , GOODFELLOW I , BENGIO S . Adversarial examples in the physical world [EB/OL ] . ( 2016-07-08 )[ 2022-07-11 ] . https://arxiv.org/abs/1607.02533 https://arxiv.org/abs/1607.02533 .

LEE H , HAN S , LEE J . Generative adversarial trainer: Defense to adversarial perturbations with GAN [EB/OL ] . ( 2017-05-09 ) [ 2022-07-11 ] . https://arxiv.org/abs/1705.03387 https://arxiv.org/abs/1705.03387 .

WANG J Y , CHEN J L , SUN Y C , et al . RobOT: Robustness-oriented testing for deep learning systems [C ] // 2021 IEEE/ACM 43rd International Conference on Software Engineering . Madrid : IEEE , 2021 : 300 - 311 .

KIM J , FELDT R , YOO S . Guiding deep learning system testing using surprise adequacy [C ] // 2019 IEEE/ACM 41st International Conference on Software Engineering . Montrea : IEEE , 2019 : 1039 - 1049 .

XU H , CARAMANIS C , MANNOR S . Robustness and regularization of support vector machines [J ] . Journal of Machine Learning Research , 2008 , 10 : 1485 - 1510 .

DEMONTIS A , RUSSU P , BIGGIO B , et al . On security and sparsity of linear classifiers for adversarial settings [C ] // Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR) . Mérida : Springer , 2016 : 322 - 332 .

CHEN H , ZHANG H , BONING D , et al . Robust decision trees against adversarial examples [C ] // International Conference on Machine Learning . Florida : PMLR , 2019 : 1122 - 1131 .

XIE X Y , HO J W K , MURPHY C , et al . Testing and validating machine learning classifiers by metamorphic testing [J ] . The Journal of Systems and Software , 2011 , 84 ( 4 ): 544 - 558 .

DWARAKANATH A , AHUJA M , SIKAND S , et al . Identifying implementation bugs in machine learning based image classifiers using metamorphic testing [C ] // Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis . Amsterdam : ACM , 2018 : 118 - 128 .

AL-AZANI S , HASSINE J . Validation of machine learning classifiers using metamorphic testing and feature selection techniques [C ] // International Workshop on Multi-disciplinary Trends in Artificial Intelligence . Gadong : Springer , 2017 : 77 - 91 .

MA L , JUEFEI-XU F , ZHANG F Y , et al . DeepGauge: multi-granularity testing criteria for deep learning systems [C ] // 2018 33rd IEEE/ACM International Conference on Automated Software Engineering . Montpellier : IEEE , 2018 : 120 - 131 .

SUN Y C , WU M , RUAN W J , et al . Concolic testing for deep neural networks [C ] // Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering . Montpellier : ACM , 2018 : 109 - 119 .

TIAN Y C , PEI K X , JANA S , et al . DeepTest: Automated testing of deep-neural-network-driven autonomous cars [C ] // Proceedings of the 40th International Conference on Software Engineering . Gothenburg : ACM , 2018 : 303 - 314 .

BRAIEK H BEN , KHOMH F . DeepEvolution: A search-based testing approach for deep neural networks [C ] // 2019 IEEE International Conference on Software Maintenance and Evolution . Cleveland : IEEE , 2019 : 454 - 458 .

YAN S N , TAO G H , LIU X W , et al . Correlations between deep neural network model coverage criteria and model quality [C ] // Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Virtual Conference : ACM , 2020 : 775 - 787 .

GERASIMOU S , ENISER H F , SEN A , et al . Importance-driven deep learning system testing [C ] // Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings . Seoul : ACM , 2020 : 322 - 323 .

XIE X F , MA L , WANG H J , et al . DiffChaser: Detecting disagreements for deep neural networks [C ] // Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2019 : 5772 - 5778 .

YANG W , XIE T . Telemade: A testing framework for learning-based malware detection systems [C ] // Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence . Palo Alto : AAAI Press , 2018 : 400 - 403 .

CHEN T Y , POON P L , QIU K , et al . Use of metamorphic relations as knowledge carriers to train deep neural networks [EB/OL ] . ( 2021-04-10 )[ 2022-07-11 ] . https://arxiv.org/abs/2104.04718 https://arxiv.org/abs/2104.04718 .

XIE X , GUO W , MA L , et al . RNNrepair: Automatic RNN repair via model-based analysis [C ] // Proceedings of the 38th International Conference on Machine Learning . Virtual Conference : PMLR.org , 2021 : 11383 - 11392 .

AGGARWAL A , LOHIA P , NAGAR S , et al . Black box fairness testing of machine learning models [C ] // Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Tallinn : ACM , 2019 : 625 - 635 .

ZHANG P , WANG J , SUN J , et al . Automatic Fairness Testing of Neural Classifiers through Adversarial Sampling [J ] . IEEE Transactions on Software Engineering , 2022 : 3593 - 3612 .

ZHANG P X , WANG J Y , SUN J , et al . White-box fairness testing through adversarial sampling [C ] // 2020 IEEE/ACM 42nd International Conference on Software Engineering . Seoul : IEEE , 2020 : 949 - 960 .

DOSHI-VELEZ F , KIM B . Towards a rigorous science of interpretable machine learning [EB/OL ] . ( 2017-02-28 )[ 2022-07-11 ] . https://arxiv.org/abs/1702.08608 https://arxiv.org/abs/1702.08608 .

CHENG C H , N¨HRENBERG G , HUANG C H , et al . Towards dependability metrics for neural networks [C ] // 2018 16th ACM/IEEE International Conference on Formal Methods and Models for System Design . Beijing : IEEE , 2018 : 1 - 4 .

ROSS A , CHEN N N , HANG E Z , et al . Evaluating the interpretability of generative models by interactive reconstruction [C ] // Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . Yokohama : ACM , 2021 : 1 - 15 .

SCHIELZETH H . Simple means to improve the interpretability of regression coefficients [J ] . Methods in Ecology and Evolution , 2010 , 1 ( 2 ): 103 - 113 .

CHEN W J , SAHINER B , SAMUELSON F , et al . Calibration of medical diagnostic classifier scores to the probability of disease [J ] . Statistical Methods in Medical Research , 2018 , 27 ( 5 ): 1394 - 1409 .

KOKHLIKYAN N , MIGLANI V , MARTIN M , et al . Captum: A unified and generic model interpretability library for PyTorch [EB/OL ] . ( 2020-09-16 )[ 2022-07-11 ] . https://arxiv.org/abs/2009.07896 https://arxiv.org/abs/2009.07896 .

YANG Z J , WANG B H , LI H R , et al . On detecting growing-up behaviors of malicious accounts in privacy-centric mobile social networks [C ] // Annual Computer Security Applications Conference . Virtual Conference : ACM , 2021 : 297 - 310 .

BICHSEL B , GEHR T , DRACHSLER-COHEN D , et al . DP-finder: Finding differential privacy violations by sampling and optimization [C ] // Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security . Toronto : ACM , 2018 : 508 - 524 .

TRAMÈR F , ZHANG F , JUELS A , et al . Stealing machine learning models via prediction APIs [C ] // 25th USENIX security symposium (USENIX Security 16) . Berkeley : USENIX Association , 2016 : 601 - 618 .

WANG B H , GONG N Z . Stealing hyperparameters in machine learning [C ] // 2018 IEEE Symposium on Security and Privacy . San Francisco : IEEE , 2018 : 36 - 52 .

JAGIELSKI M , CARLINI N , BERTHELOT D , et al . High accuracy and high fidelity extraction of neural networks [C ] // Proceedings of the 29th USENIX Conference on Security Symposium . Virtual Conference : USENIX Association , 2020 : 1345 - 1362 .

XIE P T , BILENKO M , FINLEY T , et al . Crypto-nets: Neural networks over encrypted data [EB/OL ] . ( 2014-12-18 )[ 2022-07-11 ] . https://arxiv.org/abs/1412.6181 https://arxiv.org/abs/1412.6181 .

GILAD-BACHRACH R , DOWLIN N , LAINE K , et al . Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy [C ] // Proceedings of the 33nd International Conference on Machine Learning . Virtual Conference : JMLR.org , 2016 : 201 - 210 .

HESAMIFARD E , TAKABI H , GHASEMI M . CryptoDL: Deep neural networks over encrypted data [EB/OL ] . ( 2017-11-14 )[ 2022-07-11 ] . https://arxiv.org/abs/1711.05189 https://arxiv.org/abs/1711.05189 .

LINDELL Y , PINKAS B . Privacy preserving data mining [C ] // Advances in Cryptology — CRYPTO 2000 . California : Springer , 2000 : 36 - 54 .

JUVEKAR C , VAIKUNTANATHAN V , CHANDRAKASAN A . GAZELLE: A low latency framework for secure neural network inference [C ] // 27th USENIX Security Symposium (USENIX Security 18) . Berkeley : USENIX Association , 2018 : 1651 - 1669 .

CHANDRAN N , GUPTA D , RASTOGI A , et al . EzPC: Programmable and efficient secure two-party computation for machine learning [C ] // 2019 IEEE European Symposium on Security and Privacy . Stockholm : IEEE , 2019 : 496 - 511 .

ZHENG W , DENG R , CHEN W , et al . Cerebro: A platform for multi-party cryptographic collaborative learning [C ] // 30th USENIX Security Symposium (USENIX Security 21) . Berkeley : USENIX Association , 2021 : 2723 - 2740 .

KNOTT B , VENKATARAMAN S , HANNUN A , et al . Crypten: Secure multi-party computation meets machine learning [C ] // Advances in Neural Information Processing Systems . Virtual Conference : Curran Associates, Inc. , 2021 : 4961 - 4973 .

ISLAM M J , NGUYEN G , PAN R , et al . A comprehensive study on deep learning bug characteristics [C ] // Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Tallinn : ACM , 2019 : 510 - 520 .

JIA L , ZHONG H , WANG X Y , et al . An empirical study on bugs inside tensorflow [C ] // International Conference on Database Systems for Advanced Applications . Jeju : Springer , 2020 : 604 - 620 .

GU J Z , LUO X C , ZHOU Y F , et al . Muffin: Testing deep learning libraries via neural architecture fuzzing [EB/OL ] . ( 2022-04-19 )[ 2022-07-11 ] . https://arxiv.org/abs/2204.08734 https://arxiv.org/abs/2204.08734 .

MURPHY C , SHEN K , KAISER G . Automatic system testing of programs without test oracles [C ] // Proceedings of the eighteenth international symposium on Software Testing and Analysis . Chicago : ACM , 2009 : 189 - 200 .

DING J H , KANG X J , HU X H . Validating a deep learning framework by metamorphic testing [C ] /Proceedings of the 2nd International Workshop on Metamorphic Testing . Buenos Aires : IEEE , 2017 : 28 - 34 .

MA L , ZHANG F Y , SUN J Y , et al . DeepMutation: Mutation testing of deep learning systems [C ] // 2018 IEEE 29th International Symposium on Software Reliability Engineering . Memphis : IEEE , 2018 : 100 - 111 .

XIE D N , LI Y T , KIM M , et al . DocTer: Documentation-guided fuzzing for testing deep learning API functions [C ] // Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis . Virtual Conference : ACM , 2022 : 176 - 188 .

LIU J W , WEI Y X , YANG S , et al . Coverage-guided tensor compiler fuzzing with joint IR-pass mutation [J ] . Proceedings of the ACM on Programming Languages , 2022 , 6(OOPSLA 1 ) : 73 ( 1-26 ).

LIU L , WU Y Z , WEI W Q , et al . Benchmarking deep learning frameworks: Design considerations, metrics and beyond [C ] // 2018 IEEE 38th International Conference on Distributed Computing Systems . Vienna : IEEE , 2018 : 1258 - 1269 .

SRISAKAOKUL S , WU Z , ASTORGA A , et al . Multiple-implementation testing of supervised learning software [C ] // Workshops at the thirty-second AAAI conference on artificial intelligence . Palo Alto : AAAI Press , 2018 : 384 - 391 .

WANG Z , YAN M , CHEN J J , et al . Deep learning library testing via effective model generation [C ] // Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Virtual Conference : ACM , 2020 : 788 - 799 .

ZHANG X F , LIU J W , SUN N , et al . Duo: differential fuzzing for deep learning operators [J ] . IEEE Transactions on Reliability , 2021 , 70 ( 4 ): 1671 - 1685 .

Keras . Keras 2 . 3 . 0 : This is also the last major release of multi-backend Keras[EB/OL ] . ( 2019-07-18 )[ 2022-07-11 ] . https://github.com/keras-team/keras/releases/tag/2.3.0 https://github.com/keras-team/keras/releases/tag/2.3.0 .

MURPHY C , SHEN K , KAISER G . Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles [C ] // 2009 International Conference on Software Testing Verification and Validation . Denver : IEEE , 2009 : 436 - 445 .

WANG C J , SHEN J , FANG C R , et al . Accuracy measurement of deep neural network accelerator via metamorphic testing [C ] // 2020 IEEE International Conference on Artificial Intelligence Testing . Oxford : IEEE , 2020 : 55 - 61 .

HU Q , MA L , XIE X F , et al . DeepMutation: A mutation testing framework for deep learning systems [C ] // 2019 34th IEEE/ACM International Conference on Automated Software Engineering . San Diego : IEEE , 2019 : 1158 - 1161 .

LUO W S , CHAI D , RUN X Y , et al . Graph-based fuzz testing for deep learning inference engines [C ] // Proceedings of the 43rd International Conference on Software Engineering . Madrid : IEEE , 2021 : 288 - 299 .

ZHANG X F , YANG Y L , FENG Y , et al . Software engineering practice in the development of deep learning applications [EB/OL ] . ( 2019-10-08 )[ 2022-07-11 ] . https://arxiv.org/abs/1910.03156 https://arxiv.org/abs/1910.03156 .

ZHANG Y H , CHEN Y F , CHEUNG S C , et al . An empirical study on TensorFlow program bugs [C ] // Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis . Amsterdam : ACM , 2018 : 129 - 140 .

CHEN Z P , YAO H H , LOU Y L , et al . An empirical study on deployment faults of deep learning based mobile applications [C ] // Proceedings of the 43rd International Conference on Software Engineering . Madrid : IEEE , 2021 : 674 - 685 .

LAM A N , NGUYEN A T , NGUYEN H A , et al . Bug localization with combination of deep learning and information retrieval [C ] // 2017 IEEE/ACM 25th International Conference on Program Comprehension . Buenos Aires : IEEE , 2017 : 218 - 229 .

QI B H , SUN H L , YUAN W , et al . DreamLoc: A deep relevance matching-based framework for bug localization [J ] . IEEE Transactions on Reliability , 2022 , 71 ( 1 ): 235 - 249 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于镜像选择序优化的MART算法

一种新的多路径覆盖测试数据进化生成方法

一种面向性质的实时系统测试方法

基于抽象解释的变量值范围分析及应用