面向机器学习模型安全的测试与修复

doi:10.12263/DZXB.20220821

电子学报 ›› 2022, Vol. 50 ›› Issue (12): 2884-2918.DOI: 10.12263/DZXB.20220821

• 《电子学报》创刊60周年专栏 • 上一篇下一篇

面向机器学习模型安全的测试与修复

张笑宇¹^,², 沈超¹^,²(), 蔺琛皓¹^,², 李前¹^,², 王骞³, 李琦⁴^,⁵, 管晓宏¹^,²

^1.西安交通大学电子与信息学部网络空间安全学院，陕西西安 710049
^2.智能网络与网络安全教育部重点实验室（西安交通大学），陕西西安 710049
^3.武汉大学国家网络安全学院，湖北武汉 430072
^4.清华大学网络科学与网络空间研究院，北京 100084
^5.中关村实验室，北京 100094

收稿日期:2022-07-14 修回日期:2022-10-20 出版日期:2022-12-25
- 通讯作者:
- 沈超
- 作者简介:
- 张笑宇男，出生于1999年，河南郑州人.西安交通大学网络空间安全学院博士研究生.主要研究领域为人工智能软件测试.E-mail: zxy0927@stu.xjtu.edu.cn
  沈超男，出生于1985年，重庆人.博士，西安交通大学教授、博士生导师.主要研究领域为可信人工智能、人工智能安全和信息物理系统安全.
  蔺琛皓男，出生于1989年，陕西西安人.博士，西安交通大学特聘研究员、博士生导师.主要研究领域为人工智能安全、对抗机器学习、智能身份认证.E-mail: linchenhao@xjtu.edu.cn
  李前男，出生于1992年，陕西宝鸡人.博士，西安交通大学助理教授.主要研究领域为人工智能安全、对抗机器学习.E-mail: qianlix@xjtu.edu.cn
  王骞男，出生于1980年，湖北武汉人.博士，武汉大学教授、博士生导师.主要研究领域为人工智能安全、云计算安全与隐私、无线系统安全、应用密码学.E-mail: qianwang@whu.edu.cn
  李琦男，出生于1979年，浙江临安人.博士，清华大学副教授、博士生导师.主要研究领域为互联网和云安全、移动安全、机器学习与安全、大数据安全、区块链与安全.E-mail: qli01@tsinghua.edu.cn
  管晓宏男，出生于1955年，四川泸州人.博士，西安交通大学教授、博士生导师，中国科学院院士.主要研究领域为网络信息安全、网络化系统、电力系统优化调度.E-mail: xhguan@xjtu.edu.cn
- 基金资助:
- 科技创新2030——“新一代人工智能”重大项目(2020AAA0107702);国家自然科学基金(62161160337);陕西重点研发计划项目(2021ZD LGY01-02)

The Testing and Repairing Methods for Machine Learning Model Security

ZHANG Xiao-yu¹^,², SHEN Chao¹^,²(), LIN Chen-hao¹^,², LI Qian¹^,², WANG Qian³, LI Qi⁴^,⁵, GUAN Xiao-hong¹^,²

^1.School of Cyber Science and Engineering，Faculty of Electronic and Information Engineering，Xi’an Jiaotong;University，Xi’an，Shaanxi 710049，China
^2.Key Laboratory for Intelligent Networks and Network Security（Xi’an Jiaotong University），Xi’an，Shaanxi 710049，China
^3.School of Cyber Science and Engineering，Wuhan University，Wuhan，Hubei 430072，China
^4.Institute for Network Sciences and Cyberspace，Tsinghua University，Beijing 100084，China
^5.Zhongguancun Laboratory，Beijing 100094，China

Received:2022-07-14 Revised:2022-10-20 Online:2022-12-25 Published:2023-03-20
- Corresponding author:
- SHEN Chao

摘要/Abstract

摘要：

近年来，以机器学习算法为代表的人工智能技术在计算机视觉、自然语言处理、语音识别等领域取得了广泛的应用，各式各样的机器学习模型为人们的生活带来了巨大的便利.机器学习模型的工作流程可以分为三个阶段.首先，模型接收人工收集或算法生成的原始数据作为输入，并通过预处理算法（如数据增强和特征提取）对数据进行预处理.随后，模型定义神经元或层的架构，并通过运算符（例如卷积和池）构建计算图.最后，模型调用机器学习框架的函数功能实现计算图并执行计算，根据模型神经元的权重计算输入数据的预测结果.在这个过程中，模型中单个神经元输出的轻微波动可能会导致完全不同的模型输出，从而带来巨大的安全风险.然而，由于对机器学习模型的固有脆弱性及其黑箱特征行为的理解不足，研究人员很难提前识别或定位这些潜在的安全风险，这为个人生命财产安全乃至国家安全带来了诸多风险和隐患.研究机器学习模型安全的相关测试与修复方法，对深刻理解模型内部风险与脆弱性、全面保障机器学习系统安全性以及促进人工智能技术的广泛应用有着重要意义.本文从不同安全测试属性出发，详细介绍了现有的机器学习模型安全测试和修复技术，总结和分析了现有研究中的不足，探讨针对机器学习模型安全的测试与修复的技术进展和未来挑战，为模型的安全应用提供了指导和参考.本文首先介绍了机器学习模型的结构组成和主要安全测试属性，随后从机器学习模型的三个组成部分即数据、算法和实现，六种模型安全相关测试属性即正确性、鲁棒性、公平性、效率、可解释性和隐私性，分析、归纳和总结了相关的测试与修复方法及技术，并探讨了现有方法的局限.最后本文讨论和展望了机器学习模型安全的测试与修复方法的主要技术挑战和发展趋势.

关键词: 人工智能安全, 机器学习安全, 机器学习模型测试, 机器学习模型修复, 软件测试, 软件修复

Abstract:

In recent years, artificial intelligence technology led by machine learning algorithms has been widely used in many fields, such as computer vision, natural language processing, speech recognition, etc. A variety of machine learning models have greatly facilitated people's lives. The workflow of a machine learning model consists of three stages. First, the model receives the raw data which is collected or generated by the developers as the model input and preprocesses the data through preprocessing algorithms, such as data augmentation and feature extraction. Subsequently, the model defines the architecture of neurons or layers in the model and constructs a computational graph through operators(e.g., convolution and pooling). Finally, the model calls the machine learning framework function to implement the operators and calculates the prediction result of the input data according to the weights of model neurons. In this process, slight fluctuations in the output of individual neurons in the model may lead to an entirely different model output, which can bring huge security risks. However, due to the insufficient understanding of the inherent vulnerability of machine learning models and their black box characteristic behaviors, it is difficult for researchers to identify or locate these potential security risks in advance. This brings many risks and hidden dangers to personal property safety and even national security. There is great significance to studying the testing and repairing methods for machine learning model security, which can help deeply understand the internal risks and vulnerabilities of models, comprehensively guarantee the security of machine learning systems, and widely apply artificial intelligence technology. The existing testing research for the machine learning model security has mainly focused on the correctness, robustness, and other testing properties of the model, and this research has achieved certain results. This paper intends to start from different security testing attributes, introduces the existing machine learning model security testing and repair technology in detail, summarizes and analyzes the deficiencies in the existing research, and discusses the technical progress and challenges of machine learning model security testing and repairing, providing guidance and reference for the safe application of the model. In this paper, we first introduce the structural composition and main testing properties of the machine learning model security. Afterwards, we systematically summarize and analyze the existing work from the three components of the machine learning model—data, algorithm, and implementation, and six model security-related testing properties-correctness, robustness, fairness, efficiency, interpretability, and privacy. We also discuss the effectiveness and limitations of the existing testing and repairing methods. Finally, we discuss several technical challenges and potential development directions of the testing and repairing methods for machine learning model security in the future.

Key words: artificial intelligence security, machine learning security, machine learning model testing, machine learning model repairing, software testing, software repairing

中图分类号:

TP391

张笑宇, 沈超, 蔺琛皓, 李前, 王骞, 李琦, 管晓宏. 面向机器学习模型安全的测试与修复[J]. 电子学报, 2022, 50(12): 2884-2918.

ZHANG Xiao-yu, SHEN Chao, LIN Chen-hao, LI Qian, WANG Qian, LI Qi, GUAN Xiao-hong. The Testing and Repairing Methods for Machine Learning Model Security[J]. Acta Electronica Sinica, 2022, 50(12): 2884-2918.

图/表 6

参考文献 231

1	WORTSMAN M, ILHARCO G, GADRE S Y, et al. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time[EB/OL]. (2022-03-10)[2022-07]. .
2	BAO H B, DONG L, PIAO S H, et al. BEiT: BERT pre-training of image transformers[EB/OL]. (2021-06-15)[2022-07]. .
3	TAN M X, LE Q. Efficientnet: Rethinking model scaling for convolutional neural networks[C]//International Conference on Machine Learning. New Orleans: PMLR.org, 2019: 6105-6114.
4	BROWN T B, MANN B, RYDER N, et al. Language models are few-shot learners[C]//34th International Conference on Neural Information Processing Systems. Vancouver: Curran Associates Inc., 2020: 1877-1901.
5	MELIS G, KOČISKÝ T, BLUNSOM P. Mogrifier LSTM[EB/OL]. (2019-09-04) [2022-07]. .
6	YAMADA I, ASAI A, SHINDO H, et al. LUKE: Deep contextualized entity representations with entity-aware self-attention[EB/OL]. (2020-10-02)[2022-07]. .
7	KOLOBOV R, OKHAPKINA O, OMELCHISHINA O, et al. MediaSpeech: Multilanguage ASR benchmark and dataset[EB/OL]. (2021-03-30)[2022-07]. .
8	PARK D S, ZHANG Y, JIA Y, et al. Improved noisy student training for automatic speech recognition[EB/OL]. (2020-05-19)[2022-07]. .
9	XU Q T, BAEVSKI A, LIKHOMANENKO T, et al. Self-training and pre-training are complementary for speech recognition[C]//2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Toronto: IEEE, 2021: 3030-3034.
10	JHA D, RIEGLER M A, JOHANSEN D, et al. DoubleU-net: A deep convolutional neural network for medical image segmentation[C]//2020 IEEE 33rd International Symposium on Computer-Based Medical Systems. Rochester: IEEE, 2020: 558-564.
11	SRIVASTAVA A, JHA D, CHANDA S, et al. MSRF-net: A multi-scale residual fusion network for biomedical image segmentation[EB/OL]. (2021-05-16)[2022-07]. .
12	WANG J F, HUANG Q M, TANG F L, et al. Stepwise feature fusion: Local guides global[EB/OL]. (2022-03-07)[2022-07]. .
13	STOICA I, SONG D, POPA R A, et al. A Berkeley view of systems challenges for AI[EB/OL]. (2017-12-15)[2022-07]. .
14	Research and Market. Edge AI Market – Forecasts from 2021 to 2026[EB/OL]. (2021-03)[2022-07]. .
15	ABADI M. TensorFlow: Learning functions at scale[J]. ACM SIGPLAN Notices, 2016, 51(9): 1.
16	马艳军, 于佃海, 吴甜, 等, 飞桨 : 源于产业实践的开源深度学习平台[J]. 数据与计算发展前沿, 2019, 1(1): 105-115.
	MA Y, YU D, WU T, et al. PaddlePaddle: An open-source deep learning platform from industrial practice[J]. Frontiers of Data and Domputing, 2019, 1(1): 105-115. (in Chinese)
17	CHEN T Q, LI M, LI Y T, et al. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems[EB/OL]. (2015-12-03)[2022-07]. .
18	PASZKE A, GROSS S, MASSA F, et al. Pytorch: An imperative style, high-performance deep learning library[C]//33rd International Conference on Neural Information Processing Systems. Vancouver: Curran Associates, Inc., 2019: 8026-8037.
19	Google.AI and machine learning products[EB/OL]. (2022)[2022-07-11]. .
20	Baidu. Baidu Al open platform[EB/OL]. (2021)[2022-07-11]. .
21	JULIA A, JEFF L, SURYA M, et al. Machine Bias[R/OL]. (2016-05-23)[2022-07-11]. .
22	WAKABAYASHI D. Self-driving uber car kills pedestrian in Arizona,where robots roam[EB/OL]. (2018-03-19)[2022-07-11]. .
23	ELSOM J. Moment an Amazon Alexa tells a terrified mother, 29, to “stab yourself in the heart for the greater good” while reading from rogue Wikipedia text[EB/OL]. (2019-12-19)[2022-07-11]. .
24	XIE X F, MA L, JUEFEI-XU F, et al. DeepHunter: A coverage-guided fuzz testing framework for deep neural networks[C]//Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. Beijing: ACM, 2019: 146-157.
25	ZHANG X Y, ZHAI J, MA S Q, et al. AUTOTRAINER: An automatic DNN training problem detection and repair system[C]//2021 IEEE/ACM 43rd International Conference on Software Engineering. Madrid: IEEE, 2021: 359-371.
26	ODENA A, OLSSON C, ANDERSEN D, et al. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing[C]//Proceedings of the 36th International Conference on Machine Learning. Virtual Conference: PMLR.org, 2019: 4901-4911.
27	GAO X Q, ZHAI J, MA S Q, et al. FairNeuron: Improving deep neural network fairness with adversary games on selective neurons[EB/OL]. (2022-04-06)[2022-07-11]. .
28	中华人民共和国工业和信息化部. 工业和信息化部关于印发《促进新一代人工智能产业发展三年行动计划(2018—2020年)》的通知[EB/OL]. (2017-12-13)[2022-07-11]. .
29	The White House Office Of Science And Technology Policy. American AI Initiative One Year Annua Report[R/OL]. 2020. .
30	纪守领, 杜天宇, 李进锋, 等. 机器学习模型安全与隐私研究综述[J]. 软件学报, 2021, 32(1): 41-67.
	JI S L, DU T Y, LI J F, et al. Security and privacy of machine learning models: A survey[J]. Journal of Software, 2021, 32(1): 41-67. (in Chinese)
31	HUANG X W, KROENING D, RUAN W J, et al. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability[J]. Computer Science Review, 2020, 37: 100270.
32	ZHANG J M, HARMAN M, MA L, et al. Machine learning testing: Survey, landscapes and horizons[J]. IEEE Transactions on Software Engineering, 2022, 48(1): 1-36.
33	BRAIEK H B, KHOMH F. On testing machine learning programs[J]. Journal of Systems and Software, 2020, 164: 110542.
34	MEHRABI N, MORSTATTER F, SAXENA N, et al. A survey on bias and fairness in machine learning[J]. ACM Computing Surveys, 2021, 54(6): 1-35.
35	AMERSHI S, BEGEL A, BIRD C, et al. Software engineering for machine learning: A case study[C]//2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice. Montreal: IEEE, 2019: 291-300.
36	JESMEEN M Z H, HOSSEN J, SAYEED S, et al. A survey on cleaning dirty data using machine learning paradigm for big data analytics[J]. Indonesian Journal of Electrical Engineering and Computer Science, 2018, 10(3): 1234-1243.
37	KHALID S, KHALIL T, NASREEN S. A survey of feature selection and feature extraction techniques in machine learning[C]//2014 Science and Information Conference. London: IEEE, 2014: 372-378.
38	ROH Y, HEO G, WHANG S E. A survey on data collection for machine learning: A big data - AI integration perspective[J]. IEEE Transactions on Knowledge and Data Engineering, 2021, 33(4): 1328-1347.
39	REFAEILZADEH P, TANG L, LIU H. Cross-validation[M]//Encyclopedia of Database Systems. Boston: Springer, 2009: 532-538.
40	SHAHROKNI A, FELDT R. A systematic review of software robustness[J]. Information and Software Technology, 2013, 55(1): 1-17.
41	IEEE. IEEE Standard Glossary of Software Engineering Terminology[A/OL]. (1990-12-31) [2022-07-11]. .
42	纪守领, 杜天宇, 邓水光, 等. 深度学习模型鲁棒性研究综述[J]. 计算机学报, 2022, 45(1): 190-206.
	JI S L, DU T Y, DENG S G, et al. Robustness certification research on deep learning models: A survey[J]. Chinese Journal of Computers, 2022, 45(1): 190-206. (in Chinese)
43	GAJANE P, PECHENIZKIY M. On formalizing fairness in prediction with machine learning[EB/OL]. (2017-10-09)[2022-07-11]. .
44	HARDT M, PRICE E, SREBRO N. Equality of opportunity in supervised learning[J]. Advances in Neural Information Processing Systems. Barcelona: Curran Associates Inc., 2016: 29.
45	ZAFAR M B, VALERA I, ROGRIGUEZ M G, et al. Fairness constraints: Mechanisms for fair classification[C]//Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Fort Laud-erdale: PMLR, 2017: 962-970.
46	KUSNER M J, LOFTUS J, RUSSELL C, et al. Counterfactual fairness[J]. Advances in Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017: 30.
47	DWORK C, HARDT M, PITASSI T, et al. Fairness through awareness[C]//Proceedings of the 3rd Innovations in Theoretical Computer Science Conference. Beijing: ACM, 2012: 214-226.
48	GUO Q Y, CHEN S, XIE X F, et al. An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms[C]//34th IEEE/ACM International Conference on Automated Software Engineering. San Diego: IEEE, 2019: 810-822.
49	GOODMAN B, FLAXMAN S. European union regulations on algorithmic decision-making and a “right to explanation”[J]. AI Magazine, 2017, 38(3): 50-57.
50	DWORK C. Differential privacy: A survey of results[C]//International Conference on Theory and Applications of Models of Computation. Berlin: Springer, 2008: 1-19.
51	GUO Q Y, XIE X F, LI Y, et al. Audee: Automated testing for deep learning frameworks[C]//35th IEEE/ACM International Conference on Automated Software Engineering. Virtual Conference: ACM, 2020: 486-498.
52	PHAM H V, LUTELLIER T, QI W Z, et al. CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries[C]//2019 IEEE/ACM 41st International Conference on Software Engineering. Montreal: IEEE, 2019: 1027-1038.
53	WANG J N, LUTELLIER T, QIAN S S, et al. EAGLE: Creating equivalent graphs to test deep learning libraries[C]//2022 IEEE/ACM 44th International Conference on Software Engineering. Pittsburgh: IEEE, 2022: 798-810.
54	ZHANG X F, SUN N, FANG C R, et al. Predoo: precision testing of deep learning operators[C]//Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual Conference: ACM, 2021: 400-412.
55	SANTOS S H N, SILVEIRA B N C DA, ANDRADE S A, et al. An experimental study on applying metamorphic testing in machine learning applications[C]//Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing. Natal: ACM, 2020: 98-106.
56	XIAO D W, LIU Z B, YUAN Y Y, et al. Metamorphic testing of deep learning compilers[J]. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2022, 6(1): 1-28.
57	GOODFELLOW I J, SHLENS J, SZEGEDY C. Explaining and harnessing adversarial examples[EB/OL]. (2014-12-20)[2022-07-11]. .
58	PAPERNOT N, FAGHRI F, CARLINI N, et al. Technical report on the CleverHans v2.1.0 adversarial examples library[EB/OL]. (2016-10-03)[2022-07-11]. .
59	XIE C H, WANG J Y, ZHANG Z S, et al. Mitigating adversarial effects through randomization[EB/OL]. (2017-11-06)[2022-07-11]. .
60	KOLBEINSSON A, KOSSAIFI J, PANAGAKIS Y, et al. Tensor dropout for robust learning[J]. IEEE Journal of Selected Topics in Signal Processing, 2021, 15(3): 630-640.
61	XU W L, EVANS D, QI Y J. Feature squeezing: Detecting adversarial examples in deep neural networks[EB/OL]. (2017-04-04)[2022-07-11]. .
62	XU W L, EVANS D, QI Y J. Feature squeezing mitigates and detects carlini/Wagner adversarial examples[EB/OL]. (2017-05-30)[2022-07-11]. .
63	WANG J Y, DONG G L, SUN J, et al. Adversarial sample detection for deep neural network through model mutation testing[C]//2019 IEEE/ACM 41st International Conference on Software Engineering. Montreal: IEEE, 2019: 1245-1256.
64	ZHAO Z, CHEN G K, WANG J Y, et al. Attack as defense: Characterizing adversarial examples using robustness[C]//Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual Conference: ACM. 2021: 42-55.
65	NGUYEN G H, BOUZERDOUM A, PHUNG S L. A supervised learning approach for imbalanced data sets[C]//2008 19th International Conference on Pattern Recognition. Tampa: IEEE, 2008: 1-4.
66	AMINI A, SOLEIMANY A P, SCHWARTING W, et al. Uncovering and mitigating algorithmic bias through learned latent structure[C]//Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Honolulu: ACM, 2019: 289-295.
67	MULLICK S S, DATTA S, DHEKANE S G, et al. Appropriateness of performance indices for imbalanced data classification: An analysis[J]. Pattern Recognition, 2020, 102: 107197.
68	KAMIRAN F, CALDERS T. Classifying without discriminating[C]//2009 2nd International Conference on Computer, Control and Communication. Karachi: IEEE, 2009: 1-6.
69	AMINI A, SCHWARTING W, ROSMAN G, et al. Variational autoencoder for end-to-end control of autonomous driving with novelty detection and training de-biasing[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid: IEEE, 2018: 568-575.
70	TOMALIN M, BYRNE B, CONCANNON S, et al. The practical ethics of bias reduction in machine translation: Why domain adaptation is better than data debiasing[J].Ethics and Information Technology, 2021, 23(3): 419-433.
71	HOLLAND S, HOSNY A, NEWMAN S, et al. The dataset nutrition label: A framework to drive higher data quality standards[EB/OL]. (2018-05-09)[2022-07-11]. .
72	HYNES N, SCULLEY D, TERRY M. The data linter: Lightweight automated sanity checking for ML data sets[C]//NIPS MLSys Workshop. Cambridge: MIT Press, 2017: 1.
73	KRISHNAN S, WU E. AlphaClean: Automatic generation of data cleaning pipelines[EB/OL]. (2019-04-26)[2022-07-11]. .
74	LAISHRAM R, PHOHA V V. Curie: A method for protecting SVM Classifier from Poisoning Attack[EB/OL]. (2016-06-05)[2022-07-11]. .
75	ZHANG W N, WANG D, TAN X Y. Robust class-specific autoencoder for data cleaning and classification in the presence of label noise[J]. Neural Processing Letters, 2019, 50(2): 1845-1860.
76	STEINHARDT J, KOH P W, LIANG P. Certified defenses for data poisoning attacks[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017: 3520-3532.
77	SHOKRI R, STRONATI M, SONG C Z, et al. Membership inference attacks against machine learning models[C]//2017 IEEE Symposium on Security and Privacy. San Jose: IEEE, 2017: 3-18.
78	PAPERNOT N, ABADI M, ERLINGSSON Ú, et al. Semi-supervised knowledge transfer for deep learning from private training data[EB/OL]. (2016-10-18)[2022-07-11]. .
79	HUANG K, LIU X M, FU S J, et al. A lightweight privacy-preserving CNN feature extraction framework for mobile sensing[J]. IEEE Transactions on Dependable and Secure Computing, 2021, 18(3): 1441-1455.
80	BONAWITZ K, IVANOV V, KREUTER B, et al. Practical secure aggregation for privacy-preserving machine learning[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. Dallas: ACM, 2017: 1175-1191.
81	SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[EB/OL]. (2013-12-31)[2022-07-11]. .
82	CARLINI N, WAGNER D. Towards evaluating the robustness of neural networks[C]//2017 IEEE Symposium on Security and Privacy. San Jose: IEEE, 2017: 39-57.
83	MOOSAVI-DEZFOOLI S M, FAWZI A, FROSSARD P. DeepFool: A simple and accurate method to fool deep neural networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2574-2582.
84	GOPINATH D, KATZ G, PASAREANU C S,et al. DeepSafe: A data-driven approach for assessing robustness of neural networks[C]//International Symposium on Automated Technology for Verification and Analysis. Los Angeles: Springer, 2018: 3-19.
85	SHEN M, YU H, ZHU L H, et al. Effective and robust physical-world attacks on deep learning face recognition systems[J]. IEEE Transactions on Information Forensics and Security, 2021, 16: 4063-4077.
86	HAN S C, LIN C H, SHEN C, et al. Rethinking adversarial examples exploiting frequency-based analysis[C]//International Conference on Information and Communications Security. Chongqing: Springer, 2021: 73-89.
87	MU J M, WANG B H, LI Q, et al. A hard label black-box adversarial attack against graph neural networks[C]//Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. Virtual Conference: ACM, 2021: 108-125.
88	MAHMOOD K, MAHMOOD R, VAN DIJK M. On the robustness of vision transformers to adversarial examples[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 7818-7827.
89	BALUJA S, FISCHER I. Learning to attack: Adversarial transformation networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Lousiana: AAAI Press, 2018, 32(1): 2687-2695.
90	CARLINI N, WAGNER D. Audio adversarial examples: Targeted attacks on speech-to-text[C]//2018 IEEE Security and Privacy Workshops. San Francisco: IEEE, 2018: 1-7.
91	CISSE M, ADI Y, NEVEROVA N, et al. Houdini: Fooling deep structured prediction models[EB/OL]. (2017-07-17)[2022-07-11]. .
92	ZHENG B L, JIANG P P, WANG Q, et al. Black-box adversarial attacks on commercial speech platforms with minimal information[C]//Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. Virtual Conference: ACM, 2021: 86-107.
93	BROWN T B, MANÉ D, ROY A, et al. Adversarial patch[EB/OL]. (2017-12-27)[2022-07-11]. .
94	GOODFELLOW I J, PAPERNOT N, MCDANIEL P. Cleverhans V 0.1: An adversarial machine learning library[EB/OL]. (2016-10-03)[2022-07-11].
	1610.00768v1.
95	RAUBER J, BRENDEL W, BETHGE M. Foolbox: A Python toolbox to benchmark the robustness of machine learning models[EB/OL]. (2017-07-13)[2022-07-11]. .
96	NICOLAE M I, SINN M, TRAN M N, et al. Adversarial robustness toolbox v 1.0.0[EB/OL]. (2018-07-03)[2022-07-11]. .
97	任奎, ZHENG Tianhang, 秦湛, 等. 深度学习中的对抗性攻击和防御[J]. Engineering, 2020, 6(3): 307-339.
	REN K, ZHEBG T, QIN Z, et al. Adversarial attacks and defenses in deep learning[J]. Engineering, 2020, 6(3): 307-339. (in Chinese)
98	LIU X Q, CHENG M H, ZHANG H, et al. Towards robust neural networks via random self-ensemble[C]//European Conference on Computer Vision. Munich: Springer, 2018: 381-397.
99	GUO C, RANA M, CISSE M, et al. Countering adversarial images using input transformations[EB/OL]. (2017-10-31)[2022-07-11]. .
100	LUO T G, CAI T L, ZHANG M X, et al. RANDOM MASK: Towards robust convolutional neural networks[EB/OL]. (2020-07-27)[2022-07-11]. .
101	SHARMA Y, CHEN P Y. Bypassing feature squeezing by increasing adversary strength[EB/OL]. (2018-03-27)[2022-07-11]. .
102	SAMANGOUEI P, KABKAB M, CHELLAPPA R. Defense-GAN: Protecting classifiers against adversarial attacks using generative models[EB/OL]. (2018-03-17)[2022-07-11]. .
103	LIAO F Z, LIANG M, DONG Y P, et al. Defense against adversarial attacks using high-level representation guided denoiser[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 1778-1787.
104	SHEN S W, JIN G Q, GAO K, et al. APE-GAN: Adversarial perturbation elimination with GAN[EB/OL]. (2017-07-18)[2022-07-11]. .
105	YANG R, CHEN X Q, CAO T J. APE-GAN++: An improved APE-GAN to eliminate adversarial perturbations[J]. IAENG International Journal of Computer Science, 2021, 48(3): 827-844.
106	KHERCHOUCHE A, FEZZA S A, HAMIDOUCHE W. Detect and defense against adversarial examples in deep learning using natural scene statistics and adaptive denoising[J]. Neural Computing and Applications, 2022, 34(24): 21567-21582.
107	ESMAEILPOUR M, CARDINAL P, KOERICH A L. Class-conditional defense GAN against end-to-end speech attacks[C]//ICASSP 2021 – 2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Toronto: IEEE, 2021: 2565-2569.
108	METZEN J H, GENEWEIN T, FISCHER V, et al. On detecting adversarial perturbations[EB/OL]. (2017-02-14)[2022-07-11]. .
109	CARLINI N, WAGNER D. Adversarial examples are not easily detected: Bypassing ten detection methods[C]//Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security. Dallas: ACM, 2017: 3-14.
110	BRECK E, POLYZOTIS N, ROY S, et al. Data validation for machine learning[C]//Proceedings of Machine Learning and Systems. Stanford: mlsys.org, 2019: 334-347.
111	GEBRU T, MORGENSTERN J, VECCHIONE B, et al. Datasheets for datasets[J]. Communications of the ACM, 2021, 64(12): 86-92.
112	BENDER E M, FRIEDMAN B. Data statements for natural language processing: Toward mitigating system bias and enabling better science[J]. Transactions of the Association for Computational Linguistics, 2018, 6: 587-604.
113	CHAKRABORTY J, XIA T P, FAHID F M, et al. Software engineering for fairness: A case study with hyperparameter optimization[EB/OL]. (2019-05-14)[2022-07-11]. .
114	KAMIRAN F, CALDERS T. Data preprocessing techniques for classification without discrimination[J]. Knowledge and Information Systems, 2012, 33(1): 1-33.
115	SATTIGERI P, HOFFMAN S C, CHENTHAMARAKSHAN V, et al. Fairness GAN[EB/OL]. (2018-05-24)[2022-07-11]. .
116	AÏVODJI U, BIDET F, GAMBS S, et al. Local data debiasing for fairness based on generative adversarial training[J]. Algorithms, 2021, 14(3): 87.
117	JALAL A, KARMALKAR S, HOFFMANN J, et al. Fairness for image generation with uncertain sensitive attributes[C]//Proceedings of the 38th International Conference on Machine Learning. Virtual Conference: PMLR, 2021: 4721-4732.
118	KRISHNAN S, WANG J N, WU E, et al. ActiveClean: Interactive data cleaning for statistical modeling[J]. Proceedings of the VLDB Endowment, 2016, 9(12): 948-959.
119	KRISHNAN S, FRANKLIN M J, GOLDBERG K, et al. BoostClean: automated error detection and repair for machine learning[EB/OL]. (2017-11-03)[2022-07-11]. .
120	SONG J, HE Y Y. Auto-validate: Unsupervised data validation using data-domain patterns inferred from data lakes[C]//Proceedings of the 2021 International Conference on Management of Data. Virtual Conference: ACM, 2021: 1678-1691.
121	RUBINSTEIN B I P, NELSON B, HUANG L, et al. ANTIDOTE: understanding and defending against poisoning of anomaly detectors[C]//Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement. Chicago: ACM, 2009: 1-14.
122	RAHM E, DO H. Data cleaning: Problems and current approaches[J]. IEEE Data Eng. Bull., 2000, 23: 3-13.
123	FREDRIKSON M, LANTZ E, JHA S, et al. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing[C]//Proceedings of the 23rd USENIX Security Symposium. Berkeley: USENIX Association, 2014, 2014: 17-32.
124	HITAJ B, ATENIESE G, PEREZ-CRUZ F. Deep models under the GAN: Information leakage from collaborative deep learning[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. Dallas: ACM, 2017: 603-618.
125	ATENIESE G, FELICI G, MANCINI L V, et al. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers[EB/OL]. (2013-06-19)[2022-07-11]. .
126	ERLINGSSON Ú, PIHUR V, KOROLOVA A. RAPPOR: randomized aggregatable privacy-preserving ordinal response[C]//Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. Scottsdale: ACM, 2014: 1054-1067.
127	SALEM A, ZHANG Y, HUMBERT M, et al. ML-leaks: Model and data independent membership inference attacks and defenses on machine learning models[EB/OL]. (2018-06-04)[2022-07-11]. .
128	李强, 颜浩, 陈克非. 安全多方计算协议的研究与应用[J]. 计算机科学, 2003, 30(8): 52-55.
	LI Q, YAN H, CHEN K F. Research and application of secure multi-party computation protocols[J]. Computer Science, 2003, 30(8): 52-55. (in Chinese)
129	YAO A C. Protocols for secure computations[C]//23rd Annual Symposium on Foundations of Computer Science. Chicago: IEEE, 1982: 160-164.
130	GOLDREICH O, MICALI S, WIGDERSON A. How to play ANY mental game[C]//Proceedings of the Nineteenth Annual ACM Symposium on Theory of Computing. New York: ACM, 1987: 218-229.
131	VAIDYA J, CLIFTON C. Privacy-preserving k-means clustering over vertically partitioned data[C]//Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Washington: ACM, 2003: 206-215.
132	MEHNAZ S, BELLALA G, BERTINO E. A secure sum protocol and its application to privacy-preserving multi-party analytics[C]//Proceedings of the 22nd ACM on Symposium on Access Control Models and Technologies. Indianapolis: ACM, 2017: 219-230.
133	MOHASSEL P, ZHANG Y P. SecureML: A system for scalable privacy-preserving machine learning[C]//2017 IEEE Symposium on Security and Privacy. San Jose: IEEE, 2017: 19-38.
134	ROUHANI B D, RIAZI M S, KOUSHANFAR F. DeepSecure: scalable provably-secure deep learning[C]//55th ACM/ESDA/IEEE Design Automation Conference. San Francisco: IEEE, 2018: 1-6.
135	KONEČNÝ J, MCMAHAN H B, YU F X, et al. Federated learning: Strategies for improving communication efficiency[EB/OL]. (2016-10-18)[2022-07-11]. .
136	MCMAHAN H B, RAMAGE D, TALWAR K, et al. Learning differentially private recurrent language models[EB/OL]. (2017-10-18)[2022-07-11]. .
137	WENG J S, WENG J, ZHANG J L, et al. DeepChain: Auditable and privacy-preserving deep learning with blockchain-based incentive[J]. IEEE Transactions on Dependable and Secure Computing, 2021, 18(5): 2438-2455.
138	GOEL K, RAJANI N, VIG J, et al. Robustness gym: Unifying the NLP evaluation landscape[EB/OL]. (2021-01-13)[2022-07-11]. .
139	PAULI P, KOCH A, BERBERICH J, et al. Training robust neural networks using lipschitz bounds[J]. IEEE Control Systems Letters, 2022, 6: 121-126.
140	PEI K X, CAO Y Z, YANG J F, et al. DeepXplore: automated whitebox testing of deep learning systems[C]//Proceedings of the 26th Symposium on Operating Systems Principles. Shanghai: ACM, 2017: 1-18.
141	GUO J M, JIANG Y, ZHAO Y, et al. DLFuzz: Differential fuzzing testing of deep learning systems[C]//Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Lake Buena Vista: ACM, 2018: 739-743.
142	MURPHY C, KAISER G, HU L F, et al. Properties of machine learning applications for use in metamorphic testing[C]//Proceedings of the Twentieth International Conference on Software Engineering & Knowledge Engineering. San Francisco: Knowledge Systems Institute Graduate School, 2008: 867-872.
143	XIE X Y, ZHANG Z Y, CHEN T Y, et al. METTLE: A METamorphic testing approach to assessing and validating unsupervised machine learning systems[J]. IEEE Transactions on Reliability, 2020, 69(4): 1293-1322.
144	JIANG M Y, CHEN T Y, WANG S. On the effectiveness of testing sentiment analysis systems with metamorphic testing[J]. Information and Software Technology, 2022, 150: 106966.
145	MA S Q, LIU Y Q, LEE W C, et al. MODE: Automated neural network model debugging via state differential analysis and input selection[C]//Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Lake Buena Vista: ACM, 2018: 175-186.
146	YU B, QI H, GUO Q, et al. DeepRepair: Style-guided repairing for deep neural networks in the real-world operational environment[J]. IEEE Transactions on Reliability, 2022, 71(4): 1401-1416.
147	SUN Z Y, ZHANG J M, HARMAN M, et al. Automatic testing and improvement of machine translation[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. Seoul: ACM, 2020: 974-985.
148	WARDAT M, LE W, RAJAN H. DeepLocalize: Fault localization for deep neural networks[C]//2021 IEEE/ACM 43rd International Conference on Software Engineering. Madrid: IEEE, 2021: 251-262.
149	TRAMÈR F, ATLIDAKIS V, GEAMBASU R, et al. FairTest: Discovering unwarranted associations in data-driven applications[C]//2017 IEEE European Symposium on Security and Privacy. Paris: IEEE, 2017: 401-416.
150	ANGELL R, JOHNSON B, BRUN Y, et al. Themis: automatically testing software for discrimination[C]//Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Lake Buena Vista: ACM, 2018: 871-875.
151	UDESHI S, ARORA P, CHATTOPADHYAY S. Automated directed fairness testing[C]//Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. Montpellier: ACM, 2018: 98-108.
152	BLACK E, YEOM S, FREDRIKSON M. FlipTest: Fairness testing via optimal transport[C]//Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. Barcelona: ACM, 2020: 111-121.
153	KAMIRAN F, MANSHA S, KARIM A, et al. Exploiting reject option in classification for social discrimination control[J]. Information Sciences, 2018, 425: 18-33.
154	YANG Z, JAIN H, SHI J K, et al. BiasHeal: On-the-fly black-box healing of bias in sentiment analysis systems[C]//2021 IEEE International Conference on Software Maintenance and Evolution. Luxembourg: IEEE, 2021: 644-648.
155	SLACK D, FRIEDLER S A, SCHEIDEGGER C, et al. Assessing the local interpretability of machine learning models[EB/OL]. (2019-02-09)[2022-07-11]. .
156	ZHOU Z Q, SUN L Q, CHEN T Y, et al. Metamorphic relations for enhancing system understanding and use[J]. IEEE Transactions on Software Engineering, 2020, 46(10): 1120-1154.
157	MOLNAR C. Interpretable Machine Learning[M]. Morrisville: Lulu Press, 2019.
158	CHEN H J, JI Y F. Learning variational word masks to improve the interpretability of neural text classifiers[EB/OL]. (2020-10-01)[2022-07-11]. .
159	DING Z Y, WANG Y X, WANG G H, et al. Detecting violations of differential privacy[C]//Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. Toronto: ACM, 2018: 475-489.
160	CARLINI N, JAGIELSKI M, MIRONOV I. Cryptanalytic extraction of neural network models[C]//Annual International Cryptology Conference. Santa Barbara: Springer, 2020: 189-218.
161	LIU J, JUUTI M, LU Y, et al. Oblivious neural network predictions via MiniONN transformations[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. Dallas: ACM, 2017: 619-631.
162	RUAN W J, WU M, SUN Y C, et al. Global robustness evaluation of deep neural networks with provable guarantees for the L0 norm[EB/OL]. (2018-04-16)[2022-07-11]. .
163	MANGAL R, NORI A V, ORSO A. Robustness of neural networks: A probabilistic and practical approach[C]//2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results. Montreal: IEEE, 2019: 93-96.
164	LORENZ T, RUOSS A, BALUNOVIĆ M, et al. Robustness certification for point cloud models[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 7588-7598.
165	BHOJANAPALLI S, CHAKRABARTI A, GLASNER D, et al. Understanding robustness of transformers for image classification[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 10211-10221.
166	MADRY A, MAKELOV A, SCHMIDT L, et al. Towards deep learning models resistant to adversarial attacks[EB/OL]. (2017-06-19)[2022-07-11]. .
167	CARLINI N, KATZ G, BARRETT C, et al. Provably minimally-distorted adversarial examples[EB/OL]. (2017-09-29)[2022-07-11]. .
168	KURAKIN A, GOODFELLOW I, BENGIO S. Adversarial examples in the physical world[EB/OL]. (2016-07-08)[2022-07-11]. .
169	LEE H, HAN S, LEE J. Generative adversarial trainer: Defense to adversarial perturbations with GAN[EB/OL]. (2017-05-09) [2022-07-11]. .
170	WANG J Y, CHEN J L, SUN Y C, et al. RobOT: Robustness-oriented testing for deep learning systems[C]//2021 IEEE/ACM 43rd International Conference on Software Engineering. Madrid: IEEE, 2021: 300-311.
171	KIM J, FELDT R, YOO S. Guiding deep learning system testing using surprise adequacy[C]//2019 IEEE/ACM 41st International Conference on Software Engineering. Montrea: IEEE, 2019: 1039-1049.
172	XU H, CARAMANIS C, MANNOR S. Robustness and regularization of support vector machines[J]. Journal of Machine Learning Research, 2008, 10: 1485-1510.
173	DEMONTIS A, RUSSU P, BIGGIO B, et al. On security and sparsity of linear classifiers for adversarial settings[C]//Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Mérida: Springer, 2016: 322-332.
174	CHEN H, ZHANG H, BONING D, et al. Robust decision trees against adversarial examples[C]//International Conference on Machine Learning. Florida: PMLR, 2019: 1122-1131.
175	XIE X Y, HO J W K, MURPHY C, et al. Testing and validating machine learning classifiers by metamorphic testing[J]. The Journal of Systems and Software, 2011, 84(4): 544-558.
176	DWARAKANATH A, AHUJA M, SIKAND S, et al. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing[C]//Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. Amsterdam: ACM, 2018: 118-128.
177	AL-AZANI S, HASSINE J. Validation of machine learning classifiers using metamorphic testing and feature selection techniques[C]//International Workshop on Multi-disciplinary Trends in Artificial Intelligence. Gadong: Springer, 2017: 77-91.
178	MA L, JUEFEI-XU F, ZHANG F Y, et al. DeepGauge: multi-granularity testing criteria for deep learning systems[C]//2018 33rd IEEE/ACM International Conference on Automated Software Engineering. Montpellier: IEEE, 2018: 120-131.
179	SUN Y C, WU M, RUAN W J, et al. Concolic testing for deep neural networks[C]//Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. Montpellier: ACM, 2018: 109-119.
180	TIAN Y C, PEI K X, JANA S, et al. DeepTest: Automated testing of deep-neural-network-driven autonomous cars[C]//Proceedings of the 40th International Conference on Software Engineering. Gothenburg: ACM, 2018: 303-314.
181	BRAIEK H BEN, KHOMH F. DeepEvolution: A search-based testing approach for deep neural networks[C]//2019 IEEE International Conference on Software Maintenance and Evolution. Cleveland: IEEE, 2019: 454-458.
182	YAN S N, TAO G H, LIU X W, et al. Correlations between deep neural network model coverage criteria and model quality[C]//Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Virtual Conference: ACM, 2020: 775-787.
183	GERASIMOU S, ENISER H F, SEN A, et al. Importance-driven deep learning system testing[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings. Seoul: ACM, 2020: 322-323.
184	XIE X F, MA L, WANG H J, et al. DiffChaser: Detecting disagreements for deep neural networks[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, 2019: 5772-5778.
185	YANG W, XIE T. Telemade: A testing framework for learning-based malware detection systems[C]//Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018: 400-403.
186	CHEN T Y, POON P L, QIU K, et al. Use of metamorphic relations as knowledge carriers to train deep neural networks[EB/OL]. (2021-04-10)[2022-07-11]. .
187	XIE X, GUO W, MA L, et al. RNNrepair: Automatic RNN repair via model-based analysis[C]//Proceedings of the 38th International Conference on Machine Learning. Virtual Conference: PMLR.org, 2021: 11383-11392.
188	AGGARWAL A, LOHIA P, NAGAR S, et al. Black box fairness testing of machine learning models[C]//Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Tallinn: ACM, 2019: 625-635.
189	ZHANG P, WANG J, SUN J, et al. Automatic Fairness Testing of Neural Classifiers through Adversarial Sampling[J]. IEEE Transactions on Software Engineering, 2022: 3593-3612.
190	ZHANG P X, WANG J Y, SUN J, et al. White-box fairness testing through adversarial sampling[C]//2020 IEEE/ACM 42nd International Conference on Software Engineering. Seoul: IEEE, 2020: 949-960.
191	DOSHI-VELEZ F, KIM B. Towards a rigorous science of interpretable machine learning[EB/OL]. (2017-02-28)[2022-07-11]. .
192	CHENG C H, N¨HRENBERG G, HUANG C H, et al. Towards dependability metrics for neural networks[C]//2018 16th ACM/IEEE International Conference on Formal Methods and Models for System Design. Beijing: IEEE, 2018: 1-4.
193	ROSS A, CHEN N N, HANG E Z, et al. Evaluating the interpretability of generative models by interactive reconstruction[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Yokohama: ACM, 2021: 1-15.
194	SCHIELZETH H. Simple means to improve the interpretability of regression coefficients[J]. Methods in Ecology and Evolution, 2010, 1(2): 103-113.
195	CHEN W J, SAHINER B, SAMUELSON F, et al. Calibration of medical diagnostic classifier scores to the probability of disease[J]. Statistical Methods in Medical Research, 2018, 27(5): 1394-1409.
196	KOKHLIKYAN N, MIGLANI V, MARTIN M, et al. Captum: A unified and generic model interpretability library for PyTorch[EB/OL]. (2020-09-16)[2022-07-11]. .
197	YANG Z J, WANG B H, LI H R, et al. On detecting growing-up behaviors of malicious accounts in privacy-centric mobile social networks[C]//Annual Computer Security Applications Conference. Virtual Conference: ACM, 2021: 297-310.
198	BICHSEL B, GEHR T, DRACHSLER-COHEN D, et al. DP-finder: Finding differential privacy violations by sampling and optimization[C]//Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. Toronto: ACM, 2018: 508-524.
199	TRAMÈR F, ZHANG F, JUELS A, et al. Stealing machine learning models via prediction APIs[C]//25th USENIX security symposium (USENIX Security 16). Berkeley: USENIX Association, 2016: 601-618.
200	WANG B H, GONG N Z. Stealing hyperparameters in machine learning[C]//2018 IEEE Symposium on Security and Privacy. San Francisco: IEEE, 2018: 36-52.
201	JAGIELSKI M, CARLINI N, BERTHELOT D, et al. High accuracy and high fidelity extraction of neural networks[C]//Proceedings of the 29th USENIX Conference on Security Symposium. Virtual Conference: USENIX Association, 2020: 1345-1362.
202	XIE P T, BILENKO M, FINLEY T, et al. Crypto-nets: Neural networks over encrypted data[EB/OL]. (2014-12-18)[2022-07-11]. .
203	GILAD-BACHRACH R, DOWLIN N, LAINE K, et al. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy[C]//Proceedings of the 33nd International Conference on Machine Learning. Virtual Conference: JMLR.org, 2016: 201-210.
204	HESAMIFARD E, TAKABI H, GHASEMI M. CryptoDL: Deep neural networks over encrypted data[EB/OL]. (2017-11-14)[2022-07-11]. .
205	LINDELL Y, PINKAS B. Privacy preserving data mining[C]//Advances in Cryptology — CRYPTO 2000. California: Springer, 2000: 36-54.
206	JUVEKAR C, VAIKUNTANATHAN V, CHANDRAKASAN A. GAZELLE: A low latency framework for secure neural network inference[C]//27th USENIX Security Symposium (USENIX Security 18). Berkeley: USENIX Association, 2018: 1651-1669.
207	CHANDRAN N, GUPTA D, RASTOGI A, et al. EzPC: Programmable and efficient secure two-party computation for machine learning[C]//2019 IEEE European Symposium on Security and Privacy. Stockholm: IEEE, 2019: 496-511.
208	ZHENG W, DENG R, CHEN W, et al. Cerebro: A platform for multi-party cryptographic collaborative learning[C]//30th USENIX Security Symposium (USENIX Security 21). Berkeley: USENIX Association, 2021: 2723-2740.
209	KNOTT B, VENKATARAMAN S, HANNUN A, et al. Crypten: Secure multi-party computation meets machine learning[C]//Advances in Neural Information Processing Systems. Virtual Conference: Curran Associates, Inc., 2021: 4961-4973.
210	ISLAM M J, NGUYEN G, PAN R, et al. A comprehensive study on deep learning bug characteristics[C]//Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Tallinn: ACM, 2019: 510-520.
211	JIA L, ZHONG H, WANG X Y, et al. An empirical study on bugs inside tensorflow[C]//International Conference on Database Systems for Advanced Applications. Jeju: Springer, 2020: 604-620.
212	GU J Z, LUO X C, ZHOU Y F, et al. Muffin: Testing deep learning libraries via neural architecture fuzzing[EB/OL]. (2022-04-19)[2022-07-11]. .
213	MURPHY C, SHEN K, KAISER G. Automatic system testing of programs without test oracles[C]//Proceedings of the eighteenth international symposium on Software Testing and Analysis. Chicago: ACM, 2009: 189-200.
214	DING J H, KANG X J, HU X H. Validating a deep learning framework by metamorphic testing[C]/Proceedings of the 2nd International Workshop on Metamorphic Testing. Buenos Aires: IEEE, 2017: 28-34.
215	MA L, ZHANG F Y, SUN J Y, et al. DeepMutation: Mutation testing of deep learning systems[C]//2018 IEEE 29th International Symposium on Software Reliability Engineering. Memphis: IEEE, 2018: 100-111.
216	XIE D N, LI Y T, KIM M, et al. DocTer: Documentation-guided fuzzing for testing deep learning API functions[C]//Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual Conference: ACM, 2022: 176-188.
217	LIU J W, WEI Y X, YANG S, et al. Coverage-guided tensor compiler fuzzing with joint IR-pass mutation[J]. Proceedings of the ACM on Programming Languages, 2022, 6(OOPSLA1): 73(1-26).
218	LIU L, WU Y Z, WEI W Q, et al. Benchmarking deep learning frameworks: Design considerations, metrics and beyond[C]//2018 IEEE 38th International Conference on Distributed Computing Systems. Vienna: IEEE, 2018: 1258-1269.
219	SRISAKAOKUL S, WU Z, ASTORGA A, et al. Multiple-implementation testing of supervised learning software[C]//Workshops at the thirty-second AAAI conference on artificial intelligence. Palo Alto: AAAI Press, 2018: 384-391.
220	WANG Z, YAN M, CHEN J J, et al. Deep learning library testing via effective model generation[C]//Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Virtual Conference: ACM, 2020: 788-799.
221	ZHANG X F, LIU J W, SUN N, et al. Duo: differential fuzzing for deep learning operators[J]. IEEE Transactions on Reliability, 2021, 70(4): 1671-1685.
222	Keras. Keras 2.3.0: This is also the last major release of multi-backend Keras[EB/OL]. (2019-07-18)[2022-07-11]. .
223	MURPHY C, SHEN K, KAISER G. Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles[C]//2009 International Conference on Software Testing Verification and Validation. Denver: IEEE, 2009: 436-445.
224	WANG C J, SHEN J, FANG C R, et al. Accuracy measurement of deep neural network accelerator via metamorphic testing[C]//2020 IEEE International Conference on Artificial Intelligence Testing. Oxford: IEEE, 2020: 55-61.
225	HU Q, MA L, XIE X F, et al. DeepMutation: A mutation testing framework for deep learning systems[C]//2019 34th IEEE/ACM International Conference on Automated Software Engineering. San Diego: IEEE, 2019: 1158-1161.
226	LUO W S, CHAI D, RUN X Y, et al. Graph-based fuzz testing for deep learning inference engines[C]//Proceedings of the 43rd International Conference on Software Engineering. Madrid: IEEE, 2021: 288-299.
227	ZHANG X F, YANG Y L, FENG Y, et al. Software engineering practice in the development of deep learning applications[EB/OL]. (2019-10-08)[2022-07-11]. .
228	ZHANG Y H, CHEN Y F, CHEUNG S C, et al. An empirical study on TensorFlow program bugs[C]//Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. Amsterdam: ACM, 2018: 129-140.
229	CHEN Z P, YAO H H, LOU Y L, et al. An empirical study on deployment faults of deep learning based mobile applications[C]//Proceedings of the 43rd International Conference on Software Engineering. Madrid: IEEE, 2021: 674-685.
230	LAM A N, NGUYEN A T, NGUYEN H A, et al. Bug localization with combination of deep learning and information retrieval[C]//2017 IEEE/ACM 25th International Conference on Program Comprehension. Buenos Aires: IEEE, 2017: 218-229.
231	QI B H, SUN H L, YUAN W, et al. DreamLoc: A deep relevance matching-based framework for bug localization[J]. IEEE Transactions on Reliability, 2022, 71(1): 235-249.

功能描述	方法类别	应用领域	方法描述	效果	相关工作
数据鲁棒性测评	对抗输入生成	图像、文本、音频	生成对抗样本直接测试模型	弱	文献[57]等
数据鲁棒性测评	对抗输入生成	图像、文本、音频	构建对抗输入生成库测试模型鲁棒性	强	文献[58]等
数据鲁棒性修复	随机化	图像	随机化变换调整输入数据	弱	文献[59]
	随机化	图像/数值数据	利用张量衰减调整模型内数据特征	强	文献[60]
	去噪	图像	压缩图像对输入数据进行去噪	弱	文献[61]
	去噪	图像	利用特征压缩的方法对数据去噪	强	文献[62]
	对抗输入检测	图像	基于模型变异检测对变异敏感的对抗样本	弱	文献[63]
	对抗输入检测	图像	评估数据的鲁棒性来区分对抗样本	弱	文献[64]
数据公平性测评	数据偏差测试	数值数据	无监督聚类采样检测数据的类不平衡	弱	文献[65]
		图像	使用自动编码器学习数据特征并检测偏差	强	文献[66]
		数值数据	检测数据分布与特征的倾斜问题	弱	文献[67]
数据公平性修复	数据集修正	主要为数值数据	修复数据集标签或内容	强	文献[68]等
	良性数据生成	图像	生成非歧视性数据以解决训练数据不均衡	强	文献[69]
	良性数据生成	文本	构造良性数据集训练或微调模型	强	文献[70]
	修复框架与工具	数值数据	自动化诊断与修复框架	弱	文献[71]
数据正确性测评	异常数据检测工具	数值数据	检查数据示例并识别特定模式的潜在问题	弱	文献[72]
		主要为数值数据	自动化异常数据检测方法搜索框架	强	文献[73]
		图像	分析特征空间以识别异常数据并进行过滤	弱	文献[74]
数据正确性修复	数据清理工具	图像	基于自动编码器对存在噪声数据进行清理	强	文献[75]
		图像、文本	加入数据检测以在模型计算前剔除异常值	强	文献[76]
		主要为数值数据	自动化搜索数据清理方法并清理异常数据	强	文献[73]
数据隐私性测评	私密信息窃取	主要为数值数据	构造私密数据窃取攻击以测试模型隐私性	弱	文献[77]等
数据隐私性修复	基于差分隐私的数据隐私保护	图像	训练多个教师模型并聚合预测结果	强	文献[78]
	基于安全多方计算的数据隐私保护	图像、数值数据	基于安全多方计算协议交互私密数据	强	文献[79]等
	基于联邦学习的数据隐私保护	图像、数值数据	通过安全聚合等方法构建联邦学习训练模型	强	文献[80]等

功能描述	方法类别	应用领域	方法描述	效果	相关工作
数据鲁棒性测评	对抗输入生成	图像、文本、音频	生成对抗样本直接测试模型	弱	文献[57]等
数据鲁棒性测评	对抗输入生成	图像、文本、音频	构建对抗输入生成库测试模型鲁棒性	强	文献[58]等
数据鲁棒性修复	随机化	图像	随机化变换调整输入数据	弱	文献[59]
	随机化	图像/数值数据	利用张量衰减调整模型内数据特征	强	文献[60]
	去噪	图像	压缩图像对输入数据进行去噪	弱	文献[61]
	去噪	图像	利用特征压缩的方法对数据去噪	强	文献[62]
	对抗输入检测	图像	基于模型变异检测对变异敏感的对抗样本	弱	文献[63]
	对抗输入检测	图像	评估数据的鲁棒性来区分对抗样本	弱	文献[64]
数据公平性测评	数据偏差测试	数值数据	无监督聚类采样检测数据的类不平衡	弱	文献[65]
		图像	使用自动编码器学习数据特征并检测偏差	强	文献[66]
		数值数据	检测数据分布与特征的倾斜问题	弱	文献[67]
数据公平性修复	数据集修正	主要为数值数据	修复数据集标签或内容	强	文献[68]等
	良性数据生成	图像	生成非歧视性数据以解决训练数据不均衡	强	文献[69]
	良性数据生成	文本	构造良性数据集训练或微调模型	强	文献[70]
	修复框架与工具	数值数据	自动化诊断与修复框架	弱	文献[71]
数据正确性测评	异常数据检测工具	数值数据	检查数据示例并识别特定模式的潜在问题	弱	文献[72]
		主要为数值数据	自动化异常数据检测方法搜索框架	强	文献[73]
		图像	分析特征空间以识别异常数据并进行过滤	弱	文献[74]
数据正确性修复	数据清理工具	图像	基于自动编码器对存在噪声数据进行清理	强	文献[75]
		图像、文本	加入数据检测以在模型计算前剔除异常值	强	文献[76]
		主要为数值数据	自动化搜索数据清理方法并清理异常数据	强	文献[73]
数据隐私性测评	私密信息窃取	主要为数值数据	构造私密数据窃取攻击以测试模型隐私性	弱	文献[77]等
数据隐私性修复	基于差分隐私的数据隐私保护	图像	训练多个教师模型并聚合预测结果	强	文献[78]
	基于安全多方计算的数据隐私保护	图像、数值数据	基于安全多方计算协议交互私密数据	强	文献[79]等
	基于联邦学习的数据隐私保护	图像、数值数据	通过安全聚合等方法构建联邦学习训练模型	强	文献[80]等

功能描述	方法类别	应用领域	方法描述	效果	相关工作
算法鲁棒性测评	鲁棒性评估与测试准则	图像	计算欺骗模型的最小扰动	强	文献[83]
		图像	识别模型输入空间鲁棒区域	弱	文献[84]
		文本	利用对抗样本等多种范式评估模型	强	文献[138]
算法鲁棒性修复	对抗训练	主要为图像	使用对抗样本重训练模型	强	文献[57]等
算法鲁棒性修复	鲁棒优化	主要为图像	使用正则化方法处理并优化模型,削弱扰动影响	强	文献[139]等
算法正确性测评	模型差异测试	图像	通过白盒交叉验证方法测试流行模型的差异行为	弱	文献[140]
	模型差异测试	图像	变异模糊测试并最大化原始与变异输入的差异	强	文献[141]
	模型蜕变测试	数值数据	利用蜕变测试的方法测试机器学习模型属性	弱	文献[142]
		图像	设计了多个通用蜕变关系测试机器学习系统特征	强	文献[143]
		文本	针对NLP系统设计了蜕变关系并测试	弱	文献[144]
	测试充分性评估	图像	基于覆盖率的模糊测试和基于属性的测试结合	弱	文献[26]
	测试充分性评估	图像	利用神经元覆盖率等覆盖率准则进行模糊测试	强	文献[24]
	模型调试	图像	分析模型差分状态并识别模型"故障神经元"	强	文献[145]
算法正确性修复	重训练	图像	生成并机器学习系统的异常行为样例并重训练	弱	文献[140]
		图像	基于神经风格转换学习故障样本并重训练模型	弱	文献[146]
		图像、文本	应用多种策略修复模型训练问题并重训练	强	文献[25]
	模型调试修复	图像、文本	构建影响模型描述网络中数据的状态并分析错误	强	文献[147]
	模型调试修复	主要为数值数据	调试机器学习模型算法故障并定位问题的原因	弱	文献[148]
算法公平性测评	公平性测试工具/框架	数值数据	结合了多个指标细粒度探索偏差并进行严格评估	弱	文献[149]
		数值数据	自动化生成包含敏感属性的输入并测试歧视问题	弱	文献[150]
		数值数据	在输入空间随机抽样歧视性样例并在邻域搜索	强	文献[151]
		数值数据	通过分析模型行为以发现潜在的群体公平性问题	强	文献[152]
算法公平性修复	处理中修复	数值数据	将发掘的歧视性样例放入数据集并进行重训练	弱	文献[151]
		数值数据	将公平性作为机器学习模型优化目标性	强	文献[113]
		数值数据	丢弃部分公平性与准确率优化方向矛盾神经元	强	文献[27]
	后处理修复	数值数据	拒绝对接近决策边界的输出样本	弱	文献[153]
	后处理修复	文本	自动检测并修复输出偏差结果并重构公平输出	弱	文献[154]
算法可解释性测评	人工可解释性测评	数值数据	调研参与者在输入变化下给出模型预期输出变化	弱	文献[155]
算法可解释性测评	自动化可解释性测评	数值数据	设计蜕变关系评测系统功能可解释性	弱	文献[156]
算法可解释性修复	可解释性提升	数值数据	使用可解释性强的算法构建模型	弱	文献[157]
算法可解释性修复	可解释性提升	文本数据	自动学习任务中重要文字并减少无关信息	强	文献[158]
算法隐私性测评	隐私性评估	数值数据	多次运行候选算法并统计对算法隐私的侵犯程度	弱	文献[159]
算法隐私性测评	模型萃取攻击	图像	通过查询ReLU临界点的查询窃取模型参数信息	强	文献[160]
算法隐私性修复	基于加密的算法隐私保护	数值数据、图像	基于同态加密等方法对模型组件设计加密算法	强	文献[161]等
算法隐私性修复	基于安全多方计算隐私保护	数值数据、图像	设计安全多方计算协议保障模型算法信息隐私性	强	文献[80]等

功能描述	方法类别	应用领域	方法描述	效果	相关工作
算法鲁棒性测评	鲁棒性评估与测试准则	图像	计算欺骗模型的最小扰动	强	文献[83]
		图像	识别模型输入空间鲁棒区域	弱	文献[84]
		文本	利用对抗样本等多种范式评估模型	强	文献[138]
算法鲁棒性修复	对抗训练	主要为图像	使用对抗样本重训练模型	强	文献[57]等
算法鲁棒性修复	鲁棒优化	主要为图像	使用正则化方法处理并优化模型,削弱扰动影响	强	文献[139]等
算法正确性测评	模型差异测试	图像	通过白盒交叉验证方法测试流行模型的差异行为	弱	文献[140]
	模型差异测试	图像	变异模糊测试并最大化原始与变异输入的差异	强	文献[141]
	模型蜕变测试	数值数据	利用蜕变测试的方法测试机器学习模型属性	弱	文献[142]
		图像	设计了多个通用蜕变关系测试机器学习系统特征	强	文献[143]
		文本	针对NLP系统设计了蜕变关系并测试	弱	文献[144]
	测试充分性评估	图像	基于覆盖率的模糊测试和基于属性的测试结合	弱	文献[26]
	测试充分性评估	图像	利用神经元覆盖率等覆盖率准则进行模糊测试	强	文献[24]
	模型调试	图像	分析模型差分状态并识别模型"故障神经元"	强	文献[145]
算法正确性修复	重训练	图像	生成并机器学习系统的异常行为样例并重训练	弱	文献[140]
		图像	基于神经风格转换学习故障样本并重训练模型	弱	文献[146]
		图像、文本	应用多种策略修复模型训练问题并重训练	强	文献[25]
	模型调试修复	图像、文本	构建影响模型描述网络中数据的状态并分析错误	强	文献[147]
	模型调试修复	主要为数值数据	调试机器学习模型算法故障并定位问题的原因	弱	文献[148]
算法公平性测评	公平性测试工具/框架	数值数据	结合了多个指标细粒度探索偏差并进行严格评估	弱	文献[149]
		数值数据	自动化生成包含敏感属性的输入并测试歧视问题	弱	文献[150]
		数值数据	在输入空间随机抽样歧视性样例并在邻域搜索	强	文献[151]
		数值数据	通过分析模型行为以发现潜在的群体公平性问题	强	文献[152]
算法公平性修复	处理中修复	数值数据	将发掘的歧视性样例放入数据集并进行重训练	弱	文献[151]
		数值数据	将公平性作为机器学习模型优化目标性	强	文献[113]
		数值数据	丢弃部分公平性与准确率优化方向矛盾神经元	强	文献[27]
	后处理修复	数值数据	拒绝对接近决策边界的输出样本	弱	文献[153]
	后处理修复	文本	自动检测并修复输出偏差结果并重构公平输出	弱	文献[154]
算法可解释性测评	人工可解释性测评	数值数据	调研参与者在输入变化下给出模型预期输出变化	弱	文献[155]
算法可解释性测评	自动化可解释性测评	数值数据	设计蜕变关系评测系统功能可解释性	弱	文献[156]
算法可解释性修复	可解释性提升	数值数据	使用可解释性强的算法构建模型	弱	文献[157]
算法可解释性修复	可解释性提升	文本数据	自动学习任务中重要文字并减少无关信息	强	文献[158]
算法隐私性测评	隐私性评估	数值数据	多次运行候选算法并统计对算法隐私的侵犯程度	弱	文献[159]
算法隐私性测评	模型萃取攻击	图像	通过查询ReLU临界点的查询窃取模型参数信息	强	文献[160]
算法隐私性修复	基于加密的算法隐私保护	数值数据、图像	基于同态加密等方法对模型组件设计加密算法	强	文献[161]等
算法隐私性修复	基于安全多方计算隐私保护	数值数据、图像	设计安全多方计算协议保障模型算法信息隐私性	强	文献[80]等

功能描述	方法类别	测试框架	方法描述	效果	相关工作
实现正确性测评	实现差异测试	TenorFlow, CNTK, Theano	对比框架同一功能在输入下的输出差异	弱	文献[52]
		TenorFlow, CNTK, Theano	基于模糊测试生成不同的模型以探索框架	弱	文献[212]
		TensorFlow, PyTorch	构造等效图对同一功能实现进行对比	强	文献[53]
	实现蜕变测试	Weka, C4.5等	检查蜕变关系执行前后一致并自动化测试	弱	文献[213]
	实现蜕变测试	Scikit-learn, TensorFlow	基于增减数据等蜕变关系测试框架实现	弱	文献[214]
	测试样本生成方法	TensorFlow, Keras	基于模型突变测试的方法检测实现问题	弱	文献[215]
	测试样本生成方法	TensorFlow, PyTorch, MXNet	自动化提取框架功能约束并生成样例	强	文献[216]
	框架漏洞研究	TensorFlow, Torch等	调研开源社区上的机器学习框架漏洞特性	弱	文献[210]
	框架底层库测试	TVM	覆盖度指导变异低级中间表示以模糊测试	强	文献[217]
	框架底层库测试	TensorFlow	对算子误差进行了计算评估并与实际对比	强	文献[54]
实现效率测评	效率问题实证研究	TensorFlow, Caffe, Torch	实证研究不同框架上训练时间等性能差异	弱	文献[218]
实现效率测评	效率问题实证研究	TensorFlow, CNTK, PyTorch, MXNet	测试部署环境迁移对实现的性能影响	弱	文献[48]

面向机器学习模型安全的测试与修复

The Testing and Repairing Methods for Machine Learning Model Security

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 231

相关文章 15

编辑推荐

Metrics

机器学习模型安全测试相关综述	参考文献最新年份	涵盖方法内容
		鲁棒性		正确性		公平性		效率		可解释性		隐私性
		测试	修复	测试	修复	测试	修复	测试	修复	测试	修复	测试	修复
文献[33]	2019	—	—	√	√	—	—	—	—	—	—	—	—
文献[34]	2019	—	—	—	—	√	√	—	—	—	—	—	—
文献[30]	2019	√	√	—	—	—	—	—	—	—	—	√	√
文献[31]	2020	√	√	√	√	—	—	—	—	—	—	—	—
文献[32]	2020	√	—	√	—	√	—	√	—	√	—	√	—
本文	2022	√	√	√	√	√	√	√	√	√	√	√	√

测试属性	测试阶段	特性描述
正确性	数据、算法、实现	模型正确行使功能并完成任务的能力
鲁棒性	数据	模型在输入干扰下正确运行的能力
公平性	数据、算法	模型不受敏感输入属性的影响的能力
效率	实现	模型执行完成指定任务的开销
可解释性	算法	模型的决策可以被观察者理解的能力
隐私性	数据、算法	模型保护相关私密数据的能力

[1]	李志博, 李清宝, 兰明敬, 孙剑帆. 基于镜像选择序优化的MART算法[J]. 电子学报, 2022, 50(2): 314-325.
[2]	范书平, 张岩, 马宝英, 万里, 姚念民, 宋妍. 基于均衡优化理论的路径覆盖测试数据进化生成[J]. 电子学报, 2020, 48(7): 1303-1310.
[3]	夏春艳, 张岩, 万里, 宋妍, 肖楠, 郭冰. 基于否定选择遗传算法的路径覆盖测试数据生成[J]. 电子学报, 2019, 47(12): 2630-2638.
[4]	巩敦卫, 秦备, 田甜. 基于语句重要度的变异测试对象选择方法[J]. 电子学报, 2017, 45(6): 1518-1522.
[5]	赵祖威, 冯世宁, 汤恩义, 陈鑫, 李宣东, 潘敏学, 赵晨. 一种符号执行制导的循环内界分析方法[J]. 电子学报, 2017, 45(11): 2582-2592.
[6]	廖伟志. 基于路径自动分割的测试数据生成方法[J]. 电子学报, 2016, 44(9): 2254-2261.
[7]	王红阳, 姜淑娟, 王兴亚, 鞠小林, 张艳梅. 基于子路径扩展的不可达路径检测方法[J]. 电子学报, 2015, 43(8): 1555-1560.
[8]	聂楚江, 刘海峰, 苏璞睿, 冯登国. 一种面向程序动态分析的循环摘要生成方法[J]. 电子学报, 2014, 42(6): 1110-1117.
[9]	张岩;巩敦卫. 基于搜索空间自动缩减的路径覆盖测试数据进化生成[J]. 电子学报, 2012, 40(5): 1011-1016.
[10]	王雅文;宫云战;肖庆;杨朝红. 基于抽象解释的变量值范围分析及应用[J]. 电子学报, 2011, 39(2): 296-303.
[11]	巩敦卫;张岩. 一种新的多路径覆盖测试数据进化生成方法[J]. 电子学报, 2010, 38(6): 1299-1304.
[12]	李书浩, 王戟, 齐治昌, 董威, . 一种面向性质的实时系统测试方法[J]. 电子学报, 2005, 33(5): 827-834.
[13]	缪力, 张大方, 季洁, 宣恒农. 非定态路径测试问题的分析与一种转换算法[J]. 电子学报, 2005, 33(2): 258-261.
[14]	李书浩, 王戟, 董威, 齐治昌. 反应式系统面向性质测试的方法框架[J]. 电子学报, 2004, 32(S1): 226-230.
[15]	刘晖, 李明禄. 基于抽象状态机的网格系统设计和分析[J]. 电子学报, 2003, 31(S1): 2096-2100.