通用CPU性能基准测试研究综述

史惠康; 王泽胜; 张士宗; 高翔; 赵有健

doi:10.12263/DZXB.20220169

您当前的位置：

首页 >

文章列表页 >

通用CPU性能基准测试研究综述

综述评论 | 更新时间：2025-07-02

- 通用CPU性能基准测试研究综述
- Performance Evaluation Benchmark of General-Purpose CPU：A Survey
- 电子学报 2023年51卷第1期页码：246-256
- 作者机构：
  
  1.清华大学计算机科学与技术系，北京 100084
  2.中国电子技术标准化研究院，北京 100007
  3.龙芯中科技术股份有限公司，北京 100095
- 作者简介：
  
  [ "史惠康男，1975年2月出生，山西岢岚人.1999年在中国科学院计算技术研究所获工学硕士学位，现为清华大学计算机科学与技术系博士研究生.主要从事电子与信息方面的研究工作.E-mail: shk@pku.org.cn" ]
  [ "王泽胜男，1987年1月出生，河北保定人.2018年在北京交通大学获工学博士学位，现为中国电子技术标准化研究院软件应用与服务研究中心工程师.主要从事云计算、信息技术服务等方面的研究工作.E-mail: 815345591@163.com" ]
  [ "张士宗男，1989年4月出生，山东东营人.2018年在北京邮电大学获工学博士学位，现为中国电子技术标准化研究院信息技术研究中心工程师.主要从事计算性能基准测试、计算机网络应用等方面的研究工作.E-mail: zhangsz@cesi.cn" ]
  [ "高翔男，1982年出生，湖北荆州人.教授级高级工程师.2007年在中国科学技术大学获工学博士学位.现为龙芯中科技术股份有限公司副总经理.主要从事高性能计算机体系结构、并行处理和操作系统等方面的研究工作.E-mail: gaoxiang@loongson.cn" ]
  [ "赵有健（通讯作者）男，1969年出生，甘肃会宁人.1999年在东北大学获工学博士学位，现为清华大学计算机科学与技术系教授、博士生导师.主要从事高速互联网体系结构、交换与路由和高速网络设备等方面的研究工作." ]
- 基金信息：
  
  本刊2021年第49卷第12期《基于迹变换和旋转增量调制特征的模糊人脸识别》（作者：汪宇玲，陈立，黎明，钟国韵，何月顺，常玉祥，宋伟宁）一文中;应为“国家自然科学基金(62066003;61866025;41872243);国家重点研发计划(2018YFB1702702);江西省核地学数据科学与系统工程技术研究中心开放基金(JETRCNGDSS202006┫”.特此更正)
- DOI：10.12263/DZXB.20220169
  中图分类号： TP306;
- 收稿：2022-02-15，
  
  修回：2022-10-30，
  
  纸质出版：2023-01-25
- 稿件说明：
移动端阅览
史惠康,王泽胜,张士宗等.通用CPU性能基准测试研究综述[J].电子学报,2023,51(01):246-256.

SHI Hui-kang,WANG Ze-sheng,ZHANG Shi-zong,et al.Performance Evaluation Benchmark of General-Purpose CPU：A Survey[J].ACTA ELECTRONICA SINICA,2023,51(01):246-256.
史惠康,王泽胜,张士宗等.通用CPU性能基准测试研究综述[J].电子学报,2023,51(01):246-256. DOI： 10.12263/DZXB.20220169.

SHI Hui-kang,WANG Ze-sheng,ZHANG Shi-zong,et al.Performance Evaluation Benchmark of General-Purpose CPU：A Survey[J].ACTA ELECTRONICA SINICA,2023,51(01):246-256. DOI： 10.12263/DZXB.20220169.

摘要

CPU性能基准测试旨在给出可对比、定量的指标数据，为产品选型提供依据，它已成为引领计算产业发展的风向标之一.CPU技术发展迅速，性能基准测试也在不断演进.本文对包含SPEC CPU在内的主流基准测试进行了研究，从测试目标、测试方法等角度，综述主流CPU基准测试的演进过程、最新研究成果，以及通用CPU性能指标和基准测试需求，分析了通用CPU性能基准测试所面临的挑战，并对今后可能的研究趋势进行了展望.

Abstract

CPU performance evaluation benchmark aims to provide comparative and quantitative index data for product selection. It is one of the vane leading the development of computing industry

and as CPU technology evolves rapidly

performance benchmarks are evolving. This paper systematically reviews the mainstream benchmarks including the SPEC CPU. From the perspectives of evaluation objectives and methods

the evolution

recent research results of the mainstream CPU benchmarks

and the performance metrics and benchmark requirements of general-purpose CPU are reviewed. Finally

this paper analyzes the challenges of general-purpose CPU performance evaluation benchmarks and prospects for possible future research trends.

关键词

Keywords

references

TICHY W F , LUKOWICZ P , PRECHELT L , et al . Experimental evaluation in computer science: A quantitative study [J]. Journal of Systems and Software , 1995 , 28 ( 1 ): 9 - 18 .

TAO J H , DU Z D , GUO Q , et al . BenchIP: Benchmarking intelligence processors [J]. Journal of Computer Science and Technology , 2018 , 33 ( 1 ): 1 - 23 .

THAKKAR P , NATHAN S , VISWANATHAN B . Performance benchmarking and optimizing hyperledger fabric blockchain platform [C]// 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems . Milwaukee : IEEE , 2018 : 264 - 276 .

王文凯 . 基于DNN的智能芯片性能评估及优化 [D]. 武汉 : 华中科技大学 , 2018 .

WANG W K . Intelligent Chip Performance Evaluation and Optimization Based on DNN [D]. Wuhan : Huazhong University of Science and Technology , 2018 . (in Chinese)

HSIEH S Y , LIU C S , BUYYA R , et al . Utilization-prediction-aware virtual machine consolidation approach for energy-efficient cloud data centers [J]. Journal of Parallel and Distributed Computing , 2020 , 139 : 99 - 109 .

NIDER J , FEDOROVA A S . The last CPU [C]// Proceedings of the Workshop on Hot Topics in Operating Systems . New York : ACM , 2021 : 1 - 8 .

PAWANEKAR S , UDGIRKAR G . Performance of Reinforcement Learning Simulation: X86 vs ARM [M]// Communications in Computer and Information Science . Cham : Springer International Publishing , 2021 : 420 - 430 .

MATHÁ R , KIMOVSKI D , ZABROVSKIY A , et al . Where to encode: A performance analysis of x86 and arm-based Amazon EC2 instances [C]// 2021 IEEE 17th International Conference on eScience . Innsbruck : IEEE , 2021 : 118 - 127 .

LIMAYE A , ADEGBIJA T . A workload characterization of the SPEC CPU2017 benchmark suite [C]// 2018 IEEE International Symposium on Performance Analysis of Systems and Software . Belfast : IEEE , 2018 : 149 - 158 .

BACH M , KRETZ M , LINDENSTRUTH V , et al . Optimized HPL for AMD GPU and multi-core CPU usage [J]. Computer Science - Research and Development , 2011 , 26 ( 3 ): 153 - 164 .

KALYANASUNDARAM K . SPEC HPG benchmarks [C]// Proceedings of the 2006 ACM/IEEE Conference on Supercomputing . Tampa : ACM , 2006 : 17-es.

WEISS A R . The standardization of embedded benchmarking: Pitfalls and opportunities [C]// Proceedings 1999 IEEE International Conference on Computer Design: VLSI in Computers and Processors(Cat. No.99CB37040) . Austin : IEEE , 1999 : 492 - 508 .

CAPRA M , BUSSOLINO B , MARCHISIO A , et al . Hardware and software optimizations for accelerating deep neural networks: Survey of current trends, challenges, and the road ahead [J]. IEEE Access , 8 : 225134 - 225180 .

SEN A , DENIZ E . Thread-level synthetic benchmarks for multicore systems [J]. Microprocessors and Microsystems , 2015 , 39 ( 7 ): 471 - 479 .

KARKI A , KESHAVA C P , SHIVAKUMAR S M , et al . Tango: A deep neural network benchmark suite for various accelerators [EB/OL]. ( 2019 )[2022]. https://arxiv.org/abs/1901.04987 https://arxiv.org/abs/1901.04987 .

LEWIS B C , CREWS A E . The evolution of benchmarking as a computer performance evaluation technique [J]. MIS Quarterly , 1985 , 9 ( 1 ): 7 - 16 .

WEICKER R P . An overview of common benchmarks [J]. Computer , 1990 , 23 ( 12 ): 65 - 75 .

JOHN L K , EECKHOUT L . Performance Evaluation and Benchmarking [M]. Boca Raton : CRC Press , 2006 .

CHONG N , SORENSEN T , WICKERSON J . The semantics of transactions and weak memory in x86, power, ARM, and C++ [C]// PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation . Philadelphia : ACM Press , 2018 : 211 - 225 .

Standard Performance Evaluation Corporation . SPEC CPU 2017 [EB/OL]. [2022] . http://www.spec.org/cpu2017/ http://www.spec.org/cpu2017/ .

AMARAL J N , BORIN E , ASHLEY D R , et al . The Alberta workloads for the SPEC CPU 2017 benchmark suite [C]// 2018 IEEE International Symposium on Performance Analysis of Systems and Software . Belfast : IEEE , 2017 : 159 - 168 .

DIXIT K M . New CPU benchmark suites from SPEC [C]// Digest of Papers COMPCON Spring . San Francisco : IEEE , 1992 : 305 - 310 .

SIMON J , VIETH M , WEICKER R . Workload Analysis of Computation Intensive Tasks: Case Study on SPEC CPU95 Benchmarks [M]// Euro-Par'97 Parallel Processing . Berlin, Heidelberg : Springer , 1997 : 971 - 984 .

廖秋林 , 莫玮 , 陈大为 . SPEC CPU2000性能测试程序分析及其应用 [J]. 国外电子测量技术 , 2006 , 25 ( 6 ): 65 - 68 .

LIAO Q L , MO W , CHEN D W . Analysis and application of SPEC CPU2000 performance test program [J]. Foreign Electronic Measurement Technology , 2006 , 25 ( 6 ): 65 - 68 . (in Chinese)

NAIR A A , JOHN L K . Simulation points for SPEC CPU 2006 [C]// 2008 IEEE International Conference on Computer Design . Lake Tahoe : IEEE , 2008 : 397 - 403 .

PANDA R , SONG S , DEAN J , et al . Wait of a decade: Did SPEC CPU 2017 broaden the performance horizon? [C]// 2018 IEEE International Symposium on High Performance Computer Architecture . Vienna : IEEE , 2018 : 271 - 282 .

SONG S , WU Q Z , FLOLID S , et al . Experiments with SPEC CPU 2017: Similarity, balance, phase behavior and simPoints [EB/OL]. ( 2018 )[2022]. http://lca.ece.utexas.edu/pubs/UT_LCA_TR-180515-01.pdf http://lca.ece.utexas.edu/pubs/UT_LCA_TR-180515-01.pdf .

SINGH S , AWASTHI M . Memory centric characterization and analysis of SPEC CPU2017 suite [C]// Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering . New York : ACM , 2019 : 285 - 292 .

BUCEK J , LANGE K D , KISTOWSKI J V . SPEC CPU2017: Next-generation compute benchmark [C]// ICPE'18: Companion of the 2018 ACM/SPEC International Conference on Performance Engineering . Berlin : ACM , 2018 : 41 - 42 .

WALLACE R . Performance Benchmark and assessment of a mixed batch and real-time transaction processing system [C]// Proceedings of Computer Measurement Group Conference . Orlando : DBLP Computer Science Bibliography , 1997 : 863 - 872 .

GRAY J . A view of database system performance measures [J]. ACM SIGMETRICS Performance Evaluation Review , 1987 , 15 ( 1 ): 3 - 4 .

LEVINE C . Why TPC-A and TPC-B are obsolete [C]// Digest of Papers. Compcon Spring . San Francisco : IEEE , 1993 : 215 - 221 .

LEUTENEGGER S T , DIAS D . Modeling study of the TPC-C benchmark [J]. ACM SIGMOD Record , 1993 , 22 ( 2 ): 22 - 31 .

CHEN S M , AILAMAKI A , ATHANASSOULIS M , et al . TPC-E vs TPC-C: Characterizing the new TPC-E benchmark via an I/O comparison study [J]. SIGMOD Record , 2010 , 39 ( 3 ): 5 - 10 .

KANDASWAMY M A , KNIGHTEN R L . I/O phase characterization of TPC-H query operations [C]// Proceedings IEEE International Computer Performance and Dependability Symposium . IPDS . Chicago : IEEE , 2000 : 81 - 90 .

TRIVEDI M , CHEN Z Q . Lessons Learned from the Industry's First TPC Benchmark DS(TPC-DS) [M]// Performance Evaluation and Benchmarking for the Era of Artificial Intelligence . Cham : Springer International Publishing , 2019 : 140 - 154 .

DEEHR E , FANG W Q , REZA TAHERI H , et al . Performance Analysis of Database Virtualization with the TPC-VMS Benchmark [M]// Lecture Notes in Computer Science . Cham : Springer International Publishing , 2015 : 156 - 172 .

刘建鹏 , 刘尧 . 将TPC-DS工具合入HiBench测试框架的方法 [J]. 数字技术与应用 , 2019 , 37 ( 10 ): 64 - 65 .

LIU J P , LIU Y . Method of incorporating TPC-DS tools into the HiBench test framework [J]. Digital Technology & Application , 2019 , 37 ( 10 ): 64 - 65 . (in Chinese)

ZHAI J D , ZHANG F , LI Q W , et al . Characterizing and optimizing TPC-C workloads on large-scale systems using SSD arrays [J]. Science China Information Sciences , 2016 , 59 ( 9 ): 92104 .

冯志丹 . 基于SCF中间件的TPC-C测试系统的设计和开发 [D]. 北京 : 北京邮电大学 , 2016 .

FENG Z D . The Design and Development of TPC-C Test System Based on SCF Middleware [D]. Beijing : Beijing University of Posts and Telecommunications , 2016 . (in Chinese)

Labs Primate . Geekbench 5 . 1 . 1 [EB/OL]. [2022] . https://www.geekbench.com/blog/2020/04/geekbench-511 https://www.geekbench.com/blog/2020/04/geekbench-511 .

CORNERO M , ANYURU A . Multiprocessing in Mobile Platforms: The Marketing and the Reality [R]. Genève : ST-ERICSSON , 2013 .

KOZHIRBAYEV Z , SINNOTT R O . A performance comparison of container-based technologies for the Cloud [J]. Future Generation Computer Systems , 2017 , 68 : 175 - 182 .

POLVINEN T , YLIKÄNNÖ T , MÄKELÄINEN A , et al . Building a virtualized environment for programming courses [C]// WorldCIST 2020: Trends and Innovations in Information Systems and Technologies(AISC , volume 1160 ). Cham : Springer , 2020 : 45 - 55 .

MORABITO R , KJÄLLMAN J , KOMU M . Hypervisors vs. lightweight virtualization: A performance comparison [C]// 2015 IEEE International Conference on Cloud Engineering . Tempe : IEEE , 2015 : 386 - 393 .

BARKER A , VARGHESE B , THAI L . Cloud services brokerage: A survey and research roadmap [C]// 2015 IEEE 8th International Conference on Cloud Computing . New York : IEEE , 2015 : 1029 - 1032 .

WANG Y , LEE V , WEI G Y , et al . Predicting new workload or CPU performance by analyzing public datasets [J]. ACM Transactions on Architecture and Code Optimization , 2019 , 15 ( 4 ): 1 - 21 .

SINGH J P , WEBER W D , GUPTA A . SPLASH: Stanford parallel applications for shared-memory [J]. SIGARCH Computer Architecture News , 1992 , 20 ( 1 ): 5 - 44 .

WOO S C , OHARA M , TORRIE E , et al . The SPLASH-2 programs: characterization and methodological considerations [C]// Proceedings of the 22nd Annual International Symposium on Computer Architecture . Santa Margherita Ligure : IEEE , 1995 : 24 - 36 .

SAKALIS C , LEONARDSSON C , KAXIRAS S , et al . Splash-3: A properly synchronized benchmark suite for contemporary research [C]// 2016 IEEE International Symposium on Performance Analysis of Systems and Software . Uppsala : IEEE , 2016 : 101 - 111 .

BIENIA C , KUMAR S , SINGH J P , et al . The PARSEC benchmark suite: Characterization and architectural implications [C]// PACT'08: Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques . Toronto : IEEE , 2008 : 72 - 81 .

BIENIA C , KUMAR S , LI K . PARSEC vs. SPLASH-2: A quantitative comparison of two multithreaded benchmark suites on Chip-Multiprocessors [C]// 2008 IEEE International Symposium on Workload Characterization . Seattle : IEEE , 2008 : 47 - 56 .

BARROW-WILLIAMS N , FENSCH C , MOORE S . A communication characterisation of splash-2 and parsec [C]// 2009 IEEE International Symposium on Workload Characterization . Austin : IEEE , 2009 : 86 - 97 .

BIENIA C , LI K . Fidelity and scaling of the PARSEC benchmark inputs [C]// IEEE International Symposium on Workload Characterization . Atlanta : IEEE , 2010 : 1 - 10 .

CHASAPIS D , CASAS M , MORETÓ M , et al . PARSECSs: Evaluating the impact of task parallelism in the PARSEC benchmark suite [J]. ACM Transactions on Architecture and Code Optimization , 2016 , 12 ( 4 ): 41 .

CEBRIAN J M , JAHRE M , NATVIG L . ParVec: Vectorizing the PARSEC benchmark suite [J]. Computing , 2015 , 97 ( 11 ): 1077 - 1100 .

HUYNH A , HELM C , IWASAKI S , et al . TP-PARSEC: A task parallel PARSEC benchmark suite [J]. Journal of Information Processing , 2019 , 27 : 211 - 220 .

HEINECKE A , VAIDYANATHAN K , SMELYANSKIY M , et al . Design and implementation of the linpack benchmark for single and multi-node systems based on intel® xeon phi coprocessor [C]// 2013 IEEE 27th International Symposium on Parallel and Distributed Processing . Cambridge : IEEE , 2013 : 126 - 137 .

BLIN A , COURTAUD C , SOPENA J , et al . Understanding the memory consumption of the MiBench embedded benchmark [C]// NETYS 2016: Networked Systems(LNCCN , volume 9944 ). Cham : Springer , 2016 : 71 - 86 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

类不平衡对软件缺陷预测模型稳定性和预测性能的影响分析方法