Performance Evaluation Benchmark of General-Purpose CPU：A Survey

SHI Hui-kang; WANG Ze-sheng; ZHANG Shi-zong; GAO Xiang; ZHAO You-jian

doi:10.12263/DZXB.20220169

您当前的位置：

首页 >

文章列表页 >

Performance Evaluation Benchmark of General-Purpose CPU：A Survey

SURVEYS AND REVIEWS | 更新时间：2025-07-02

- Performance Evaluation Benchmark of General-Purpose CPU：A Survey
- ACTA ELECTRONICA SINICA Vol. 51, Issue 1, Pages: 246-256(2023)
- 作者机构：
  
  1.清华大学计算机科学与技术系，北京 100084
  2.中国电子技术标准化研究院，北京 100007
  3.龙芯中科技术股份有限公司，北京 100095
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20220169
  CLC： TP306;
- Received：15 February 2022，
  
  Revised：2022-10-30，
  
  Published：25 January 2023
- 稿件说明：
移动端阅览
史惠康,王泽胜,张士宗等.通用CPU性能基准测试研究综述[J].电子学报,2023,51(01):246-256.

SHI Hui-kang,WANG Ze-sheng,ZHANG Shi-zong,et al.Performance Evaluation Benchmark of General-Purpose CPU：A Survey[J].ACTA ELECTRONICA SINICA,2023,51(01):246-256.
史惠康,王泽胜,张士宗等.通用CPU性能基准测试研究综述[J].电子学报,2023,51(01):246-256. DOI： 10.12263/DZXB.20220169.

SHI Hui-kang,WANG Ze-sheng,ZHANG Shi-zong,et al.Performance Evaluation Benchmark of General-Purpose CPU：A Survey[J].ACTA ELECTRONICA SINICA,2023,51(01):246-256. DOI： 10.12263/DZXB.20220169.

摘要

CPU性能基准测试旨在给出可对比、定量的指标数据，为产品选型提供依据，它已成为引领计算产业发展的风向标之一.CPU技术发展迅速，性能基准测试也在不断演进.本文对包含SPEC CPU在内的主流基准测试进行了研究，从测试目标、测试方法等角度，综述主流CPU基准测试的演进过程、最新研究成果，以及通用CPU性能指标和基准测试需求，分析了通用CPU性能基准测试所面临的挑战，并对今后可能的研究趋势进行了展望.

Abstract

CPU performance evaluation benchmark aims to provide comparative and quantitative index data for product selection. It is one of the vane leading the development of computing industry

and as CPU technology evolves rapidly

performance benchmarks are evolving. This paper systematically reviews the mainstream benchmarks including the SPEC CPU. From the perspectives of evaluation objectives and methods

the evolution

recent research results of the mainstream CPU benchmarks

and the performance metrics and benchmark requirements of general-purpose CPU are reviewed. Finally

this paper analyzes the challenges of general-purpose CPU performance evaluation benchmarks and prospects for possible future research trends.

关键词

Keywords

references

TICHY W F , LUKOWICZ P , PRECHELT L , et al . Experimental evaluation in computer science: A quantitative study [J]. Journal of Systems and Software , 1995 , 28 ( 1 ): 9 - 18 .

TAO J H , DU Z D , GUO Q , et al . BenchIP: Benchmarking intelligence processors [J]. Journal of Computer Science and Technology , 2018 , 33 ( 1 ): 1 - 23 .

THAKKAR P , NATHAN S , VISWANATHAN B . Performance benchmarking and optimizing hyperledger fabric blockchain platform [C]// 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems . Milwaukee : IEEE , 2018 : 264 - 276 .

王文凯 . 基于DNN的智能芯片性能评估及优化 [D]. 武汉 : 华中科技大学 , 2018 .

WANG W K . Intelligent Chip Performance Evaluation and Optimization Based on DNN [D]. Wuhan : Huazhong University of Science and Technology , 2018 . (in Chinese)

HSIEH S Y , LIU C S , BUYYA R , et al . Utilization-prediction-aware virtual machine consolidation approach for energy-efficient cloud data centers [J]. Journal of Parallel and Distributed Computing , 2020 , 139 : 99 - 109 .

NIDER J , FEDOROVA A S . The last CPU [C]// Proceedings of the Workshop on Hot Topics in Operating Systems . New York : ACM , 2021 : 1 - 8 .

PAWANEKAR S , UDGIRKAR G . Performance of Reinforcement Learning Simulation: X86 vs ARM [M]// Communications in Computer and Information Science . Cham : Springer International Publishing , 2021 : 420 - 430 .

MATHÁ R , KIMOVSKI D , ZABROVSKIY A , et al . Where to encode: A performance analysis of x86 and arm-based Amazon EC2 instances [C]// 2021 IEEE 17th International Conference on eScience . Innsbruck : IEEE , 2021 : 118 - 127 .

LIMAYE A , ADEGBIJA T . A workload characterization of the SPEC CPU2017 benchmark suite [C]// 2018 IEEE International Symposium on Performance Analysis of Systems and Software . Belfast : IEEE , 2018 : 149 - 158 .

BACH M , KRETZ M , LINDENSTRUTH V , et al . Optimized HPL for AMD GPU and multi-core CPU usage [J]. Computer Science - Research and Development , 2011 , 26 ( 3 ): 153 - 164 .

KALYANASUNDARAM K . SPEC HPG benchmarks [C]// Proceedings of the 2006 ACM/IEEE Conference on Supercomputing . Tampa : ACM , 2006 : 17-es.

WEISS A R . The standardization of embedded benchmarking: Pitfalls and opportunities [C]// Proceedings 1999 IEEE International Conference on Computer Design: VLSI in Computers and Processors(Cat. No.99CB37040) . Austin : IEEE , 1999 : 492 - 508 .

CAPRA M , BUSSOLINO B , MARCHISIO A , et al . Hardware and software optimizations for accelerating deep neural networks: Survey of current trends, challenges, and the road ahead [J]. IEEE Access , 8 : 225134 - 225180 .

SEN A , DENIZ E . Thread-level synthetic benchmarks for multicore systems [J]. Microprocessors and Microsystems , 2015 , 39 ( 7 ): 471 - 479 .

KARKI A , KESHAVA C P , SHIVAKUMAR S M , et al . Tango: A deep neural network benchmark suite for various accelerators [EB/OL]. ( 2019 )[2022]. https://arxiv.org/abs/1901.04987 https://arxiv.org/abs/1901.04987 .

LEWIS B C , CREWS A E . The evolution of benchmarking as a computer performance evaluation technique [J]. MIS Quarterly , 1985 , 9 ( 1 ): 7 - 16 .

WEICKER R P . An overview of common benchmarks [J]. Computer , 1990 , 23 ( 12 ): 65 - 75 .

JOHN L K , EECKHOUT L . Performance Evaluation and Benchmarking [M]. Boca Raton : CRC Press , 2006 .

CHONG N , SORENSEN T , WICKERSON J . The semantics of transactions and weak memory in x86, power, ARM, and C++ [C]// PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation . Philadelphia : ACM Press , 2018 : 211 - 225 .

Standard Performance Evaluation Corporation . SPEC CPU 2017 [EB/OL]. [2022] . http://www.spec.org/cpu2017/ http://www.spec.org/cpu2017/ .

AMARAL J N , BORIN E , ASHLEY D R , et al . The Alberta workloads for the SPEC CPU 2017 benchmark suite [C]// 2018 IEEE International Symposium on Performance Analysis of Systems and Software . Belfast : IEEE , 2017 : 159 - 168 .

DIXIT K M . New CPU benchmark suites from SPEC [C]// Digest of Papers COMPCON Spring . San Francisco : IEEE , 1992 : 305 - 310 .

SIMON J , VIETH M , WEICKER R . Workload Analysis of Computation Intensive Tasks: Case Study on SPEC CPU95 Benchmarks [M]// Euro-Par'97 Parallel Processing . Berlin, Heidelberg : Springer , 1997 : 971 - 984 .

廖秋林 , 莫玮 , 陈大为 . SPEC CPU2000性能测试程序分析及其应用 [J]. 国外电子测量技术 , 2006 , 25 ( 6 ): 65 - 68 .

LIAO Q L , MO W , CHEN D W . Analysis and application of SPEC CPU2000 performance test program [J]. Foreign Electronic Measurement Technology , 2006 , 25 ( 6 ): 65 - 68 . (in Chinese)

NAIR A A , JOHN L K . Simulation points for SPEC CPU 2006 [C]// 2008 IEEE International Conference on Computer Design . Lake Tahoe : IEEE , 2008 : 397 - 403 .

PANDA R , SONG S , DEAN J , et al . Wait of a decade: Did SPEC CPU 2017 broaden the performance horizon? [C]// 2018 IEEE International Symposium on High Performance Computer Architecture . Vienna : IEEE , 2018 : 271 - 282 .

SONG S , WU Q Z , FLOLID S , et al . Experiments with SPEC CPU 2017: Similarity, balance, phase behavior and simPoints [EB/OL]. ( 2018 )[2022]. http://lca.ece.utexas.edu/pubs/UT_LCA_TR-180515-01.pdf http://lca.ece.utexas.edu/pubs/UT_LCA_TR-180515-01.pdf .

SINGH S , AWASTHI M . Memory centric characterization and analysis of SPEC CPU2017 suite [C]// Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering . New York : ACM , 2019 : 285 - 292 .

BUCEK J , LANGE K D , KISTOWSKI J V . SPEC CPU2017: Next-generation compute benchmark [C]// ICPE'18: Companion of the 2018 ACM/SPEC International Conference on Performance Engineering . Berlin : ACM , 2018 : 41 - 42 .

WALLACE R . Performance Benchmark and assessment of a mixed batch and real-time transaction processing system [C]// Proceedings of Computer Measurement Group Conference . Orlando : DBLP Computer Science Bibliography , 1997 : 863 - 872 .

GRAY J . A view of database system performance measures [J]. ACM SIGMETRICS Performance Evaluation Review , 1987 , 15 ( 1 ): 3 - 4 .

LEVINE C . Why TPC-A and TPC-B are obsolete [C]// Digest of Papers. Compcon Spring . San Francisco : IEEE , 1993 : 215 - 221 .

LEUTENEGGER S T , DIAS D . Modeling study of the TPC-C benchmark [J]. ACM SIGMOD Record , 1993 , 22 ( 2 ): 22 - 31 .

CHEN S M , AILAMAKI A , ATHANASSOULIS M , et al . TPC-E vs TPC-C: Characterizing the new TPC-E benchmark via an I/O comparison study [J]. SIGMOD Record , 2010 , 39 ( 3 ): 5 - 10 .

KANDASWAMY M A , KNIGHTEN R L . I/O phase characterization of TPC-H query operations [C]// Proceedings IEEE International Computer Performance and Dependability Symposium . IPDS . Chicago : IEEE , 2000 : 81 - 90 .

TRIVEDI M , CHEN Z Q . Lessons Learned from the Industry's First TPC Benchmark DS(TPC-DS) [M]// Performance Evaluation and Benchmarking for the Era of Artificial Intelligence . Cham : Springer International Publishing , 2019 : 140 - 154 .

DEEHR E , FANG W Q , REZA TAHERI H , et al . Performance Analysis of Database Virtualization with the TPC-VMS Benchmark [M]// Lecture Notes in Computer Science . Cham : Springer International Publishing , 2015 : 156 - 172 .

刘建鹏 , 刘尧 . 将TPC-DS工具合入HiBench测试框架的方法 [J]. 数字技术与应用 , 2019 , 37 ( 10 ): 64 - 65 .

LIU J P , LIU Y . Method of incorporating TPC-DS tools into the HiBench test framework [J]. Digital Technology & Application , 2019 , 37 ( 10 ): 64 - 65 . (in Chinese)

ZHAI J D , ZHANG F , LI Q W , et al . Characterizing and optimizing TPC-C workloads on large-scale systems using SSD arrays [J]. Science China Information Sciences , 2016 , 59 ( 9 ): 92104 .

冯志丹 . 基于SCF中间件的TPC-C测试系统的设计和开发 [D]. 北京 : 北京邮电大学 , 2016 .

FENG Z D . The Design and Development of TPC-C Test System Based on SCF Middleware [D]. Beijing : Beijing University of Posts and Telecommunications , 2016 . (in Chinese)

Labs Primate . Geekbench 5 . 1 . 1 [EB/OL]. [2022] . https://www.geekbench.com/blog/2020/04/geekbench-511 https://www.geekbench.com/blog/2020/04/geekbench-511 .

CORNERO M , ANYURU A . Multiprocessing in Mobile Platforms: The Marketing and the Reality [R]. Genève : ST-ERICSSON , 2013 .

KOZHIRBAYEV Z , SINNOTT R O . A performance comparison of container-based technologies for the Cloud [J]. Future Generation Computer Systems , 2017 , 68 : 175 - 182 .

POLVINEN T , YLIKÄNNÖ T , MÄKELÄINEN A , et al . Building a virtualized environment for programming courses [C]// WorldCIST 2020: Trends and Innovations in Information Systems and Technologies(AISC , volume 1160 ). Cham : Springer , 2020 : 45 - 55 .

MORABITO R , KJÄLLMAN J , KOMU M . Hypervisors vs. lightweight virtualization: A performance comparison [C]// 2015 IEEE International Conference on Cloud Engineering . Tempe : IEEE , 2015 : 386 - 393 .

BARKER A , VARGHESE B , THAI L . Cloud services brokerage: A survey and research roadmap [C]// 2015 IEEE 8th International Conference on Cloud Computing . New York : IEEE , 2015 : 1029 - 1032 .

WANG Y , LEE V , WEI G Y , et al . Predicting new workload or CPU performance by analyzing public datasets [J]. ACM Transactions on Architecture and Code Optimization , 2019 , 15 ( 4 ): 1 - 21 .

SINGH J P , WEBER W D , GUPTA A . SPLASH: Stanford parallel applications for shared-memory [J]. SIGARCH Computer Architecture News , 1992 , 20 ( 1 ): 5 - 44 .

WOO S C , OHARA M , TORRIE E , et al . The SPLASH-2 programs: characterization and methodological considerations [C]// Proceedings of the 22nd Annual International Symposium on Computer Architecture . Santa Margherita Ligure : IEEE , 1995 : 24 - 36 .

SAKALIS C , LEONARDSSON C , KAXIRAS S , et al . Splash-3: A properly synchronized benchmark suite for contemporary research [C]// 2016 IEEE International Symposium on Performance Analysis of Systems and Software . Uppsala : IEEE , 2016 : 101 - 111 .

BIENIA C , KUMAR S , SINGH J P , et al . The PARSEC benchmark suite: Characterization and architectural implications [C]// PACT'08: Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques . Toronto : IEEE , 2008 : 72 - 81 .

BIENIA C , KUMAR S , LI K . PARSEC vs. SPLASH-2: A quantitative comparison of two multithreaded benchmark suites on Chip-Multiprocessors [C]// 2008 IEEE International Symposium on Workload Characterization . Seattle : IEEE , 2008 : 47 - 56 .

BARROW-WILLIAMS N , FENSCH C , MOORE S . A communication characterisation of splash-2 and parsec [C]// 2009 IEEE International Symposium on Workload Characterization . Austin : IEEE , 2009 : 86 - 97 .

BIENIA C , LI K . Fidelity and scaling of the PARSEC benchmark inputs [C]// IEEE International Symposium on Workload Characterization . Atlanta : IEEE , 2010 : 1 - 10 .

CHASAPIS D , CASAS M , MORETÓ M , et al . PARSECSs: Evaluating the impact of task parallelism in the PARSEC benchmark suite [J]. ACM Transactions on Architecture and Code Optimization , 2016 , 12 ( 4 ): 41 .

CEBRIAN J M , JAHRE M , NATVIG L . ParVec: Vectorizing the PARSEC benchmark suite [J]. Computing , 2015 , 97 ( 11 ): 1077 - 1100 .

HUYNH A , HELM C , IWASAKI S , et al . TP-PARSEC: A task parallel PARSEC benchmark suite [J]. Journal of Information Processing , 2019 , 27 : 211 - 220 .

HEINECKE A , VAIDYANATHAN K , SMELYANSKIY M , et al . Design and implementation of the linpack benchmark for single and multi-node systems based on intel® xeon phi coprocessor [C]// 2013 IEEE 27th International Symposium on Parallel and Distributed Processing . Cambridge : IEEE , 2013 : 126 - 137 .

BLIN A , COURTAUD C , SOPENA J , et al . Understanding the memory consumption of the MiBench embedded benchmark [C]// NETYS 2016: Networked Systems(LNCCN , volume 9944 ). Cham : Springer , 2016 : 71 - 86 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Influence Analysis Method of Class Imbalance on Software Defect Prediction Model Stability and Prediction Performance

Related Author

ZHAO You-jian

SHI Hui-kang

YUAN Guan

JIANG Shu-juan

ZHI Sheng-lin

ZHANG Yan-mei

ZHI Sheng-lin

Related Institution

Department of Computer Science and Technology， Tsinghua University

Mine Digitization Engineering Research Center of the Ministry of Education， China University of Mining and Technology

School of Computer Science and Technology， China University of Mining and Technology

KeHua Data CO.， LTD

科华数据股份有限公司

⁰