Stochastic Computing and Hybrid Stochastic Computing

LI Hong-ge; CHEN Yu-hao; WU Jun-yi; SONG Yin-jie; ZHU Xin-yu

doi:10.12263/DZXB.20230222

您当前的位置：

首页 >

文章列表页 >

Stochastic Computing and Hybrid Stochastic Computing

PAPERS | 更新时间：2025-12-11

- Stochastic Computing and Hybrid Stochastic Computing
- ACTA ELECTRONICA SINICA Vol. 52, Issue 2, Pages: 428-440(2024)
- 作者机构：
  
  北京航空航天大学电子信息工程学院，北京市 100191
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62071019)
- DOI：10.12263/DZXB.20230222
  CLC： TP391;
- Received：09 March 2023，
  
  Revised：2023-06-30，
  
  Published：25 February 2024
- 稿件说明：
移动端阅览
李洪革,陈宇昊,吴俊毅,等.概率计算及混合概率计算[J].电子学报,2024,52(02):428-440.

LI Hong-ge, CHEN Yu-hao, WU Jun-yi, et al.Stochastic Computing and Hybrid Stochastic Computing[J].Acta Electronica Sinica, 2024, 52(02): 428-440.
李洪革,陈宇昊,吴俊毅,等.概率计算及混合概率计算[J].电子学报,2024,52(02):428-440. DOI：10.12263/DZXB.20230222

LI Hong-ge, CHEN Yu-hao, WU Jun-yi, et al.Stochastic Computing and Hybrid Stochastic Computing[J].Acta Electronica Sinica, 2024, 52(02): 428-440. DOI：10.12263/DZXB.20230222

摘要

非位置概率数的计算机制已经成为边缘计算片上系统的新范式.本文介绍了概率计算（stochastic computing）的起源、发展和目前国内外的研究现状.针对传统概率计算存在诸如计算时延长、脉冲串信息携带效率低等问题，本文提出了二进制数-概率脉冲串混合编码的混合概率数概念，并从数的表示机理上阐释了二进制数、概率数和混合概率数的数理关系，进而揭示了混合概率计算所具备的低时延、高算力和高能效比的计算特点.本文基于40 nm CMOS工艺设计混合概率深度神经网络，该神经网络芯片在内核面积仅0.73 mm × 0.73 mm的条件下，设计4 544个乘累加（MAC）单元.在时钟频率400 MHz条件下，总功率为102.3 mW，其中动态功耗仅97 μW.通过ASIC芯片的实验测试表明，混合概率计算作为一种全新的颠覆性计算范式，与其他确定性、可扩展和全并行等概率计算方案相比，其能效比分别提高了50倍、2.5倍和3.26倍.

Abstract

The calculation principle of non-positional stochastic number (SN) is a promising technique for realizing high-performance computing owing to its extremely low hardware cost. This paper introduces detailly the origin

development process and the domestic and foreign development present situation. However

a disadvantage of stochastic bitstream is that the computing latency

and information-carrying efficiency and so on. We presented a hybrid stochastic computing (HSC) based on a hybrid bitstream to solve these problems

which achieves a lower hardware cost

better accuracy

and faster speed. The HSC neural networks is fabricated by 40 nm low-power CMOS process

with a core area of 0.73 mm × 0.73 mm

power of 102.3 mW and clock of 400 MHz

which has 4 544 multiply and accumulation (MAC).The proposed Hybrid stochastic computing is tested by FPGA and ASIC.Compared with other stochastic computing method

the method proposed gains 50×

2.5×

and 3.26× energy efficiency than other methods of traditional stochastic computing.

关键词

Keywords

references

VON NEUMANN J . Probabilistic Logics and the synthesis of reliable organisms from unreliable components [M]// Automata Studies (AM-34) . Princeton : Princeton University Press , 1956 : 43 - 98 .

GAINES B R . Stochastic computing [C]// Proceedings of the Spring Joint Computer Conference on - AFIPS’67 . New York : ACM , 1967 : 149 - 156 .

POPPELBAUM W J , AFUSO C , ESCH J W . Stochastic computing elements and systems [C]// Proceedings of the Fall Joint Computer Conference . New York : ACM , 1967 : 635 - 644 .

GAINES B R . Stochastic computing systems [M]//TOU JT. Advances in Information Systems Science . Boston, MA : Springer , 1969 : 37 - 172 .

KONDO Y , SAWADA Y . Functional abilities of a stochastic logic neural network [J]. IEEE Transactions on Neural Networks , 1992 , 3 ( 3 ): 434 - 443 .

ALAGHI A , HAYES J P . Survey of stochastic computing [J]. ACM Transactions on Embedded Computing Systems , 2013 , 12 ( 2S ): 92 .

HAYES J P . Introduction to stochastic computing and its challenges [C]// 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC) . Piscataway : IEEE , 2015 : 1 - 3 .

CANALS V , MORRO A , OLIVER A , et al . A new stochastic computing methodology for efficient neural network implementation [J]. IEEE Transactions on Neural Networks and Learning Systems , 2016 , 27 ( 3 ): 551 - 564 .

LIU S T , HAN J . Energy efficient stochastic computing with Sobol sequences [C]//Design, Automation & Test in Europe Conference & Exhibition (DATE) . Piscataway : IEEE , 2017 : 650 - 653 .

JENSON D , RIEDEL M . A deterministic approach to stochastic computation [C]// 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) . Piscataway : IEEE , 2016 : 1 - 8 .

ZHANG Z D , WANG R S , ZHANG Z , et al . Circuit reliability comparison between stochastic computing and binary computing [J]. IEEE Transactions on Circuits and Systems Ⅱ: Express Briefs , 2020 , 67 ( 12 ): 3342 - 3346 .

HU Y X , ZHANG Y W , WANG R S , et al . A 28-nm 198.9-TOPS/W fault-tolerant stochastic computing neural network processor [J]. IEEE Solid-State Circuits Letters , 2022 , 5 : 198 - 201 .

QIAN W K , LI X , RIEDEL M D , et al . An architecture for fault-tolerant computation with stochastic logic [J]. IEEE Transactions on Computers , 2011 , 60 ( 1 ): 93 - 105 .

QIAN W K , RIEDEL M D , ROSENBERG I . Uniform approximation and Bernstein polynomials with coefficients in the unit interval [J]. European Journal of Combinatorics , 2011 , 32 ( 3 ): 448 - 463 .

CHEN Z Y , MA Y F , WANG Z F . Hybrid stochastic-binary computing for low-latency and high-precision inference of CNNs [J]. IEEE Transactions on Circuits and Systems I: Regular Papers , 2022 , 69 ( 7 ): 2707 - 2720 .

ICHIHARA H , ISHII S , SUNAMORI D , et al . Compact and accurate stochastic circuits with shared random number sources [C]// 2014 IEEE 32nd International Conference on Computer Design (ICCD) . Piscataway : IEEE , 2014 : 361 - 366 .

XIE Y , LIAO S Y , YUAN B , et al . Fully-parallel area-efficient deep neural network design using stochastic computing [J]. IEEE Transactions on Circuits and Systems Ⅱ: Express Briefs , 2017 , 64 ( 12 ): 1382 - 1386 .

YANG M , LI B Z , LILJA D J , et al . Towards theoretical cost limit of stochastic number generators for stochastic computing [C]// 2018 IEEE Computer Society Annual Symposium on VLSI (ISVLSI) . Piscataway : IEEE , 2018 : 154 - 159 .

AHMAD SALEHI S . Low-cost stochastic number generators for stochastic computing [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2020 , 28 ( 4 ): 992 - 1001 .

CHEN T H , TING P S , HAYES J P . Achieving progressive precision in stochastic computing [C]// 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP) . Piscataway : IEEE , 2017 : 1320 - 1324 .

KIM K , LEE J , CHOI K . An energy-efficient random number generator for stochastic circuits [C]// 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC) . Piscataway : IEEE , 2016 : 256 - 261 .

CHEN T H , HAYES J P . Design of division circuits for stochastic computing [C]// 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI) . Piscataway : IEEE , 2016 : 116 - 121 .

SIM H , NGUYEN D , LEE J , et al . Scalable stochastic-computing accelerator for convolutional neural networks [C]// 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC) . Piscataway : IEEE , 2017 : 696 - 701 .

BROWN B D , CARD H C . Stochastic neural computation. I. Computational elements [J]. IEEE Transactions on Computers , 2001 , 50 ( 9 ): 891 - 905 .

TING P S , HAYES J P . Stochastic logic realization of matrix operations [C]// 2014 17th Euromicro Conference on Digital System Design . Piscataway : IEEE , 2014 : 356 - 364 .

SIM H , LEE J . A new stochastic computing multiplier with application to deep convolutional neural networks [C]// 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC) . Piscataway : IEEE , 2017 : 1 - 6 .

KIM K , KIM J , YU J , et al . Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks [C]// 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC) . Piscataway : IEEE , 2016 : 1 - 6 .

HOJABR R , GIVAKI K , TAYARANIAN S R , et al . SkippyNN: an embedded stochastic-computing accelerator for convolutional neural networks [C]// 2019 56th ACM/IEEE Design Automation Conference (DAC) . Piscataway : IEEE , 2019 : 1 - 6 .

CHU S I . New divider design for stochastic computing [J]. IEEE Transactions on Circuits and Systems Ⅱ: Express Briefs , 2020 , 67 ( 1 ): 147 - 151 .

ARDAKANI A , LEDUC-PRIMEAU F , ONIZAWA N , et al . VLSI implementation of deep neural network using integral stochastic computing [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2017 , 25 ( 10 ): 2688 - 2699 .

ZHAKATAYEV A , LEE S , SIM H , et al . Sign-magnitude SC: Getting 10X accuracy for free in stochastic computing for deep neural networks [C]// 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC) . Piscataway : IEEE , 2018 : 1 - 6 .

ALAGHI A , HAYES J P . Fast and accurate computation using stochastic circuits [C]// 2014 Design , Automation & Test in Europe Conference & Exhibition (DATE) . Piscataway : IEEE , 2014 : 1 - 4 .

LIU S T , HAN J . Toward energy-efficient stochastic circuits using parallel sobol sequences [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2018 , 26 ( 7 ): 1326 - 1339 .

LI B Z , NAJAFI M H , YUAN B , et al . Quantized neural networks with new stochastic multipliers [C]// 2018 19th International Symposium on Quality Electronic Design (ISQED) . Piscataway : IEEE , 2018 : 376 - 382 .

GROSS W J , GAUDET V C , MILNER A . Stochastic implementation of LDPC decoders [C]// Conference Record of the Thirty-Ninth Asilomar Conference onSignals, Systems and Computers . Piscataway : IEEE , 2006 : 713 - 717 .

CHEN Y H , LI H G . Stochastic computing using amplitude and frequency encoding [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2022 , 30 ( 5 ): 656 - 660 .

LI H G , CHEN Y H . Hybrid logic computing of binary and stochastic [J]. IEEE Embedded Systems Letters , 2022 , 14 ( 4 ): 171 - 174 .

LIU Y D , LIU L B , LOMBARDI F , et al . An energy-efficient and noise-tolerant recurrent neural network using stochastic computing [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2019 , 27 ( 9 ): 2213 - 2221 .

LI Z , LI J , REN A , et al . HEIF: Highly efficient stochastic computing-based inference framework for deep neural networks [J]. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2019 , 38 ( 8 ): 1543 - 1556 .

LIU Y D , WANG Y Z , LOMBARDI F , et al . An energy-efficient stochastic computational deep belief network [C]// 2018 Design , Automation & Test in Europe Conference & Exhibition (DATE) . Piscataway : IEEE , 2018 : 1175 - 1178 .

LI J , REN A , LI Z , et al . Towards acceleration of deep convolutional neural networks using stochastic computing [C]// 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC) . Piscataway : IEEE , 2017 : 115 - 120 .

BROWN B D , CARD H C . Stochastic neural computation Ⅱ Soft competitive learning [J]. IEEE Transactions on Computers , 2001 , 50 ( 9 ): 906 - 920 .

LIU Y D , LIU S T , WANG Y Z , et al . A stochastic computational multi-layer perceptron with backward propagation [J]. IEEE Transactions on Computers , 2018 , 67 ( 9 ): 1273 - 1286 .

REN A , LI Z , WANG Y Z , et al . Designing reconfigurable large-scale deep learning systems using stochastic computing [C]// 2016 IEEE International Conference on Rebooting Computing (ICRC) . Piscataway : IEEE , 2016 : 1 - 7 .

REN A , LI J , LI Z , et al . SC-DCNN: Highly-scalable deep convolutional neural network using stochastic computing [C]// Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems . New York : ACM Press , 2017 : 405 - 418 .

YU J , KIM K , LEE J , et al . Accurate and efficient stochastic computing hardware for convolutional neural networks [C]// 2017 IEEE International Conference on Computer Design (ICCD) . Piscataway : IEEE , 2017 : 105 - 112 .

LI J , YUAN Z H , LI Z , et al . Normalization and dropout for stochastic computing-based deep convolutional neural networks [J]. Integration , the VLSI Journal, 2019 , 65 : 395 - 403 .

SIMARD P Y , STEINKRAUS D , PLATT J C . Best practices for convolutional neural networks applied to visual document analysis [C]// 2003 Proceedings of Seventh International Conference on Document Analysis and Recognition . Piscataway : IEEE , 2003 : 958 - 963 .

ZHANG Y W , ZHANG X Y , SONG J H , et al . Parallel convolutional neural network (CNN) accelerators based on stochastic computing [C]// 2019 IEEE International Workshop on Signal Processing Systems (SiPS) . Piscataway : IEEE , 2019 : 19 - 24 .

MA X L , ZHANG Y P , YUAN G , et al . An area and energy efficient design of domain-wall memory-based deep convolutional neural networks using stochastic computing [C]// 2018 19th International Symposium on Quality Electronic Design (ISQED) . Piscataway : IEEE , 2018 : 314 - 321 .

JIA X T , YANG J L , DAI P C , et al . SPINBIS: Spintronics-based Bayesian inference system with stochastic computing [J]. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2020 , 39 ( 4 ): 789 - 802 .

ALAGHI A , HAYES J . Dimension reduction in statistical simulation of digital circuits [C]//Proceedings of the Symposium on Theory of Modeling & Simulation: DEVS Integrative M & S Symposium . New York : ACM , 2015 : 1 - 8 .

ICHIHARA H , SUGINO T , ISHII S , et al . Compact and accurate digital filters based on stochastic computing [J]. IEEE Transactions on Emerging Topics in Computing , 2019 , 7 ( 1 ): 31 - 43 .

NAJAFI M H , LILJA D J . High quality down-sampling for deterministic approaches to stochastic computing [J]. IEEE Transactions on Emerging Topics in Computing , 2021 , 9 ( 1 ): 7 - 14 .

LI P , LILJA D J , QIAN W K , et al . Computation on stochastic bit streams digital image processing case studies [J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2014 , 22 ( 3 ): 449 - 462 .

EL-DERHALLI H , LE BEUX S , TAHAR S . Design space exploration of stochastic computing architectures implemented using integrated optics [J]. IEEE Transactions on Emerging Topics in Computing , 2021 , 9 ( 4 ): 2158 - 2169 .

MIN S J , LEE E W , CHAE S I . A study on the stochastic computation using the ratio of one pulses and zero pulses [C]// 1994 IEEE International Symposium on Circuits and Systems (ISCAS) . Piscataway : IEEE , 1994 : 471 - 474 .

SATO S , NEMOTO K , AKIMOTO S , et al . Implementation of a new neurochip using stochastic logic [J]. IEEE Transactions on Neural Networks , 2003 , 14 ( 5 ): 1122 - 1127 .

LI H G , HAYAKAWA Y , SATO S , et al . Hardware implementation of an inverse function delayed neural network using stochastic logic [J]. IEICE - Transactions on Information and Systems , 2006 , E89-D( 9 ): 2572 - 2578 .

FAIX M , LAURENT R , BESSIÈRE P , et al . Design of stochastic machines dedicated to approximate Bayesian inferences [J]. IEEE Transactions on Emerging Topics in Computing , 2019 , 7 ( 1 ): 60 - 66 .

LI W J , XU N Y , WANG R S , et al . Efficient compression methods for wire-spread-based stochastic computing deep neural networks [J]. IEEE Transactions on Circuits and Systems Ⅱ: Express Briefs , 2022 , 69 ( 11 ): 4538 - 4542 .

LEE V T , ALAGHI A , HAYES J P , et al . Energy-efficient hybrid stochastic-binary neural networks for near-sensor computing [C]//Design, Automation & Test in Europe Conference & Exhibition (DATE) . Piscataway : IEEE , 2017 : 13 - 18 .

LORENTZ G G . Bernstein Polynomials [M]. Second edition . Providence : merican Mathematical Society , 1986 .

FARAJI S R , HASSAN NAJAFI M , LI B Z , et al . Energy-efficient convolutional neural networks with deterministic bit-stream processing [C]// 2019 Design , Automation & Test in Europe Conference & Exhibition (DATE) . Piscataway : IEEE , 2019 : 1757 - 1762 .

LIU S S , TANG X C , NIKNIA F , et al . Stochastic dividers for low latency neural networks [J]. IEEE Transactions on Circuits and Systems I: Regular Papers , 2021 , 68 ( 10 ): 4102 - 4115 .

MORÁN A , PARRILLA L , ROCA M , et al . Digital implementation of radial basis function neural networks based on stochastic computing [J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems , 2023 , 13 ( 1 ): 257 - 269 .

FRASSER C F , LINARES-SERRANO P , DE LOS DE RIOS I D , et al . Fully parallel stochastic computing hardware implementation of convolutional neural networks for edge computing applications [J]. IEEE Transactions on Neural Networks and Learning Systems , 2023 , 34 ( 12 ): 10408 - 10418 .

ZHANG J L , LI J . Improving the performance of OpenCL-based FPGA accelerator for convolutional neural network [C]// Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays . New York : ACM , 2017 : 25 - 34 .

ZIMMER B , VENKATESAN R , SHAO Y S , et al . A 0.32-128 TOPS, scalable multi-chip-module-based deep neural network inference accelerator with ground-referenced signaling in 16 nm [J]. IEEE Journal of Solid-State Circuits , 2020 , 55 ( 4 ): 920 - 932 .

LIU S L , FAN H X , FERIANC M , et al . Toward full-stack acceleration of deep convolutional neural networks on FPGAs [J]. IEEE Transactions on Neural Networks and Learning Systems , 2022 , 33 ( 8 ): 3974 - 3987 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Adversarial Learning and Enhanced Optimization Based Restoration Method for VC-Generated Speeches

Binary Code Similarity Detection Method Based on Cross-Modal Coordinated Representation Learning

A Hierarchical Automatic Sleep Staging Model Based on Transformer Architecture

Robust Physical Adversarial Camouflages for Image Classifiers

Related Author

CHEN Yu-hao

YUE Feng

ZHANG Guo-fu

ZHOU Xiao-lin

SU Zhao-pin

LIAN Chen-si

WANG Nian-song

YANG Hong-yu

Related Institution

Joint Laboratory of Intelligent Prevention and Recognition of Audio and Video

Intelligent Interconnected Systems Laboratory of Anhui Province (Hefei University of Technology)

Department of Physical Evidence Identification, Anhui Public Security Department

School of Computer and Information Technology, Hefei University of Technology

Key Laboratory of Civil Aviation Flight Networking, Civil Aviation University of China

⁰