Traditional camera accumulates photons during an exposure time window to generate a still image or a video in the form of image sequences, which lose the temporal process of the photons flow, leads to an irreconcilable dilemma between high-dynamic and high-speed imaging. Based on the fact that the pixels of the photoelectric sensor are independent, a new continuous photographing principle is proposed: each pixel converts the received photon flow into an electronic flow independently, continuously measures the photoelectric current and converts it into a digital flow, and represents continuously at the pixel level. Then the sequence array ranked according to the pixel layout is a continuous representation of the photons shotting on the sensing plane. The image of any moment can be obtained by intercepting the state of the sequence array, so as to realize continuous imaging.
Furthermore, the spiking continuous photographing principle that modulates the photon flow into a spike sequence is proposed: for each pixel, accumulate charge from the reset state, generate a spike as a flag once the specified threshold is reached, reset and repeat. The duration that a spike takes to be fired is called its spiking width, which is inversely proportional to the light intensity during this period. Based on this, the light intensity during this period can be estimated. The spikes sequence with natural temporal order is a digital representation of the photon flow process. The array of the spike sequences according to the spatial layout of pixels is called “viform”, which contains rich spatial and temporal information of the light process. An image at any moment can be calculated from viform, thereby ultrahigh-speed, high-dynamic and non-blurred continuous imaging is achieved, and the dilemma between high-dynamic and high-speed imaging of exposure imaging is solved completely.
The unique parameter of the spiking continuous photographing principle is the accumulation threshold Q, which corresponds to the number of photons/electrons required to fire a spike. The unique variable is the spike accumulation time τ,which is the easiest to measure accurately and can be concisely represented by natural temporal order. The light intensity at any moment is Q/τ, which is determined by the spike width τ, which range is (0, ∞). Thus, theoretically, arbitrary light intensity could by be represented, and lead to infinite dynamic range imaging. In the physical implementation, the strongest light could be represented depending on the shortest readout time τ? of the circuit, and the weakest light is the dark current, which can be accumulated as a spike within duration τ?. τ?/τ?makes up the imaging dynamic range. With conventional photoelectric devices and circuits, ultrahigh-dynamic imaging with 160 dB or even 180 dB can be achieved. When the imaging time sensitivity is required to be less than τ?, ultrahigh-dynamic imaging can be achieved by sub-threshold quantization of the accumulated voltage.
Photon and photoelectron flow are discrete statistical process that follows a Poisson distribution and is often a “segmental linear” process light intensity changes suddenly while the intensity is stable between successive mutations. A method so-called first spike encoding is proposed: only the moment of the first spike firing and its spike width are output when the intensity changes, and no spike is output thereafter to indicate repetition. This is the optimal lossless compression method of the spike sequence, where the dynamic range and time sensitivity can be significantly improved by improving the time measurement accuracy with almost no increase in representation data volume.
Using mature CMOS photonic devices and standard processes, two spike continuous photographing chips and spike cameras with spatial resolutions of 0.1 million and 1 million pixels have been developed, respectively. The chips use 40 000 Hz synchronous spike output, with a minimum spike width of 25 μs. Practical tests have verified the feasibility of the spike continuous photographing principle and its ability to achieve ultrahigh-speed, high-dynamic, and blur-free imaging.
Viform, the spike continuous photographing representation and the replacement of image and video, captures the temporal and spatial information of the photon flow, will fundamentally reshapes computer vision and visual information processing technology and industries. An open source algorithm framework, SpikeCV, achieves high-speed target detection, tracking, and recognition system that is faster than the human eye by thousands of times, with low computational complexity. As the symmetrical process of spike continuous photographing, spike continuous displaying modulates the ultrahigh-speed spike sequence into extremely high-speed photon flow, enabling high-speed display similar to natural light through glass, solving the motion blur and visual fatigue dizziness caused by low frame rate in traditional display systems. In addition, the combination of continuous photographing and continuous displaying can achieve a single-way transparent glass-like display and ultrahigh-speed light communication without medium.