The March 2022 edition of Photonics Spectra magazine has an interesting article titled "Photon-Counting CMOS Sensors: Extend Frontiers in Scientific Imaging" by Dakota Robledo, Ph.D., senior image sensor scientist at Gigajot Technology.
While CMOS imagers have evolved significantly since the 1960s, photon-counting sensitivity has still required the use of specialized sensors that often come with detrimental drawbacks. This changed recently with the emergence of new quanta image sensor (QIS) technology, which pushes CMOS imaging capabilities to their fundamental limit while also delivering high-resolution, high-speed, and low-power linear photon counting at room temperature. First proposed in 2005 by Eric Fossum, who pioneered the CMOS imaging sensor, the QIS paradigm envisioned a large array of specialized pixels, called jots, that are able to accurately detect single photons at a very fast frame rate . The technology’s unique combination of high resolution, high sensitivity, and high frame rate enables imaging capabilities that were previously impossible to achieve. The concept was also expanded further to include multibit QIS, wherein the jots can reliably enumerate more than a single photon. As a result, quanta image sensors can be used in higher light scenarios, versus other single-photon detectors, without saturating the pixels. The multibit QIS concept has already resulted in new sensor architectures using photon number resolution, with sufficient photon capacity for high-dynamic-range imaging, and the ability to achieve competitive frame rates.
The photon-counting error rate of a detector is often quantified by the bit error rate. The broadening of signals associated with various photo charge numbers causes the peaks and valleys in the overall distribution to become less distinct, and eventually to be indistinguishable. The bit error rate measures the fraction of false positive and false negative photon counts compared to the total photon count in each signal bin. Figure 4 shows the predicted bit error rate of a detector as a function of the read noise, which demonstrates the rapid rate reduction that occurs for very low-noise sensors.
The article ends with a qualitative comparison between three popular single-photon image sensor technologies.
Interestingly, SPADs are listed as "No Photon Number Resolution" and "Low Manufacturability". It may be worth referring to previous blog posts for different perspectives on this issue. [1] [2] [3]
Full article available here: https://www.photonicsspectra-digital.com/photonicsspectra/march_2022/MobilePagedReplica.action?pm=1&folio=50#pg50
The comparison table is indeed out of date and certainly SPAD-based multi-bit QIS devices have significant photon-number resolution using circuitry including a digital counter stacked under the SPAD layer.
ReplyDeleteJiaju Ma, Stanley Chan and I have written a QIS review paper for the forthcoming IEEE Trans. Electron Devices Special Issue on Solid-State Image Sensors. There is a more detailed comparison table and it talks about CIS QIS and SPAD QIS, 1b and multi-bit devices. Stanley talks about low light computation imaging or signal processing to "see in the dark." We are doing minor revisions now and perhaps the paper will be available by early access in a few weeks. Since we sent the draft around to leaders in the CIS, SPAD and computational imaging communities for additional reviewing eyes, I believe it will be fairly comprehensive and balanced with 200+ references. Personally, I am excited about recent progress in SPAD-QIS devices as well as Gigajot's CIS QIS devices. They have complementary strengths and it is safe to say photon-counting sensors have arrived.
Thanks, Eric. Looking forward to the review paper. I'll post a summary here on the blog as soon as it's out.
DeleteEric, I am a SPAD guy and I have to correct you here. SPAD is photon counting however it can't resolve photon numbers. The mechanism you described to use counter to count the photons "one by one" and "one at a time" is not called photon number resolving. SPAD due to its 1 electron full well capacity is not able to provide photon number resolution within a frame/capture. The table in the paper looks very accurate and it is not outdated. You got confused maybe due to your lack of understanding how SPAD functions.
DeleteThe table in paper is correct. SPAD doesn't provide photon number resolving. It can count photoelectrons one at every exposure. The counter is simply adds up bits (photon counts).
DeleteIt is true that if multiple photons struck a SPAD simultaneously, one could not resolve them, whereas in a CIS-QIS, one can, assuming the quantity of photoelectrons is less than the FWC. We don't usually encounter such a high photon flux for QIS applications that a significant number of photons arrive simultaneously, and it is easy enough to calculate that probability given a "window of simultaneity" and photon flux. However, in most QIS scenarios, other than flash photography, we want to know know the number of photons that struck a pixel over some time period, say a millisecond, in order to estimate the intensity of the photon field. In this case the counter approach works pretty well at giving the photon number. At, least to the extent that my lack of understanding allows me to understand it.
DeleteEric, now you are changing your word and you say for "QIS applications" SPAD is photon number resolving. What do you mean by QIS application? Photon starve application? Almost no one cares about those applications (less than 0.1% of image sensor uses if we talk about imaging (not ToF)), and for example Canon SPAD to be used in security, it is an application where many pixel can be exposed to thousands and tens of thousands of photons in a short period of time. SPAD can't resolve number of photons accurately for those pixels. In security, both low light and HDR are important. SPAD is not photon number resolver within an exposure and even for multi exposures, it is not a "reliable" photon number resolver. I strongly believe, we scientists and engineers have to be very precise about what we say.
DeleteThanks for clarifying the difference between photon number resolving v/s single photon detecting.
DeleteBut I agree with Eric that in practice that distinction might not be as important because we usually care about estimating pixel brightness i.e. intensity. This can be achieved using statistics-based spatio-temporal processing to account for the photon counts that were missed by the pixel.
I'd also disagree that "almost no one cares about [low light] applications." Low light is a relative term. For example, imagine having to capture fast motion in broad daylight. That requires high frame rate which means each frame gets very little light. Dealing with motion is a fundamental problem in any kind of imaging application. The problem only gets harder when capturing motion in low light conditions.
Recent work from many different research groups (Dartmouth, U Wisconsin, Edinburgh, EPFL,...) has shown the advantage of processing high-frame-rate low-bit-depth data from single-photon image sensors for getting rid of motion blur.
There's also some work showing the non-linear response curve of a single-photon sensor pixel gives advantages in extremely high brightness conditions as well.
I think we should mention dead time and the exposure when we say whether SPAD is able to count photons or not. You guys are talking about different applications (photon-starved and HDR) so the discussion may go wild.
DeletePersonally, from Eric's early papers on QIS (I'm referring to QIS from gigajot), I thought QIS concept was delivered targeting HDR (I recall D-logH plot). But it seems now it focuses on low-light imaging, and its HDR function doesn't seem to rely much on photon counting, but instead on DCG (from VLSI2021 paper), which is apparently not a new technique.
So I wonder how photon-counting capability of QIS improves low-light photography versus low read noise CIS, and also if there is any reason that QIS pursued multi-bit instead of single-bit while giving up the D-logH characteristic.
Maybe you should try your hand at defining "photon-number resolving." Some time ago I looked through the physics community archival literature and photon-number is used in various ways, but I didn't find any instance when it meant only simultaneous in time. Or maybe you meant within the dead-time of the SPAD pixel? Anyway this discussion is about the table entries above so of course we should be talking QIS applications which are not photon-arrival timing applications (so far). Gigajot has applied technology developed for 1bQIS and mbQIS to general purpose CMOS image sensors with deep sub-electron read noise (DSERN) with excellent results. These sensors go from photon-counting to high dynamic range in bright scenes and are well-suited for security applications, for example, among others. I guess we will see if CIS just gets merged with this DSERN technology, or SPAD-QIS evolves for general purpose photography and video, or some of both. It is just an exciting time to be working in image sensors.
DeleteFor imaging (not time of flight) Gigajot's technology is obviously superior however you look at it. SPAD is good for ToF applications, but with Gigajot's recent product announcements, SPAD doesn't have any chance in getting to consumer or security imaging as Canon is trying to market for. Dead time in a SPAD and low effective QE are two big cons. Eric, you should ask Gigajot for their reasoning for the table that I think is accurate.
DeleteI think the misunderstanding comes from the application domain.
DeleteThe common definition of a "photon resolving detector" is a detector able to record the number of photons detected, even if simultaneous. It is not written anywhere I guess, but it's established in the high energy physics community, where detectors are usually hit by ultra short bunches of photons.
With this preamble, if we talk about imaging instead of particle physics, I think that the table in the paper is surely incomplete and biased (you all, admit you did it at least once!).
It would have been much more fair to talk about the max photon counting rate: for jot QIS it is virtually infinite, while for SPADs it is hundreds of MHz. Then science wins and everyone is happy.
Last comment: I do encourage anonymous users to sign their comments, it's easy and respectful. Thanks!
Matteo
QIS ability to count beyond one photon is a nice hedge against missing the two consecutive photons, that even unlikely, can happen at a very low light levels. And with more light more reads per second are needed in order to prevent saturation of the multi bit QIS pixel. With higher light level probability to have the pixel saturated for some samples is higher, and may become relevant only when coming closer to overall saturation.
DeleteThe figure shows the QIS image (a) with higher SNR than the CIS image (b) after both images have been resized to the same resolution. QIS image has been downsized more than the CIS image and, in the process, it's noise has been reduced more.
ReplyDeleteI don't see how you can claim that the QIS outperforms the CIS despite having 4.8X smaller pixels.
I wonder if you are accounting for the fact that although the pixel resolutions are different, the pixel pitch is different too. So my guess is that showing the two images re-scaled to the same physical dimensions is not as unfair a comparison as it may seem at first sight.
DeleteThe QIS with an area of 16.7*1.1^2=20.2mm^2 is demonstrating a higher SNR image than the CIS with an area of 6.4*2.4^2=36.9mm^2. The superior SNR performance of QIS is clear.
DeleteUnless there has been no denoising algorithm applied, this means QIS SNR is clearly higher (not sure by how much) than CIS used for comparison, and QIS has at least 36.9/20.2 = 1.8 times lower noise (read and dark in total). I wonder what's the noise of CIS in figure 2.
DeleteBased on the pixel pitch and resolution, my guess is that the CIS used for comparison is a Sony IMX178 image sensor. Its read noise is 2.4e- and absolute sensitivity threshold is 2.9e-. (http://softwareservices.flir.com/BFS-U3-63S4/latest/EMVA/EMVA-Local.html)
DeleteIf CIS image has noise close to 2.9e- threshold, the QIS image shown has several times smaller threshold - considering visibly less noise and higher resolution at the same time, even at 1.8x less area under the same lens used. So its absolute sensitivity and read noise are a fraction of an electron.
DeleteIn my opinion, QIS is not that different not CIS. It just has very low readout noise, plus it uses time & spatial oversampling. This is still just a method to calculate the light intensity.
ReplyDeleteBut you know, sometimes you need a new name even for the same thing. Because new name could mean 'new' money. People like to invest into new stuff. Like AI, is that very different from programming? Like Meta, is that very different from VR? No! But you need a new name for them because the public does not care much about details.
"... just a method to calculate light intensity..." is a gross oversimplification in my opinion. For the sake of slipping down this slippery slope, I could use the same logic to argue that there isn't much difference between photographic plates and CIS.
ReplyDeleteIt is true that single-photon detection technology has been around for a long time, so one may argue that it's not really new (e.g. photomultiplier tubes have been around for over 100 years). But the ability to capture information at single-photon level that both QIS and SPADs can provide in a cheap and scalable way, with high spatio-temporal resolutions while maintaining a small form-factor cannot be summarily dismissed. Even if this technology never replace CIS, it'll increasingly play a complementary role in the years to come.
Hi,
ReplyDeleteinteresting indeed!
I also stumbled across the Hamamatsu qCMOS camera that also claims to be a "photon number resolving" detector. Any experiences or further insights here? I wonder how this technology and the QIS differ.
Cheers
Jack
That's a great question, Jack, something I have been wondering about too.
DeleteI consider Hamamatsu's qCMOS as a new generation scientific CMOS (sCMOS) camera. sCMOS cameras, as far as I understand, are a special type of CMOS image sensors that are manufactured with even more stringent fabrication methods to achieve sub-electron read noise. Some sCMOS cameras also require active cooling.
I'm tempted to clump qCMOS under the umbrella term "single photon cameras" (which includes QIS, SPAD, PMT, EMCCD, ...) I'd be curious to see how low light image quality of a state-of-the-art sCMOS camera compares with QIS/SPAD-based single-photon cameras.
Who's the winner in the end? I believe the answer will depend on the application since, in the end, it's a multifaceted problem with technical and economic constraints.
Both QIS and qCMOS look to me like an ordinary CIS having high CG. What's the definition to distinguish?
Delete(I remember Originally QIS was intended very dense and high speed pixel array with 1bit ADC).
QIS has a very high sampling rate compared to CIS.
Delete