The above slides complete the description of the noiseless frame summation general idea. There are few practical issues that need to be addressed though:
Wednesday, November 25, 2015
Noiseless Frame Summation in CMOS Sensors
As promised in comments to the earlier post on sub-electron pixel noise, here is the presentation on the noiseless frame summation in the regular charge-transfer pixel that can allow DR expansion while maintaining the usual 4T pixel noise level:
The above slides complete the description of the noiseless frame summation general idea. There are few practical issues that need to be addressed though:
The above slides complete the description of the noiseless frame summation general idea. There are few practical issues that need to be addressed though:
Well, certainly an interesting approach.
ReplyDelete1. Not noiseless at all. I think what you meant was noise<shot noise.
2. For small & intermediate level signals, you have to decide that NO signal was skimmed vs. an itsy bitsy signal was skimmed. This could be tricky. Basically you still have to have pretty low noise on the readout.
3. You have no information on low light signals until all readout frames have been performed.
Still could be useful in some cases. I hope you are able to find some licensees! Often though, as Rambus and others have discovered, you have to actually build it and show image improvement before anyone comes on board. And even then, they may come up with some competing scheme. All in all, it really needs to be so good that everyone wants to use it. Maybe this is one of those cases and you are already wealthy!
And I should say some parts of the scheme do remind me of parts of those super long Rambus binary pixel patents. Hopefully there is no interference.
I think the idea is this: skimming can not reduce remaining charge below 50%. And nothing useful is skimmed off if the charge is under 50%.
DeleteSo, on final readout, if the final readout is much lower than 50%, we know for sure that no charge was skimmed off. Thus, any skim result present is noise and can be discarded.
Thank you for your quick feedback and good points, Eric. Let me respond to your points:
Delete1. Not noiseless at all. I think what you meant was noise [less than] shot noise.
True, the added noise is masked by the shot noise, so that, for a consumer photography case, one can not distinguish between this case and an ideal noiseless summation case, unless doing special very accurate measurements.
2. 2. For small & intermediate level signals, you have to decide that NO signal was skimmed vs. an itsy bitsy signal was skimmed. This could be tricky. Basically you still have to have pretty low noise on the readout.
No, this is misunderstanding. For simplicity, let's look at the one-level skimming case. One can set the skimming level at half full well, for instance. Say, the full well is 3,000e- in a small pixel, and we set the skimming level at 1500e-. Then, one can expect that signals below 500-600e- (one-third of the skimmed level) are not skimmed, and, as we move above that 500-600e-, portions of the PD signals become skimmed more and more, and most of the signal is skimmed above 1500e-.
So, we set a threshold for frame summation at 500e- or 400e, where we are sure that there is no skimming. Below this threshold, we know that the whole signal is in full read, and there is no need to add skimmed frames. Above this threshold, the skimmed frames might or might not have part of the signal, but we add them anyway, just to be sure that no signal is missed. At that point, the shot noise is pretty high, 20e- for 400e- threshold. So, if we have 2e- read noise and sum 10 skimmed frames, the accumulated read noise is quite close to the shot one.
3. 3. You have no information on low light signals until all readout frames have been performed.
Very true. The digital signal from skimmed frames is accumulated and stored in SRAM till the last full read frame arrives. Then, the decision logic has all the data to work. One needs a frame memory for that. I guess any digital accumulation scheme needs a frame memory of some sort.
Often though, as Rambus and others have discovered, you have to actually build it and show image improvement before anyone comes on board. And even then, they may come up with some competing scheme.
We are a design services company. We are not looking for IP sell deals.
@ Oh wat mooi comment:
DeleteTrue, to an extent. The skimming is a thermionic emission process which is affected by temperature. Normally, we get kT/q-like slope of the thermionic emission. For a typical PD, the pinning voltage is about 1V. So, for a PD potential difference of 200mV, roughly 1/5 of the pinning potential, the skimming changes by 3 orders of magnitude or so.
So, I would estimate the transition from skimming to no skimming as a relatively small fraction of the full well.
any possibility of posting a PDF version that can be read by those of us squinting at eye chart tests?
ReplyDelete|-)
One can click on images and see an enlarged version of it. Does this work for you?
DeleteI think this is already patented by major players so intellectual property infringement is likely.
ReplyDeleteWell, it might happen. Would you mind to post links, patent numbers or some other info to support your statement?
DeleteI thinks patent US8582010B2 has similar idea.
DeleteYes, indeed, it's the same idea. Shame on me, as I look through the new patents every week for many years, but I've missed this one.
DeleteNow I understand why ARRI used to lead in DR for many years. I wonder if RED Dragon sensor too uses a similar technique to extend its DR.
Vlad, I am often in this position. I find some new idea was already patented. Usually by Jerry Hynecek. Once by my team at Photobit! But, sometimes it is also the case that at first read it seems like the same idea, but with a more careful read you find there are subtle differences that make a big impact on performance. Be sure to read carefully to see what differences might exist, and then assess if it is worthwhile pursuing or not. Most importantly, keep on innovating and pushing the envelope!
DeleteThank you, Eric. I keep thinking about the subtle differences, as you propose.
Deletequite useful for high light condition, although skimming itself will bring some RTS noise. But the method itself is like putting an ADC before SF, for sure it could reduce more noise.
ReplyDeleteThe RTS remark is actually a very good point. The patent has few extensions that I skipped as the slides became way too long.
DeleteOne of this extensions deals with RTS noise. Basically, it looks for a big difference between the subsequent skimmed samples. If there is an abrupt jump between one sample and the next, it might indicate RTS event. Then, digital state machine can implement a number of options: drop one sample out of many and increment the memory content, mark pixel as a bad pixel, or few other options.
Another interesting extension is a Xenon flash remover. As Xennon flash is typically few 100s of us long, we can look for a big difference in the subsequent skimmed samples to recognize it.
There are few other extensions too, I'm just trying to keep it short. Otherwise nobody would read this.
How about power consumption?
ReplyDeleteActually, our HDR proposal came primarily to leverage our low power voltage mode readout and high speed current mode readout with no PRNU. The first one consumes about 18-19mW for a 16MP, 30fps sensor. Marking it 300fps brings the power in range of 200mW, including the frame summation memory. It's not that low, but probably acceptable for an additional 20db of DR.
ReplyDeleteThe second current mode no-PRNU readout consumes more power, but it's about 20x faster. With it, one can make 1080p 60fps sensor with 40dB DR extension at about 900mW. It's is within reason for some applications, assuming that there is no motion and light flickering artifacts, no SNR drops, etc, as every and each photoelectron contributes into the final read out value.