"Clarity+ technology uses a Bayer-like pattern with high-fidelity clear pixels that collect two times the light compared to standard green, resulting in a significant improvement in captured signal," said Bob Gove, President and CTO of Aptina. "Clarity+ technology provides the leap necessary to take artifacts and noise out of the equation enabling a world-class image sensor in the mobile-ready, tiny form factor that OEMs need."
Aptina has published a whitepaper on Clarity+ technology. Clarity+ starts with the standard 2x2 Bayer RGB color filter pattern commonly used in image sensors throughout the industry, and then replaces all of the green color filters with carefully optimized clear filters. This pattern preserves the high-frequency spatial information of the Bayer pattern, eliminating the aliasing artifacts that are inevitably introduced when both green and clear filters are used in larger kernel (4x4) patterns. The green color is determined through the image processing algorithms that use subtractive and interpolation techniques. Specialized noise reduction is then applied, resulting in a 3dB improvement in low-light SNR, relative to that possible with the RGB Bayer pattern.
"Aptina’s Clarity+ CFA pattern (RC/CB) is unique in that we have the same number of B and R pixels as Bayer, but determine G through a subtraction of R and B from C. Because we have the same number of C pixels as other 50% C schemes, we can extract the same SNR improvement. However, because we have the same configuration as Bayer, we uniquely maintain the high fidelity of luma information and low level of chromatic artifacts to which the industry is accustomed. Because G is determined through a subtraction, there is a tendency to amplify noise in the conversion to the standard RGB (sRGB) representation. This is addressed by image processing algorithms that are part of the Clarity+ technology."
"As with other patterns involving C pixels, there is a very slight trade-off (a few tenths of a dB) in visual noise when compared to Bayer in bright light situations, but this is not noticeable in those situations, because SNR is high and does not typically limit quality. There is the benefit in bright light situations, that Clarity+’s increased sensitivity will result in exposure times that are half that for Bayer and this is useful for capturing action shots with less blur and motion artifacts. The 50% RGBC patterns (including Clarity+) are also more sensitive to flare and other stray light issues, but this can be managed through appropriate pixel, lens, and packaging design."
The whitepaper also has many actual pictures comparing Bayer CFA sensor with the Clarity+ one.
The AR1331CP is available now for sampling in die form with raw output for use in systems that are enabled for Clarity+ processing. The AR1331CP is also available for sampling with Clarity+ coprocessors that can either fully process the unique Clarity+ images into industry standard camera formats or “pre-process” the Clarity+ images into Bayer format for compatibility with standard image and application processors.
I am having trouble understanding why SNR improves fully by 3dB (half the signal improvement) instead of a bit less. Noise is introduced when R/B is subtracted from W, just like when you have negative off-diagonal elements in the CCM or have complementary colros. Perhaps Aptina can comment. Are there experimental results on measuring SNR or YSNR? I saw images in the white paper but no scientific data.
ReplyDeleteI guess they compare the "clear" signal vs signal with a green filter, but as you point out the green signal after CCM will be much more noisy than using a regular bayer pattern.
DeleteThey're probably not comparing the raw signal but the signal after noise reduction. One then has to ask if they gave bayer the best possible noise reduction as their paper implies it was subject to the clarity engine.
DeleteI am sure I was not the first but I note that the RWBW kernel was one I investigated when I was mostly looking at 2 layer structures. Of the single-layer-structure kernels, I found RWBW to perform about the worst wrt YSNR. I am not sure Aptina is doing the same thing, but just saying...
DeleteSee Table II in: http://ericfossum.com/Publications/Papers/2011%20IISW%20Two%20Layer%20Photodetector.pdf
I noticed that RWGW (where you get B by subtracting R/G from W) gave better YSNR results probably because the blue noise does not play into the luminance signal too much.
Lastly, I had trouble getting low color error on both RWGW and RWBW but probably Aptina has better color specialists.
Great question Eric! Given yours and the industry’s experience with subtractive and complementary color schemes, it’s understandable that further explanation is necessary.
DeleteAs you’ve inferred, image processing is key. Without appropriate processing, RCCB performs worse because of the subtractive process. As a part of the Clarity+ innovation, we manage noise in a way that takes advantage of the high SNR clear channel. We perform sophisticated chroma de-noising with a reasonable kernel size to reduce noise prior to color conversion/correction. In contrast to complementary color schemes, because the high SNR clear channel is used, the remaining chroma noise has other properties that are beneficial in the later color conversion. Finally, we have a subsequent filtering/mixing step where the Clear channel is used again after color conversion. As others have noted, this processing can also be used to benefit Bayer RGB and we’ve found that it does help SNR in that case by ~1dB. However, in the Clarity+ case, we found this processing pulls out much more improvement when the Clear channel is used, with a net 3-4dB improvement over basically identically processed (including noise reduction) Bayer RGB raw images. We have confirmed this in practical/experimental cases over many situations.
I anticipate that you probably want more details. I’ll reach out to you by email so we can continue the discussion.
Thanks Shung Chieh for your more detailed explanation. I guess with the right NR algorithm you can get any SNR value you want, and making the trade offs with visually appealing algorithms is where the rubber meets the road.
DeleteI am thinking this approach would work particularly well for sub-diffraction limit (SDL) pixels...well, I have been thinking that for a while wrt to RGBW systems.
The car image looks almost pinkish compared to bayer. Is it clarity+ color reproduction problem?
ReplyDeleteI think they've used different lenses or at least different apertures for the test shots, which is rather silly. The resolution looks worse to me but its clearly because the lens is a little out of focus or has not been stopped down because of the directionality (see the loop on the six). The image with the car has purple fringing from lateral chromatic abberation so perhaps not focused the same or a different lens/aperture?
ReplyDeleteIt is simple. You calculate the luminance without subtraction from RBC and the chrominance from a larger kernel to reduce noise.
ReplyDeleteClear pixels will saturate much quicker than the lower-response, blue and red pixels. Do they have any comments on this speed imbalance?
ReplyDeleteblooming is concern.(^^)
DeleteIs it just me or the resolution chart on page 7 actually shows the Bayer to have a much clearer image than the clarity+
ReplyDeleteI suspect the focus is out or the lens is stopped at a different aperture as the blur is uneven (more towards the edges) giving an indication its the lens. Unfortunately any loss of detail by clarity is also lost in this noise.
DeleteI see another problem with this type of filters - dealing with the lens chromatic aberration.
ReplyDeleteAndrey
I think this a very valid concern.
DeleteI was often wondering why this kind of pattern is not used, as you get almost directly YUV output. I think Y noise is more important than G noise, as in many camera systems an RGB to YUV conversion will be done anyway.
However, with the chromatic aberration problem you mentioned this (and other white/clear pixel approaches) will be more critical to lens performance and might hardly be usable for eg. zoom lenses.
I am wondering if this pattern can identify high frequency green & black strips.
ReplyDelete