"Nature has a large repertoire of animals that take advantage of naturally abundant polarization phenomena. Among them, the mantis shrimp possesses one of the most advanced and elegant visual systems nature has developed, capable of high polarization sensitivity and hyperspectral imaging. Here, we demonstrate that by shifting the design paradigm away from the conventional paths adopted in the imaging and vision sensor fields and instead functionally mimicking the visual system of the mantis shrimp, we have developed a single-chip, low-power, high-resolution color-polarization imaging system.
Our bio-inspired imager captures co-registered color and polarization information in real time with high resolution by monolithically integrating nanowire polarization filters with vertically stacked photodetectors. These photodetectors capture three different spectral channels per pixel by exploiting wavelength-dependent depth absorption of photons."
"Our bio-inspired imager comprises 1280 by 720 pixels with a dynamic range of 62 dB and a maximum signal-to-noise ratio of 48 dB. The quantum efficiency is above 30% over the entire visible spectrum, while achieving high polarization extinction ratios of ∼40∼40 on each spectral channel. This technology is enabling underwater imaging studies of marine species, which exploit both color and polarization information, as well as applications in biomedical fields."
A Youtube video shows nice pictures coming out of the camera:
Victor graciously came to Dartmouth and presented his work to the Engineering School. It is a nice assemblage of techniques to get the information needed in a compact way. I think we all know the limitations of stacked PN junctions for color imaging, but in some applications, such as Victor's, it works fine and offers some advantages to system design. BTW, their photodetector does look like Foveon's early approach US5965875A, which in turn looks similar to US5883421A filed a year earlier than Foveon by a French group. Victor does cite a Foveon patent in his references. But, no one has combined this with a polarization filter array and UV detection to the best of my knowledge.
ReplyDeleteThe limitation of such approach is that it is not very practical for reasonable yield production. Modern ASIC manufacture is good at making precise highly integrated nanometer-small designs but only spatially (in XY direction). When it comes to trying to make the same vertically (in Z direction) as this approach (and Foveon's) describes - unfortunately it can't be made very precise. Yield suffers and it becomes not very business practical.
ReplyDeleteMy understanding this was the reason Foveon failed with their sensor technology. It's good on paper, but in practice can't be manufactured at reasonable cost.
I have heard many stories about Foveon's difficulties in gaining market traction and have had a few conversations with Federico Faggin but I have not heard this one before. I looked at your linkedin profile and was trying to figure out how you would come to know this, as yield issues are usually well-guarded company secrets. Also, what is the yield issue? Is it color reproducibility? Is it short circuits? Is it dark current? I don't think the structure is that complicated that it would result in a yield issue but now you have me very curious.
DeleteThe layer stack could be made by different combinations of epitaxie with diffusion doping and deep ion implants and possible for the top layer with a pinned photodiode. The yield would be mainly the number high leakage or shorted diodes above an acceptable level. Epitaxie as well as high energy ion implant generate crystal dislocation which are responsible for band lowering and leakage.
DeleteI have no doubt that the yield would be lower because of the higher number of epitaxie or high energy implant steps but it must be acceptable for sell them.
Sorry, wrong conclusion. The devices are still in full production at reasonable yields for current Sigma cameras. Three different fabs have built these devices successfully.
DeleteThe market failure was due to the VC funding model that requires significant penetration into consumer markets to support a major IPO. Foveon built sensors for DSLRs (outside Sigma), then point-and-shoots and then mobiles. All of these failed in the market but not for sensor availability or price reasons. One big factor was that the Foveon sensor outputs did not fit into the data models required by commercial processing chips.
There were some minor market successes - one F13 sensor is on the ESA CLUPI camera headed for Mars in a couple of years, for instance - but these were not enough and so Sigma bought Foveon for its own use.
Hi Eric,
DeleteI don't have any inside knowledge, as I said it my "understanding", something one can develop from analyzing information available. I've too had discussions with former Foveon engineers (most of them were at engineering level, not executives), and you know sometimes engineers can know better ;-) since they are hands-on with the problems vs executives who just get reports..
I think in this case you can define yield as when the device falls outside of target parameter set - like it is here the paper says 2 um thickness for green, 0.8 um for blue - my question is how precise this thickness can be guaranteed by the manufacture process? What if in reality you get something like 0.3-1.1 um range for blue, and 1.5 to 2.5 um range for green. It can be device to device variation or pixel to pixel, which I think would be worse. Then the color reproduction suffers - the variance must be corrected by difference CCM, which is not very trivial to implement. Possibly to some limit the problem can be fixed with individual sample calibration, but sample calibration is expensive, and this I know for sure.
Hi Dave,
DeletePossibly I am wrong, but I have a feeling I am right to some extent :-) When you say "One big factor was that the Foveon sensor outputs did not fit into the data models required by commercial processing chips" it can describe the same thing. Being able to manufacture within strict parameters and fit with ISP processing models, or have higher variance that can't be corrected by existing ISPs - you have to scrap a lot and increase per unit cost.
Igor,
DeleteCommercial ISPs expect Bayer data and provide a set of functions that did not include the corrections needed by the Foveon data. No one wanted to modify their ISP so Foveon designed their own and put it in a mobile camera module with their F33 and F43 sensors (never generally marketed). They even went to the length of making a sensor that had a Bayer emulation mode so they could test their devices with commercial ISPs. The processing worked but the images had, of course, the problems associated with both Bayer and Foveon technologies. However, there were no yield problems with these chips, either.
Dave
Hi Dave,
DeleteGood to know, and thanks for the explain. I am just suspicious - there is a number of papers that describe layered pixels with vertical structures to separate colors. But none were successfully commercialized as far as I know. In addition to Foveon I know at least one more company that tried to make an X3-like sensor that failed (it was a lesser known Asian shop so I guess they didn't worry too much about Foveon IP).
It's why I make the conclusion it must be too expensive (low yield) to produce.
BTW, why Foveon wouldn't go into automotive and machine vision applications as of today? Clear pixel (RCCB, RCCC, RCCG) sensors show good advantages from their increased sensitivity, and when it is not required to output human color. Since it is the conversion to human color that imposes the big penalty to SNR for these non-Bayer sensors. For X3 in low-light there could be an interesting algorithm to drive color channels into monochrome mode, similarly how human vision does it with the photopic-mesopic-scotopic transition.
Igor.
Igor,
DeleteI asked them most of those questions and even wrote specifications for some industrial/scientific sensors for them. They thought these were nice ideas but did not represent a big enough market to meet their commitments to the VCs. It was too early for automotive at the time.
There is also a technical limitation for making complex chips like HDR and global shutter. They can be designed but all of the circuitry for all three layers has to go on the same plane. This means that to get a reasonable fill factor, the minimum pixel size would be about 12 microns. This would make the sensors large and expensive even for modest resolution. This also made in-pixel CDS noise cancellation a geometry problem.
We did have a couple of customers who switched between color and mono - just by adding the channels.
Dave
as many of you know, there are a few more polarizer image sensors with 45° shifted filters on every pixel. One for example by Fraunhofer: https://www.iis.fraunhofer.de/en/ff/sse/ims/tech/polarisationskamera.html. Another was presented by Sony at the Framos Techdays, I dont know if this is already a product yet.
ReplyDeleteI am not sure this Polka camera also does color separation? But if it is Bayer, then you have 12 different pixels from which to interpolate etc. In Victor's case, it is just 4. This is where the layered PN junctions provide some system advantage (and disadvantages of course).
DeleteThe Fraunhofer POLKA camera is grayscale only and is already in in-line use today.
DeleteSony announced polarization sensors, combined with color filters - but so far there don't seem to be any release plans.
Foveon F13 base sensor. Trust me.
ReplyDelete