From Transformative Optics Corporation: https://www.ai-cmos.com/
ai-CMOS
sensors solve many of today’s challenges with antiquated CMOS
technology, offering unprecedented accuracy, an expanded spectrum, plus
9-channel AI-optimized color. Extending beyond the visible spectrum into
near-ultraviolet (NUV) and near-infrared (NIR) greatly expands
capabilities for mobile photography, autonomous transport, and machine
vision.
With higher sensitivity than Bayer sensors, near-complete
color gamut, and expansion beyond visible light to near-infrared and
ultraviolet frequencies, ai-CMOS brings a unique multispectral ability
to standard cameras.
Mobile Photography.
Close the gap between performance and portability, while unlocking new potential for AI-powered apps.
More Contrast: Improved Black and White Modulation Transfer Function (MTF)
Broader Spectrum: Extension to Near Infrared (NIR) and Near Ultraviolet channel
Near-Complete Color Gamut: Improving color accuracy, automated white balance
Enhanced
Sensitivity: Twice the Light. Lower light levels, less motion blur,
plus twice the signal levels for a myriad of Integrated Signal
Processing functions.
Machine Vision.
ai-CMOS offers AI applications richer and more complete data sets for training, object detection, and object classification.
Richer Data: 3x the information over Bayer
AI
Optimizations: increased raw data content for feature vectors and 2x
the signal strength for Integrated Signal Processing aiding apps like
Super-Resolution
More Contrast: Improved Black and White Modulation Transfer Function (MTF)
Broader Spectrum: Extension to Near Infrared (NIR) and Near Ultraviolet channels
Automotive.
ai-CMOS captures more detailed data in low-light conditions, at night, and in poorer weather conditions, like fog and rain.
Spectral Sensitivity: ai-CMOS captures twice the light of current ADAS CMOS technology on the market.
Object
Detection: 25% color gamut increase and 3x Feature Vectors from
traditional sensors, greatly enhancing object detection and
classification.
Autonomous Driving: Better enable autonomous vehicles
to navigate more complex environments, and interact with other vehicles
and pedestrians.
Sensor Specs.
Resolution: 3000 x 3864
Pixel Size: 8um
Sensor Format: 35mm (dia. 39.3mm)
Spectral Response: 350nm to 850nm
Quantum Efficiency: >90%
Illumination-type: BSI
Frame Rate: 30fps in HDR
Full Well: >65,000 e-
Gain Mode: HDR and Dual Gain
Available in limited quantities in 2023.
Why are there 9. It seems enough 3 Bayers and 1 for NIR
ReplyDelete"TOC has developed a vastly superior 3x3 array of filtered pixels, called TuLIPs (Tuned Linearly Independent Polynomials)"
DeleteI seem to recall that Sony has such a 2x2 RGB-Ir pixel sensor. It is good for some applications such as machine vision and security cams, but the 3x3 sensor give much better color accuracy.
DeleteTruly hyperspectral cameras (100+ bands) have potential applications in many fields of research such as astronomy and chemistry - any field that uses spectroscopy, as well as for satellite imaging and other classified applications. The data rates for deep hyperspectral video quickly become unmanageable, but I have a light-efficient, compressed-sensing approach for 1000+ spectral band, ~8MP video using a small number of 1k-4k linear pixel arrays + fast COTS image dissectors. The imaging and spectroscopic parts of the design have been proven and published separately.
Enon Harris (contact via the obvious gmail address or Linked-In)
Thank you. Maybe it is really for a hyperspectral camera. But can you tell me what is the advantage of a hyperspectral camera with 100 spectra - over the usual prism and the decomposition of light into an infinite number of spectra?
DeleteBy having a spectrum for each pixel, especially in the IR, one can identify the materials that each thing in the image is made of. (Regular spectroscopy is only for a single point or small number of points, so is much more limited.) image segmentation and object recognition is easier. Camouflage and paint are easily distinguished from dirt and vegetation. Artificial images from projectors and other displays can readily be distinguished from reality. Different types of vegetation and different states of vegetation health are readily distinguished. Minerals such as ores can be identified. One can identify the fuel and impurities in missiles being launched. With sufficient spectral resolution, one can even get the velocity of objects.
DeleteSpectrometers don't use prisms any longer, but rather diffraction gratings, which improve the spectral resolution. Neither has infinite spectral resolution - there is a fundamental tradeoff between spatial and spectral resolution, closely related to the Heisenberg uncertainty principle. My HS camera design has 1k-4k spectral lines maximum resolution, but lower res. for smaller objects. (Because of its design, though, if one accepts a dead-zone around a small object, one can get higher resolution spectra of small objects.) It is possible to zoom in on smaller regions of the spectrum. It's pixel-agnostic, and because it uses line sensors, it can use exotic pixels that would be impractical or expensive to make in large focal plane arrays. It measures both the accepted and rejected spectra for each pixel (each pixel has a 1k-4k spectral line tuned filter that can be changed arbitrarily at ~100Hz - 30kHz, sending selected lines to one line sensor and rejected lines to another sensor), allowing light efficiency and accurate photometry. The camera design also intrinsically allows simultaneously variable effective spatial, spectral and temporal resolution within an arbitrary number of areas of interest within the image. With off-the-shelf parts, parts cost for a prototype would be $5k - $50k, depending on resolution. (Custom line sensors, diffraction gratings etc. would be much more, of course.)
Thank you. This approach is quite difficult to me)
DeleteHi Ivan, thanks for your comment
ReplyDeleteWhile it may initially seem that 3 Bayer channels and 1 for NIR could be sufficient, the 9-color channels in ai-CMOS offer a significant advantage in terms of data richness and accuracy. For instance, the expanded color channels enable AI and machine vision systems to access a more comprehensive range of spectral information, crucial in various applications.
There are several reasons for 9 channels: 1) Enhanced data richness and accuracy, 2) A comprehensive range of spectral information, 3) Multispectral capability (NIR & NUV regions) 4) Ideal for advanced applications (autonomous driving, precision agriculture, remote sensing) and 5) Supports sophisticated AI-powered solutions across industries.
Dear Bryan,
Deletecan you name which 9 filters are used in each pixel, what colors or lengths of light waves
I presume TOC filed to protect their IP a while ago. After that, for new technology, it is usually better to share the stuff that will be obvious as soon as you send the first sample or sell the first product, with the technical community and build support for the new approach. (e.g. CFA pattern and spectrum) If it is a strong contender, it will be easy to defend against the inevitable arrows that will come your way early on. Being mysterious and claiming a lot without revealing much does little to create expert champions of the technology outside the company, which is what you want when you start approaching strategic partners and significant funding sources. I do a lot of work with startups and technology licensing in my role at Dartmouth. I try to encourage companies to build technical community support as one of the foundations in the success of new technologies, after protecting their IP of course. In the case of TOC, publishing a decent conference or journal paper or two would go a long way to developing credibility in the technical space that helps counteract any marketing hype.
DeleteDear Eric,
DeleteThank you for taking the time to provide your valuable insights. We greatly appreciate your perspective, expertise, and the guidance you offer to startups like ours. I believe an old friend of mine went to work for you way back in the day, Vladimir Berezin.
We wholeheartedly agree with your points about striking a balance between protecting IP and sharing information with the technical community. Building support and fostering collaboration with experts is critical to success. We have been working in stealth mode, under NDA, with several large manufacturers. To this extent, we have experienced plenty of expert arrows already. The request to post information to the image world blog seemed opportune, nothing too much more than that.
We do expect a great deal of detailed information to start to publish as a consequence of our more than a dozen patent filings to date. Likewise, as you state, much of the tech will be quite obvious once first articles start to ship. We will confer internally and debate the idea of publishing curves and layouts earlier than our original plans.
Best regards,
Geoff Rhoads
(Founder of Digimarc back in the 90’s)
hi Geoff, Thanks for your comments and I wish TOC success! You have a great track record and the credibility of understanding imaging from a computer vision career. And yes, in the Photobit days, we were lucky to have a very strong technical team including Vladimir. -Eric
DeleteThis comment has been removed by a blog administrator.
DeleteThis comment has been removed by the author.
DeleteWhy Sean’s comment about OV vs Photobit is removed? It was very informative.
DeleteNo, don’t share your secret tech for free to the public. Keep it secret as long as you can. I know many companies don’t file many patents just to keep the ideas and inventions a secret. Patents really don’t give you as much of the defensibility as you may think. Especially for a small startup. Having a good patent portfolio can give you freedom of operation, but that’s another story.
DeleteThese xy diagrams are very perplexing: how can they have points outside of the spectral locus ?
ReplyDeleteBecause the outline is generated by the matrix (presumably 9 x 3) that converts the signals from the sensor into tristimulus coordinates. The matrix would be designed to provide best accuracy under some criteria but wouldn't be perfect anywhere. Points outside the spectral locus are just part of the error band.
DeleteRegarding publications and patents (in response to an anon. comment since removed):
ReplyDeleteAt the beginning of the JPL CMOS image sensor work, we had no intention to create a company. JPL is a taxpayer-funded enterprise known as a Federally Funded R&D Center (FFRDC). Publishing papers and transferring knowledge to others was part of the FFRDC mission. Caltech, which manages JPL, also filed patents on JPL-developed IP as it is obligated to do under the Bayh-Dole act. It was only later that the company was created, and even then it was for custom chip development. I am sure the early publications and papers aided our competitors, but they also helped establish some credibility for the new technology. Years later, I believe the JPL IP generated significant revenue for Caltech.
Why other people comments are deleted and only Eric Fossum can post comments? Who is in charge of this blog now? Is it Vladimir?
DeleteThe comment has been deleted due to a personal attack. The definition of attack is a subjective matter. What seems innocent and informative to some people, might be offensive to others.
DeleteHow come that was a personal attack? Comparing OV with Photobit is personal attack? Sean’s comment was stating only facts. Was Photobit sold for less than 3x revenue? Yes. Why this is an attack?
DeleteEverything is relative. Facts can be carefully selected. Some facts might be omitted or are just not known to the commenter. If Eric sees this comment as offending, it's probably true
DeleteIMO, it would be great if we know more about the early CIS companies. I did some research and could not find good information of Photobit besides what Eric has published. It’s beneficial for the community, especially younger crowds to learn from those stories from different sources and point of views. Photobit vs OV can be a good start. What Sean shared was interesting which I didn’t know. Every company makes mistakes which others can learn from. OV became very successful and I’m sure there are reasons for that. Why Photobit had a not successful exit has lessons in it for the imaging community to learn from.
DeleteIn this digital time and age, facts and opinions cannot be censored. I personally now have a reason to do more research about Photobit vs OV and will publish it on a free platform for others to see freely.
DeleteWhile doing your research, please add VVL, IC-Media, Y-Media, Pixelplus, Smalcamera, Biomorphic, and many others to get a complete picture
DeleteI’m honestly surprised someone from the great country of democratic Israel censors other people opinions.
ReplyDelete