2021 in Review

Here is my list of the most significant achievements in image sensor industry this year:

• New theory of RTS noise based on multi-atom metastable defects in Si crystal structure 
• If correct, it might cause a paradigm shift in the ways to reduce RTS noise
• Opens a way to simulate and analyze various ways of RTN suppression on computer models
• This is my personal favorite of 2021 that hopefully would turn into commercial product achievements in 2022-25
• Gives a hope that RTN would be eradicated in future devices
• Samsung 17nm FinFET process for stacked CIS logic die
• Samsung 2.2V supply pixel is in mass produced 108MP smartphone sensor
• Samsung Advanced Institute of Technology presented its version of 0.8um color-routing pixel
• Sony splits 4T pixel transistors between two layers in a stacked sensor
• SPAD progress:
• Sony created 6um-pitch SPAD with 20.2% QE at 940nm
• Canon presented 3.2MP SPAD sensor
• LiDAR SPAC parade continues:
• Quanergy SPAC
• Aeye SPAC
• Indie Semiconductor SPAC
• Cepton SPAC
• LiDAR consolidation: Sense Photonics has been acquired by Ouster for $68M in stock with 5-year reverse vesting for key employees
• China News:
• Galaxycore IPO makes CEO Stanly Zhao a multi-billionaire
• High productivity: Smartsens completed more than 30 tapeouts in 2020, 31 tapeouts in 2021, released 36 new products in 2021
• Smartsens IPO at $4.4B valuation
• Tyrafos raised "hundred of millions of yuan" investment

In 2021, Smartsens Released 36 New Sensors to Mass Production, Completed 31 Tapeouts

Tencent, Sohu: Smartsens reports that in 2021 it has released 36 new products to mass production, and completed 31 tapeouts of future products including 4K and 8K image sensors for security and surveillance applications.

In a separate announcement, Smartsens presents its new automotive sensors: VGA SC031AP and 1MP SC101AP, both integrated with ISP and Tx output port:

Panasonic Analyses SPAD Quenching

IEEE TED publishes Panasonic paper "Nonlinear Carrier Dynamics in a Single Photon Avalanche Diode: Stability, Bifurcation, and Quenching Condition" by Akito Inoue and Yutaka Hirose.

"Nonlinear carrier dynamics during quenching of a single photon avalanche diode (SPAD) are investigated. A Lienard type differential equation is derived from carrier continuity equations and it is solved analytically and numerically. Universal characteristics of the quenching carrier dynamics, i.e., stability, bifurcation, and quenching conditions, are analyzed on a trace-determinant plane of the Jacobian matrix and on a phase plane as functions of the quenching resistance (QR). With a finite QR or a resistive quenching (RQ), the breakdown voltage is found to be an attractor leading to a nonquenched final state. In order to produce a successful quenching, i.e., a status identified as an unstable fixed point (FP) with a zero-carrier state, two conditions are found to be necessary: 1) bias voltage dropping below breakdown voltage to ensure carrier decrease and 2) carrier extinction (CE) in this carrier decreasing period. The two conditions together lead to a threshold of QR. On the other hand, a capacitive quenching (CQ) appearing as a special case of RQ with an infinite resistance is found to show a completely different bifurcation character. CQ is derived to be equivalent to a logistic equation giving a transcritical bifurcation at the breakdown voltage and a final state identified as a stable FP with natural CE. Finally, two time constants both governed by an excess voltage are derived. In particular, one of them, a lifetime of impact ionizations, is found to be equivalent to the “avalanche frequency” of an impact ionization avalanche transit-time diode (IMPATT)."

Stratio Combines its SWIR Sensor with AI Algorithms to Detect Fakes

BusinessWire: Stratio will work with South Korea’s National IT Industry Promotion Agency (NIPA) to create a first-of-its kind app that detects design infringements. The app instantly matches photos of suspected items to similar designs from an IP database to obtain a similarity score. This is possible through a backend which utilizes several of Stratio's imaging technologies, including algorithms for object detection, image retrieval, and report generation.

"Using visible imaging technology with the help of AI is an important milestone in our technology roadmap,” explains James Lee, Co-founder and CEO of Stratio, Inc. “In terms of what's next for Stratio, Inc., we plan to merge this AI capability with our upcoming shortwave infrared (SWIR) camera BeyonSense to visualize the invisible, and open up a new world of possibilities using our proprietary SWIR image sensor.”

CIS Companies Among Largest R&D Spenders in China

JW Insights publishes a list of top R&D spenders among semiconductor companies in China. Image sensor companies Will Semi (Omnivision) and Goodix take 3rd and 5th places respectively:

Goodix also pays one of the highest average salaries in China, according to another JW Insights report. Goodix average half-year salary is $45K in the first half of this year. This means that its average annual salary is $90K:

According to yet another JW Insights report, Will Semi is one of the most valuable companies among the mobile supply chain players in China:

JW Insights also ranks Will Semi the second in size among semiconductor companies in China:

More About Canon 3.2MP SPAD Sensor

Canon publishes a press release about its SPAD sensor with 6.39 μm pixels presented at IEDM 2021:

"Canon Inc. announced today that the company, thanks to a proprietary pixel architecture that efficiently captures and greatly multiplies light, has successfully developed an ultra-small 13.2 mm x 9.9 mm SPAD sensor capable of capturing the world's highest1 resolution of 3.2-megapixel images—a higher resolution than Full HD (approximately 2.07 megapixels), even in low-light environments. Manufacture of the new sensor will commence in the second half of 2022.

The newly developed SPAD sensor employs a proprietary pixel architecture that reflects photons inside the pixel in order to effectively detect photons across the entire range of the effective pixels. Under equivalent light, this SPAD sensor can capture the same images as a conventional CMOS sensor while requiring only 1/10 of imaging area. This makes possible an ultra-small design that can be installed even in small devices and greatly increases sensitivity, including for light on the near-infrared spectrum, and realizes video capture with 3.2 megapixels under low-light conditions of 0.002 lux—darker than a starless night sky. By equipping cameras designed for low-light and monitoring applications with this new SPAD sensor, even video footage of low-light environments can be viewed as if it were recorded in bright areas, enabling identification of subject movement as though viewing with the naked eye in well-lit environments

Beginning in the second half of 2022, Canon will commence manufacturing of SPAD sensors for use in the company's security-oriented network camera products. With the inclusion of this innovative sensor, Canon's products will gain a competitive edge and contribute to the realization of a more safe and secure society.

In addition, the SPAD sensor is capable of extremely high information processing speeds on the level of 100 ps (picoseconds, one-trillionth of a second), enabling it to capture objects moving at high speeds including photons. With a resolution that surpasses Full HD and high sensitivity that enables capture of faint light, the sensor's unique rapid response functionality can be utilized in a wide range of applications including self-driving vehicles, medical treatment, diagnostic imaging devices and scientific precision measuring instruments. Canon will proactively expand its sales network in order to encourage further innovation and development of society."

Huawei Unveils ISP with AI-based Low-Light Imaging Enhancement

ITHome, GizomoChina, Huaweiupdate: Huawei semiconductor division HiSilicon presents its Yueying ISP at China International Social Public Safety Expo (China Security Expo for short).

"For the first time, the HiSilicon exhibition area fully revealed the five key capabilities of Yueying, breaking through the ceiling of traditional technology, enabling smarter and clearer picture quality in all scenes.

For the first time, HiSilicon moved the Dark Light Image Laboratory into the AMB Pavilion, allowing the public to experience the shocking effects of the latest black technology in an immersive manner. In the darkroom experience zone where “five fingers can't be seen”, one of Yueying’s key capabilities, super-sensitivity noise reduction, uses neural network deep learning to achieve intelligent noise reduction in low-light scenes, leading the industry in image quality and clarity. It also uses multi-spectral fusion technology to effectively fuse the visible light and infrared spectra, so that low-light images achieve a balance of color and detail, and reshape the colorful visual world."

Omnivision Milestones

Omnivision publishes a series of milestone videos summarizing the company's progress over the years:

Mobile Camera Module Prices in China in December 2021

SigmaIntell publishes its report on camera module prices in China and its forecast for Q1 2022:

IThome quotes SigmaIntell (Chinese name Qunzhi Consulting):

"According to the latest survey data of Qunzhi Consulting in December, in terms of low-pixel products, 2M pixel products are affected by the strategic reduction in capacity supply. It is expected that prices in the first quarter will show a flat trend. 8M pixel products, affected by the previous price increase factors, the terminal demand will continue to be sluggish in the first quarter. Qunzhi Consulting predicts that the price will show a 3-5% decline in the first quarter .

In terms of high-pixel products, for 64M pixel products, upstream head chip vendors continue to increase their production capacity for high-pixel products, and the industry's high-pixel component inventory continues to increase. Inventory clearing has become a priority strategy for chip vendors in the current period, Qunzhi Consulting predicts , The price will show a decline of about 5% in the first quarter.

The trend of multiple cameras on smart phones will continue in the future. However, due to the diminishing marginal effect of the increase in the number of cameras on the improvement of photo effects, and the increase in the cost of mobile phones brought about by 5G, mobile phone manufacturers have slowed down the upgrade of cameras."

Omdia too publishes its analysis of smartphone cameras number and resolution trends:

Howard Rhodes Lifetime Achievement Memorial

Omnivision opens Howard Rhodes Lifetime Achievement Memorial Wall in its visitor center in Santa Clara, CA:

"As our former CTO & pioneer in CMOS sensor imaging industry, Howard created more than 200 issued US patent families and was one of the founders of OmniVision technology breakthroughs. This achievement wall not only honors Howard's contributions but also inspires #FutureInSight employees to innovate and achieve more accomplishments.

Furthermore, we have created “Howard Rhodes Award“ to honor our most outstanding engineer each year. Congratulations to Keiji Mabuchi who is the winner this year and Special thanks to Rhodes' Family who kindly shared Howard's plaque patents."

Keiji Mabuchi has 123 US issued patent families (274 granted US patents), not to talk about pending applications.

A3 on Recent Image Sensor Innovations

Association for Advancing Automation (A3) publishes its list of "Image Sensor Innovations that Push Machine Vision Forward."

• ToF sensors:
• Sony DepthSense BSI family
• Teledyne e2v Hydra3D ToF CMOS
• Teledyne e2v Flash series of image sensors for 3D laser triangulation
• Prophesee event-based sensors
• Sony IMX487 UV CMOS sensor
• Sony SenSWIR sensors
• SWIR Vision Systems CQD sensors
• Emberion graphene SWIR sensor
• Gpixel GMAX32103 CMOS sensor achieving 103.7MP resolution at 24fps frame rate

ST Promises SWIR Quantum Dot Pixel Future

ST publishes an article about its recent IEDM paper "1.62µm Global Shutter Quantum Dot Image Sensor Optimized for Near and Shortwave Infrared" by Johnathan Steckel:

"Quantum dots are tiny (between 2 nm to 20 nm usually) semiconductor crystals. One of their unique properties is that their optical and electrical properties change with their size. In an image sensor, using quantum dots of various sizes makes it possible to capture different wavelengths of light beyond silicon’s absorption limitations. In ST’s IEDM 2021 paper, researchers tuned quantum dots to capture 940 nm and 1400 nm light, the latter rivaling InGaAs sensors. However, InGaAs imaging devices are challenging and costly to make. Using a conventional 300 mm silicon wafer process in existing fabs, ST can produce the quantum dot sensor for shortwave infrared at a fraction of the cost.

Images taken with our 940nm NIR QF sensor (top left) and with our 1400nm SWIR QF sensor (bottom left). Corresponding images taken using a visible smartphone camera (right). QF NIR image shows dramatically better contrast between black electrical wires hidden in the dark green leaves and tree trunks and branches hidden in front of the dark wood fence whereas the SWIR QF image shows how effective it is to use SWIR imaging to see through a Silicon wafer.

Quantum dots are not new, and scientists were already studying their properties in the early 1980s. However, it took years to colloidally synthesize crystals that could absorb infrared light and to create thin-film devices and fabrication processes that would yield the performance and stability necessary for real-world applications. More specifically, ST developed a manufacturing process that would not exceed 150ºC while also creating lithographic methodologies that would preserve the quantum dot’s integrity.

The IEDM 2021 paper explains in detail how ST created this image sensor technology on 300 mm wafers and describes the performance and reliability achieved to date. We plan to provide samples and evaluation kits to potential customers in 2022 and move to mass production in the coming years. Among the potential applications, mobile devices could use the new sensors to improve facial or object detection. Ultimately, a low-cost high-volume SWIR image sensor accessible to all consumers will open the door to new use-cases and applications."

Metasurface Photodetectors Review

MDPI publishes a review paper "Metasurface Photodetectors" by Jinzhao Li, Junyu Li, Shudao Zhou, and Fei Yi from Huazhong University of Science and Technology, National University of Defense Technology, and Raytron (China).

"Typical photodetectors only convert the intensity of light electrical output signals, leaving other electromagnetic parameters, such as the frequencies, phases, and polarization states unresolved. Metasurfaces are arrays of subwavelength structures that can manipulate the amplitude, phase, frequency, and polarization state of light. When combined with photodetectors, metasurfaces can enhance the light-matter interaction at the pixel level and also enable the detector pixels to resolve more electromagnetic parameters. In this paper, we review recent research efforts in merging metasurfaces with photodetectors towards improved detection performances and advanced detection schemes. The impacts of merging metasurfaces with photodetectors, on the architecture of optical systems, and potential applications are also discussed."

Gpixel Announces 9MP 1.1” NIR-enhanced Sensor for Intelligent Traffic Systems

Gpixel announces GMAX3809 extending the GMAX family into ITS applications next to its traditional industrial inspection segment. Gpixel optimized the GMAX product architecture with dedicated ITS features such as enhanced NIR response, pixel size of 3.8 μm, LED flicker mitigation and multiple region HDR modes. GMAX3809 is first in a series of GMAX products with optimized product features for ITS applications.

GMAX3809 fits 4096(H) x 2160(V) (9 MP) resolution into a 1.1” format with low noise, charge domain Global Shutter pixels running at 65 fps at 12-bit ADC resolution. GMAX3809’s 3.8 μm pixel achieves a FWC of 11.5 Ke- and noise of 3.6 e- which delivers more than 70 dB linear DR. The peak QE is 60%, a Parasitic Light Sensitivity is -92 dB, and angular response at > 15° is 80%.

GMAX3809 comes standard in a NIR-enhanced version using Gpixel’s Red Fox technology, offering the ultimate balance between NIR sensitivity and MTF. GMAX3809 achieves a QE of more than 30% at 850 nm and 14% at 940 nm.

GMAX3809 delivers 65 fps with 8 pairs of sub-LVDS channels each running at 960 Mbps resulting in a maximum data rate of 7.68 Gbps. On-chip functions, such as on-chip color offset calibration, channel multiplexing, multiple region HDR and LED flicker are available and programmable through SPI or I2C interface.

GMAX3809 is housed in a 163-pin ceramic LGA package with outer dimensions of 27.1 mm x 17.9 mm. The sensor assembly includes a double side AR coated cover glass lid.

GMAX3809 engineering samples can be ordered today for delivery in January, 2022.

BYD QVGA Sensor Won "China Chip" Excellent Market Performance Award

Sina Technology: BYD Semiconductor's 1/15-inch 80,000-pixel CMOS sensor BF30A2 won the 2021 "China Chip" product award for outstanding market performance in the field of home appliances.

The sensor has 2.5um 4T pixel and a maximum frame rate of 30fps. Its embedded ISP controls WB, AE, black level, filters out noise and false colors, removes dead pixels and dead pixel clusters, and  a skin color detection for more realistic skin color processing.

Since its launch in April 2020, BF30A2 has taken the lead in sales volume and applied for nearly 60 patents. It's said to have a market share of 85% in the wearable market.

Image Sensors at 2022 Photonics Spectra Conference

Photonics Spectra magazine holds is 2022 conference on January 10-13. Registration is free and available here. There are several image sensor presentations at this virtual event:

• KEYNOTE: Quanta Image Sensors: Every Photon Counts, Even in a Smartphone
  Eric Fossum from Dartmouth College talks about the quantum image sensor concept and how it has been implemented in CMOS image sensors and SPADs and what the major differences are between culminating results.
• Emerging Short-Wavelength Infrared Sensors
  Matthew Dyson from IDTechEx Ltd. examines the motivation and applications for SWIR image sensing, and assesses the opportunities, challenges, and adoption roadmap for emerging technical approaches.
• LEDs: Expanding Capabilities for Live Cell Imaging
  Isabel Goodhand from CoolLED explains how innovations such as multi-wavelength switching and TTL triggering enable faster imaging, and how multi-band filters can balance speed and contrast requirements.
• Advanced Detector Solutions Enabling Quantum Optics Research
  Colin Coates from Andor Technology presents high-performance detector solutions that are central to fundamental research on entangled photon systems and ultracold quantum gases.
• How Pixel Size and MTF Affect Modern Microscopy and See the Invisible with Microscopes
  Gerhard Holst from Excelitas PCO GmbH discusses the role of camera pixel size and MTF in the design and application of modern microscopes.
• Enabling Rapid Application Development and Deployment of Hyperspectral Imaging in a Production Environment
  William Rock from Headwall Photonics Inc. presents on the utility of hyperspectral imaging in a production environment using examples in food processing and demonstrates the expedited development cycle using novel hardware and software.
• High-Throughput Hyperspectral Imaging without Image Degradation
  Rand Swanson from Resonon Inc. examines the problem of image degradation with hyperspectral imagers and explores approaches to enhance the signal.
• New Photon-Counting Detectors Expand Frontiers in Scientific Imaging
  Jiaju Ma from Gigajot Technology Inc. explains the fundamentals of photon-counting image sensors, or quanta image sensors, beginning with the background knowledge necessary to effectively apply these devices.
• Dynamic Photodiodes: Unique Light-Sensing Technology with Tunable Sensitivity
  Serguei Okhonin, ActLight SA. Tunable sensitivity sets dynamic photodiode apart from all existing photodiodes, including SPADs. The AI in dynamic photodiode technology is able to dynamically adjust sensitivity at the pixel level to adapt to changing light conditions and keep the high precision of depth measurements. This presentation elaborates on the concept and design of these emerging photodiodes and how they are set to impact today’s sensing applications.
• Current and Future Detector Designs for Flash Lidar
  Jennifer Ruskowski, Fraunhofer IMS. The roadmap for creating lidar sensors for autonomous cars and robots is moving into a new era. Becoming ever more important are technologies such as sensor fusion and embedded AI, which are poised to enhance the performance, efficiency, and acceptance of lidar sensors. Additionally, on a hardware level, lidar components such as laser sources and detectors are becoming increasingly powerful. Jennifer Ruskowski gives a brief overview of the Fraunhofer IMS’s lidar development activities, from light detector to system design to sensor fusion and embedded AI solutions.
• FMCW and TOF Flash Automotive Lidar: Challenges and Prospects
  Slawomir Piatek, New Jersey Institute of Technology & Hamamatsu Corp. A vision of self-driving cars propels research and development for automotive lidar, vital hardware providing distance and velocity information about car surroundings. Among several lidar concepts—with some already adopted and heading toward production for automotive advanced driver-assistance systems (ADAS) and industrial markets—two newer designs have emerged with the highest potential in the future: frequency-modulated continuous wave (FMCW) lidar and time-of-flight (ToF) flash lidar. Both concepts, however, face engineering challenges impeding full adaptation. This presentation reviews operation principles of each technique and then discusses in greater detail the unique challenges each one faces. In particular, a light source with a long and stable coherence length is the primary challenge of FMCW lidar, whereas a photodetector with high photosensitivity and low noise is the challenge for ToF-flash lidar. The presentation concludes with a review of possible solutions to the aforementioned obstacles.

Recent Videos: Light Co., Harvard University, Sony

Light Co. publishes a video presenting its automotive stereo camera advantages over LiDAR featuring Guidehouse Principal Analyst Sam Abuelsamid, VP at Co-pace Continental AG Anil Rachakonda, Light CEO Dave Grannan, and Light's Chief Product Officer Prashant Velagaleti:

Politecnico di Milano publishes Harvard University's Federico Capasso lecture "Meta Optics: From Flat Lenses to Structured Light and Dark:"

Sony publishes 3 videos on its Pregius S stacked global shutter sensors (1, 2, 3):

Trieye Unveils VCSEL Powered SWIR Camera

PRNewswire:  TriEye announces "the first of its kind VCSEL powered Electro-Optic (EO) SWIR system", integrating TriEye CMOS-based sensor with VCSEL as an illumination source.

TriEye demonstrates an EO system by integrating the TriEye Raven with 1350nm SWIR VCSEL-based illumination, provided by their VCSEL partner, as such they enable the highest power density - which today is over 5 watts per mm2 . This new EO system will provide significant value for short-range applications such as mobile, biometrics, industrial automation, medical and more.

TriEye's solution is said to be the first to provide SWIR based sensing using VCSEL technology. TriEye's SWIR system opens doors to next generation perception capabilities by providing a significant value proposition compared to the NIR spectrum. This includes resilience to sunlight and other sources of ambient noises while providing an eye-safe illumination source. With this combination, the perception system will have longer range and better accuracy than previously achievable with NIR based systems.

NTT Demos 0.84um Color-Routing Pixel

NTT Device Technology Lab publishes an OSA Optica paper "Full-color-sorting metalenses for high-sensitivity image sensors" by Masashi Miyata, Naru Nemoto, Kota Shikama, Fumihide Kobayashi, and Toshikazu Hashimoto.

"Image sensors play a critical role in current technologies ranging from smartphones to autonomous vehicles. In these technologies, high-sensitivity image sensors are highly desired because they enable dark-scene/ultra-fast imaging. Unfortunately, a conventional sensor architecture that employs color filters on every pixel fundamentally limits the detected light power per pixel because of the filtering, which has been a long-standing barrier to sensitivity improvement. Here, we demonstrate polarization-insensitive metasurface lenses (metalenses) that sort primary colors on high-density pixels without the use of color filters. The metalenses simultaneously act as pixel-scale color splitters and lenses and are compatible with complementary metal–oxide-semiconductor sensor technology. An image sensor with such metalenses significantly enhances the detected light power, while affording high image quality, incident angle tolerance, and sub-micrometer spatial resolution. The demonstrated architecture opens the way to the development of high-sensitivity color image sensors that exceed current limits while maintaining consistency with state-of-the-art sensor technology."