Review of Quanta Image Sensors for Ultralow-Light Imaging (IEEE TED June 2022)

[As mentioned in some recent comments on this blog, the latest (vol. 69 no. 6, June 2022) issue of IEEE Transactions on Electron Devices has many interesting papers related to image sensors. I will post summaries here in the coming days.]

In an invited paper in the June 2022 issue of IEEE TED, Jiaju Ma et al. write:

The quanta image sensor (QIS) is a photon counting image sensor that has been implemented using different electron devices, including impact ionization gain devices, such as the single-photon avalanche detectors (SPADs), and low-capacitance, high conversion-gain devices, such as modified CMOS image sensors (CIS) with deep sub-electron read noise and/or low noise readout signal chains. This article primarily focuses on CIS QIS, but recent progress of both types is addressed. Signal processing progress, such as denoising, critical to improving apparent signal-to-noise ratio, is also reviewed as an enabling co-innovation.

https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9768129

Special Issue: Solid State Image Sensors on IEEE TED

My Friend Michael Guidash just informed me that the special issue for solid state image sensors on IEEE Transactions on Electron Devices in now available online.

Here is a link to the table of content: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9780655

The list of papers can be found here: https://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=16 

Surveillance market and SmartSens

From DigiTimes Asia news: https://www.digitimes.com/news/a20220527PD203/sensor-surveillance.html

China security surveillance market boom buoys SmartSens

The expanding security surveillance market in China continues to boost the shipments of CMOS image sensor (CIS) chips from Chinese CIS startup SmartSens Technology, which has entered the supply chains of China's first-tier security camera vendors including Hikvision Digital Technology, Uniview Technologies and Dahua Technology, according to industry sources.

IDC statistics show China's security surveillance market scale reached US$16.2 billion in 2021 and is estimated to grow to US$20.1 billion in 2022, for a CAGR of 13.6% for the period. High-definition security camera lenses have become the tipping point of market growth, fast driving CIS sales in China, the sources said.

Since launching its first CIS chip SC1035 in 2014, SmartSens has quickly built a strong presence in the security surveillance sector. Its CIS shipments topped 100 million in 2017 and grew all the way to 146 million in 2020, registering the highest global market share at 35% in the security CIS sector, according to Frost & Sullivan statistics.

Over the years, SmartSens has been dedicated to developing high-performance CIS chips with higher light sensitivity and signal-to-noise ratios, as well as better low-light performance as the core requirements, while deepening deployments in AI, intelligent perception and machine vision capabilities, the industry sources noted.

In terms of future security-use CIS development, its co-founder and CEO Richard Xu has said that as the surveillance lens application scenarios continue to expand, the features of low light and wide dynamic range (WDR) will be increasingly highlighted for security camera solutions so that they can penetrate higher-end applications.

Since late April this year, SmartSens has kicked off a plan to list its shares on China's Sci-Tech Innovation Board (STAR Market), aiming to raise CNY2.82 billion to finance equipment procurement and system construction for its R&D center as well as the development of car-use CIS products, the sources said.

Growth in wafer capacity for image sensors

From Semiconductor Digest news: 

Global installed capacity for image sensors was one million 200mm-equivalent wafers per month at the end of 2021. According to the new Global Wafer Capacity 2022 report, image sensor capacity is forecast to increase 13% in 2022.

By the end of 2026, installed capacity for image sensors is projected to be 1.8 million 200mm-equiv. wafers per month. That’s an average annual growth rate of 12.5% over the forecast period, making image sensor capacity among the fastest growing segments.

While the Covid-19 pandemic negatively impacted the image sensor market in 2020, growth returned in 2021. Demand for digital imaging is increasing in virtually all areas, including cellphones, automotive, machine vision, security cameras, webcams, drones, and more.

While Sony in Japan is the industry’s leading CMOS image sensor supplier, the combined CIS capacity of Samsung and SK Hynix made Korea the industry’s biggest source for production of image sensors at the end of 2021.

More than a decade ago, Sony set a goal to become the largest supplier of image sensors for cellphones. After claiming the top spot, Sony in 2014 took aim at becoming the largest supplier of CMOS image sensors for automotive systems and it is pursuing machine vision applications in factory automation and drones as well as image-recognition security cameras. Sony also sells 3D imaging sensors for depth ranging, face recognition, artificial intelligence, and machine vision.

Sony was the first to manufacture image sensors on 300mm wafers. The company has continued expanding its CIS capacity by converting 300mm fabs from logic to image sensor production and by acquiring 300mm fabs from other companies in Japan looking to exit the business of fabricating ICs. Sony has eight 300mm fab lines at four sites in Japan, with the newest being Fab 5 in Nagasaki. Fab 5 started mass production in 2021 and the construction of an expansion is already underway.

Samsung entered the CMOS image sensor business to diversify its business beyond DRAM and NAND flash. Since the fabrication technologies and tool sets for CIS devices are like that of DRAM, Samsung repurposed older DRAM fabs to begin making image sensors. The company became the industry’s second largest supplier of image sensors by serving most of the camera module needs of its huge cellphone business. Samsung’s image sensor production exists primarily at a large 300mm fab facility in Hawseong, South Korea.

SK Hynix has used the same strategy of turning older DRAM fabs into capacity for CMOS image sensors but lacks the benefit of having another related SK Hynix operation to buy its CIS devices. The company has a small but growing share of the global image sensor market.

The industry’s third largest supplier of image sensors is OmniVision but it relies on external foundries for the fabrication of its CIS wafers. OmniVision’s primary sources of foundry capacity are TSMC in Taiwan and SMIC and HLMC in China.

https://www.semiconductor-digest.com/strong-growth-in-wafer-capacity-for-image-sensors-expected/

Review article on photonics + deep learning

A team from UCLA has published a review article titled "At the intersection of optics and deep learning: statistical inference, computing, and inverse design" in Optica Advances in Optics and Photonics:

Deep learning has been revolutionizing information processing in many fields of science and engineering owing to the massively growing amounts of data and the advances in deep neural network architectures. As these neural networks are expanding their capabilities toward achieving state-of-the-art solutions for demanding statistical inference tasks in various applications, there appears to be a global need for low-power, scalable, and fast computing hardware beyond what existing electronic systems can offer. Optical computing might potentially address some of these needs with its inherent parallelism, power efficiency, and high speed. Recent advances in optical materials, fabrication, and optimization techniques have significantly enriched the design capabilities in optics and photonics, leading to various successful demonstrations of guided-wave and free-space computing hardware for accelerating machine learning tasks using light. In addition to statistical inference and computing, deep learning has also fundamentally affected the field of inverse optical/photonic design. The approximation power of deep neural networks has been utilized to develop optics/photonics systems with unique capabilities, all the way from nanoantenna design to end-to-end optimization of computational imaging and sensing systems. In this review, we attempt to provide a broad overview of the current state of this emerging symbiotic relationship between deep learning and optics/photonics.

https://opg.optica.org/aop/abstract.cfm?uri=aop-14-2-209

Free (author copy) pdf: https://innovate.ee.ucla.edu/wp-content/uploads/2022/05/aop-14-2-209-compressed.pdf

SmartSens Goes Public on Shanghai Stock Exchange

From PRNewsWire:

https://www.prnewswire.com/news-releases/smartsens-goes-public-on-shanghai-stock-exchange-and-sees-shares-surge-on-the-first-trading-day-301552661.html

SHANGHAI, May 23, 2022 /PRNewswire/ -- On May 20, 2022, SmartSens was officially listed on the Science and Technology Innovation Board of the Shanghai Stock Exchange (Stock Code: 688213). On the first day of trading, SmartSens shares surged by 79.82%, with a total market value of 22.66 billion yuan.

SmartSens Technology (Shanghai) Co., Ltd. (Stock Code: 688213) is a high-performance CMOS image sensor (CIS) chip design company. It is headquartered in Shanghai and has research centers in many cities around the world.

SmartSens has been dedicated to pushing forward the frontier of imaging technology and growing in popularity among customers since it was established. SmartSens' CMOS image sensors provide advanced imaging solutions for a broad range of areas such as surveillance, machine vision, automotive and cellphone cameras.

SmartSens is committed to continuous innovation of products and fueling growth in numerous industries by delivering a more comprehensive portfolio of image sensors.

Preprint on unconventional cameras for automotive applications

From arXiv.org --- You Li et al. write:

Autonomous vehicles rely on perception systems to understand their surroundings for further navigation missions. Cameras are essential for perception systems due to the advantages of object detection and recognition provided by modern computer vision algorithms, comparing to other sensors, such as LiDARs and radars. However, limited by its inherent imaging principle, a standard RGB camera may perform poorly in a variety of adverse scenarios, including but not limited to: low illumination, high contrast, bad weather such as fog/rain/snow, etc. Meanwhile, estimating the 3D information from the 2D image detection is generally more difficult when compared to LiDARs or radars. Several new sensing technologies have emerged in recent years to address the limitations of conventional RGB cameras. In this paper, we review the principles of four novel image sensors: infrared cameras, range-gated cameras, polarization cameras, and event cameras. Their comparative advantages, existing or potential applications, and corresponding data processing algorithms are all presented in a systematic manner. We expect that this study will assist practitioners in the autonomous driving society with new perspectives and insights.

https://arxiv.org/abs/2205.09383

Single Photon Workshop 2022 - Call for Papers

After a short hiatus, Single Photon Workshop will be held in person from October 31 to November 4, 2022 at the Korea Institute of Science and Technology (KIST) in Seoul, South Korea.

SPW 2022 is the tenth and latest installment in a series of biennial workshops on single-photon technologies and applications. After one-year delay due to COVID-19, SPW 2022 is intended to bring together again a broad range of people with interests in single-photon sources, single-photon detectors, photonic quantum metrology, and their applications such as quantum information processing. Researchers from universities, industry, and government will present their latest developments in single-photon devices and methods with a view toward improved performance and new application areas. It will be an exciting opportunity for those interested in single-photon technologies to learn about the state-of-the-art and to foster continuing partnerships with others seeking to advance the capabilities of such technologies.

1-page abstract submissions are now being accepted on various topics (see table below). Submission period is from Apr 25 - July 1, 2022.

The KIST campus is located in the northeast side of Seoul and can be easily reached by subway from anywhere in Seoul and near Seoul. The presentation room would be Johnson Auditorium in Building A1 with a seating capacity of 420 people. For poster presentation and exhibition, an open space in the 1st floor in L3 would be reserved.

Visit http://spw2022.org/ for more information.

PreAct and Espros working on new lidar solutions

From optics.org news:

PreAct Technologies, an Oregon-based developer of near-field flash lidar technology and Espros Photonics, Sargans, Switzerland, a firm producing time-of flight chips and 3D cameras, have announced a collaboration agreement to develop new flash lidar technologies for specific use cases in automotive, trucking, industrial automation and robotics.

The collaboration combines the dynamic abilities of PreAct’s software-definable flash lidar and the “ultra-ambient-light-robust time-of-flight technology” from Espros with the aim of creating what the partners call “next-generation near-field sensing solutions”.

Paul Drysch, CEO and co-founder of PreAct Technologies, commented, “Our goal is to provide high performance, software-definable sensors to meet the needs of customers across various industries. Looking to the future, vehicles across all industries will be software-defined, and our flash lidar solutions are built to support that infrastructure from the beginning.”


The automotive and trucking industries continue to rapidly integrate ADAS and self-driving capabilities into vehicles, and as the US NHTSA has just announced the requirement for human controls in fully automated vehicles, the need for ultra-precise, high performance sensors is paramount to ensuring safe autonomous driving.

The sensors created by PreAct and Espros are expected to address significant ADAS and self-driving features – such as traffic sign recognition, curb detection, night vision and pedestrian detection – with the highest frame rates and resolution of any sensor on the market, the partners state.


In addition to providing solutions for automotive and trucking, the partnership will also address the expanding robotics industry. According to a market report published by Allied Market Research, the global industrial robotics market size is expected to reach $116.8 billion by 2030.

The flash lidar technologies solutions will also enable a wide range of robotics and automation applications including QR code scanning, obstacle avoidance and gesture recognition.

Beat DeCoi, President and CEO of Espros, commented, “We have plans to demonstrate the capabilities of our 3D chipsets with PreAct’s hardware and software. By combining our best in class TOF chips with PreAct’s innovation and drive, we will see great results with clients benefiting from this partnership.”

Link: https://optics.org/news/13/3/41

Veoneer and BMW agreement on next-gen automotive vision systems

Veoneer to supply Cameras to BMW Group's Next Generation Vision System for Automated Driving

Stockholm, Sweden, May 18, 2022: The automotive technology company Veoneer has signed an agreement to equip BMW Group vehicles with camera heads for their next generation vision system for Automated Driving. The camera heads support the cooperation between BMW Group, Qualcomm Technologies, and Arriver™ which was announced in March this year.

The high definition 8 MP camera, mounted behind the rear-view mirror, monitors the forward path of the vehicle to provide reliable and accurate information to the vehicle control system.  

In BMW's next generation of Automated Driving Systems, BMW Group's current AD stack is combined with Arriver's Vision Perception and NCAP Drive Policy products on Qualcomm Technologies' system-on-chip, with the goal of designing best-in-class Automated Driving functions spanning NCAP, Level 2 and Level 3. Veoneer's camera heads are adapted to the current trend of a centralized and scalable software architecture and will be an essential part of the sensor set-up required for the next generation AD platform.

"We are excited to be part of the development of the next generation vision systems, planned to enter the market in 2025", says Jacob Svanberg, CEO of Veoneer. "This award with camera heads to the 5th generation vision system is another proof point that Veoneer remains at the forefront of providing safe, collaborative driving solutions."

Veoneer is an automotive technology company. As a world leader in active safety and restraint control systems, Veoneer is focused on delivering innovative, best-in-class products and solutions. Our purpose is to create trust in mobility. Veoneer is a Tier-1 hardware supplier and system integrator with products being part of more than 125 scheduled vehicle launches for 2022.  Headquartered in Stockholm, Sweden, Veoneer has 6,100 employees in 11 countries. The Company is building on a heritage of close to 70 years of automotive safety development. 

https://www.veoneer.com/en/press-releases?page=/press/perma/2022030