Assorted News: Always-On Sensors, Moon Landing LiDAR

Dongguk University, Seoul, Korea, publishes a MDPI paper "Design of an Always-On Image Sensor Using an Analog Lightweight Convolutional Neural Network" by Jaihyuk Choi, Sungjae Lee, Youngdoo Son, and Soo Youn Kim.

"This paper presents an always-on Complementary Metal Oxide Semiconductor (CMOS) image sensor (CIS) using an analog convolutional neural network for image classification in mobile applications. To reduce the power consumption as well as the overall processing time, we propose analog convolution circuits for computing convolution, max-pooling, and correlated double sampling operations without operational transconductance amplifiers. In addition, we used the voltage-mode MAX circuit for max pooling in the analog domain. After the analog convolution processing, the image data were reduced by 99.58% and were converted to digital with a 4-bit single-slope analog-to-digital converter. After the conversion, images were classified by the fully connected processor, which is traditionally performed in the digital domain. The measurement results show that we achieved an 89.33% image classification accuracy. The prototype CIS was fabricated in a 0.11 μm 1-poly 4-metal CIS process with a standard 4T-active pixel sensor. The image resolution was 160 × 120, and the total power consumption of the proposed CIS was 1.12 mW with a 3.3 V supply voltage and a maximum frame rate of 120."

Pixart QVGA PAJ6100U6 sensor is also aimed to always-on devices and consumes just 1.4mW at 30fps:

IEICE Electronics Express publishes Hamamatsu and Japan Aerospace Exploration Agency paper "Geiger-mode Three-dimensional Image Sensor for Eye-safe Flash LIDAR" by Takahide Mizuno, Hirokazu Ikeda, Kenji Makino, Yusei Tamura, Yoshihito Suzuki, Takashi Baba, Shunsuke Adachi, Tatsuya Hashi, Makoto Mita, Yuya Mimasu, and Takeshi Hoshino.

"Explorers attempting to land on a lunar or planetary surface must use three-dimensional image sensors to measure landing site topography for obstacle avoidance. Requirements for such sensors are similar to those mounted on vehicles and include the need for time synchronization within one frame. We introduce a 1K (32 × 32)-pixel three-dimensional image sensor using an array of InGaAs Geiger-mode avalanche photodiodes capable of photon counting in eye-safe bands and present evaluation results for sensitivity and resolution."

Kingpak Patents Acquired and Turned Against Other Companies

MaxVal reports that KT Imaging USA (KT) filed willful patent infringement complaints against Samsung Electronics, LG Electronics, Dynabook, HP, ACER and ASUSTeK in the Eastern and Western Texas District Courts. The image sensor packaging patents mentioned in the lawsuit are: US6,590,269; US6,876,544; US7,196,322; US7,511,261; US8,004,602; and US8,314,481.

KT acquired these patents from Kingpak in December of 2018. A year later, Kingpak has merged with Tong Hsing and now continues its business under Tong Hsing name.

In 2019, KT Imaging also sued Kyocera, Lightcomm Technology, and Panasonic over the same patents. The Kyocera and Panasonic lawsuits were terminated, possibly as a result of settlements, while the Lightcomm case is still pending.

MaxVal posts its summary of the patents-in-the-suits:

FLIR on SLS Sensor Advantages

FLIR publishes a recording of its webinar "The Advantages of SLS Cameras for R&D Applications."

"FLIR's new Type II Strained Layer Superlattice (SLS) opens up new applications and brings significant advances in thermal imaging.

Thermal imaging cameras operating in the traditional mid-wavelength IR (MWIR) tend to dominate the R&D application field due to their high sensitivity, high speed and relatively low cost compared to the cooled long-wavelength IR (LWIR) alternatives typically only accessible to military R&D professionals but the introduction of FLIRs new Type ll Strained Layer Superlattice is set to shake things up."

DTI and Pyramids in 0.9um Pixel Design

Taiwan National Cheng Kung University publishes a MDPI paper "Deep Trench Isolation and Inverted Pyramid Array Structures Used to Enhance Optical Efficiency of Photodiode in CMOS Image Sensor via Simulations" by Chang-Fu Han, Jiun-Ming Chiou, and Jen-Fin Lin. DTI and pyramids are the key elements of the modern IR-enhanced sensors from Sony, Omnivision, SmartSens, and other companies.

"The photodiode in the backside-illuminated CMOS sensor is modeled to analyze the optical performances in a range of wavelengths (300–1100 nm). The effects of changing in the deep trench isolation depth (DTI) and pitch size (d) of the inverted pyramid array (IPA) on the peak value (OEmax.) of optical efficiency (OE) and its wavelength region are identified first. Then, the growth ratio (GR) is defined for the OE change in these wavelength ranges to highlight the effectiveness of various DTI and d combinations on the OEs and evaluate the OE difference between the pixel arrays with and without the DTI + IPA structures. Increasing DTI can bring in monotonous OEmax. increases in the entire wavelength region. For a fixed DTI, the maximum OEmax. is formed as the flat plane (d = 0 nm) is chosen for the top surface of Si photodiode in the RGB pixels operating at the visible light wavelengths; whereas different nonzero value is needed to obtain the maximum OEmax. for the RGB pixels operating in the near-infrared (NIR) region. The optimum choice in d for each color pixel and DTI depth can elevate the maximum GR value in the NIR region up to 82.2%."

ActLight Signed Contract with "Leading Sensor Company"

PRNewswire: ActLight announces that it has signed a service agreement based on its Single Photon Sensitivity technology with a leading company in the sensors market.

"Even though the terms of the agreement cannot be disclosed, we are very pleased that our innovative Single Photon Sensitivity technology attracted a leading player in the sensors field," said Maxim Gureev, CTO at ActLight. "The adoption of Single Photon Avalanche Diode (SPAD) array in 3D sensing chips is growing fast. The precision of 3D sensing in applications such as smartphones, cars and smart robotics will benefit from this collaboration with our customer and our talented team of engineers is already intensively working to make it happen."

Not Only Sony: Attollo Introduces SWIR Sensor with 5um Pixel Pitch

Attollo Engineering introduces the Phoenix, a 640 x 512 SWIR camera based on its claimed to be the industry’s smallest VGA sensor with 5 µm InGaAs pixels.

"The Attollo Phoenix SWIR camera is a VGA format (640x512), uncooled SWIR camera featuring the industry’s smallest SWIR VGA sensor - 5um pixel size. The Phoenix captures snapshot SWIR imagery using Attollo Engineering’s high‑performance InGaAs detector material and the extremely small pixel pitch enables more pixels on target with a short focal length optic. The Phoenix’s sensor is designed specifically to support broadband imaging along with day and night laser see‑spot and range-gated imaging capabilities.

The high-performance, InGaAs 640 x 512, 5 µm pixel pitch SWIR camera’s spectral response ranges from 1.0 µm to 1.65 µm with more than 99.5% operability and greater than 70% quantum efficiency. Selectable frame rates include 30 Hz, 60 Hz, 120 Hz, and 220 Hz, with windowing available. The Phoenix has a global shutter imaging mode and presets and user-defined integration time of 0.1µs (minimum), plus triggering options of sync-in (low-latency see-spot and range-gating) and sync-out. Other specifications include onboard processing with non-uniformity corrections (NUCs) and bad pixel replacement."

Cars and Smartphones Drive CCM Market

RsesearchInChina report "Global and China CMOS Camera Module (CCM) Industry Report, 2020-2026" forecasts:

"The global CCM market has been ballooning thanks to expeditious penetration of multi-camera phones and advances in automotive ADAS, being worth $22.723 billion with a year-on-year spike of 16.6% in 2019, a figure projected to sustain growth at a compound annual rate of 6.1% between 2019 and 2026.

Nowadays, single-camera, dual-camera and triple-camera mobile phones prevail globally, of which dual rear camera mobile phones share 40%. However, the upcoming triple-camera, four-camera and five-camera mobile phones will undoubtedly beat dual-camera ones, and triple-camera and four-camera phone models will become the mainstream alongside the burgeoning demand for mobile phone camera modules.

The global shipments of automotive camera modules reached 250 million units in 2019. The automotive camera module market is facilitated amid a faster rise in ADAS penetration due to the incentive policies and robust consumer demand. By 2026, the global automotive camera module shipments would expectedly hit 600 million units.

In the next few years, a growing number of camera modules will be mounted onto each mobile phone and every car."

Thesis on Time to Digital Converter for SPADs

Universitat Politecnica Valencia, Spain, publishes MSc Thesis "Time to Digital and Charge to Digital converters for SiPM front ends" by Alessandro Morini.

"Two tasks have been carried out in this master thesis: implementation of a single front-end channel (composed by an amplifier and a gated integrator) taking into account specification have been set in advance; a survey on a Time to Digital Converter (TDC) and Analog to Digital Converter (ADC).

The first one accomplishes firstly a preamplifier for the integrated SiPM using a 0.35 um technology. The output current will feed a TDC (boosted for fast signals) and an ADC (boosted for charge integration). During the second step a gated charge integrator has been carried out, which will be used for the analog chain needed for the ADC. It has been settled an integration start threshold and a configurable integrating window.

Regarding the second task, we focused on different configurations for TDC that could work with the given requirements. Furthermore, a Sample and Hold (S/H) and a Successive Approximation ADC (SAR) have been implemented. The SAR is composed by a quite fast comparator, a programmed logic in Verilog-A, necessary to study bit by bit, and a DAC in the end."

Samsung to Expand to CIS Production Capacity

BusinessKorea: Samsung says that DRAM production line can be easily converted into an image sensor line because their processes are 80% identical. The company is preparing a detailed plan to convert part of its production lines for DRAMs in Hwaseong, Gyeonggi Province to CIS lines. The newspaper's sources say that mass production of image sensors at the converted lines can begin within this year after new equipment is installed, tested and stabilized. They claim that Samsung will spend at least one trillion won on this conversion project, although it requires less money than investment in fresh production facilities.

In 2018, the company converted part of its DRAM line 11, which is based on 300-mm (12-inch) wafers, to image sensor production line S4.

Pixelplus May Be Delisted from KOSDAQ

TheElec: South Korean PixelPlus faces a possibility of delisting from KOSDAQ stock exchange due to the four straight years of losses.

Automotive image sensors accounted for 70% to 80% of the company’s sales. 80% of its sales are in China. However, coronavirus pamndemic affected the sales and made the future forecast uncertain.

PixelPlus was founded in 2000 and initially manufactured image sensors for mobile phones. In its good times, it was listed on NASDAQ in 2005-2009. However, Samsung and Sony competition caused Pixelplus delisting from NASDAQ in 2009.

Next, PixelPlus has entered security and surveillance image sensors and was listed on KOSDAQ in 2015. However, price competition with Chinese companies was tough and Pixelplus reported yearly loss every year since 2016.

Then, PixelPlus has effectively given up on the security image sensor market and tried to enter automotive applications. These plans are frozen due to coronavirus slowdown now.

Face Counter-Identification Startup Raises $13.5M

Techcrunch: Israeli startup D-ID developing slight changes in pictures that virtually kill AI facial recognition algorithms raises $13.5M in round A from AXA Ventures, Pitango, Y Combinator, AI Alliance, Hyundai, Omron, Maverick. and Mindset (via IFNews):

Post-Coronavirus "Touchless Economy" to Boost Image Sensor Market

UAR National, MoneyControl: In post-COVID-19 world, most user interfaces would be redesigned to eliminate the infections spreading:

"Few months from now, your attendance will be marked by facial recognition system or by voice. In airports, you will print your boarding pass through gestures.

Touchless technology is here to stay and will witness growth much faster than earlier due to the COVID-19 outbreak. Experts point out that touchless technology is likely to accelerate adoption across sectors.

Lift manufacturer Fujitec wants passengers to select floors using only hand signals, while sensor maker Optex plans a similar concept for opening doors. Toshiba Tec, a subsidiary of Toshiba, wants to banish fingerprint-laden restaurant menus to the past with gesture-sensing, projected menus."

Panasonic Paper on SPAD CMOS Sensor

Panasonic publishes MDPI paper "Modeling and Analysis of Capacitive Relaxation Quenching in a Single Photon Avalanche Diode (SPAD) Applied to a CMOS Image Sensor" by Akito Inoue, Toru Okino, Shinzo Koyama, and Yutaka Hirose. This paper opens a Special Issue on Photon Counting Image Sensors.

"We present an analysis of carrier dynamics of the single-photon detection process, i.e., from Geiger mode pulse generation to its quenching, in a single-photon avalanche diode (SPAD). The device is modeled by a parallel circuit of a SPAD and a capacitance representing both space charge accumulation inside the SPAD and parasitic components. The carrier dynamics inside the SPAD is described by time-dependent bipolar-coupled continuity equations (BCE). Numerical solutions of BCE show that the entire process completes within a few hundreds of picoseconds. More importantly, we find that the total amount of charges stored on the series capacitance gives rise to a voltage swing of the internal bias of SPAD twice of the excess bias voltage with respect to the breakdown voltage. This, in turn, gives a design methodology to control precisely generated charges and enables one to use SPADs as conventional photodiodes (PDs) in a four transistor pixel of a complementary metal-oxide-semiconductor (CMOS) image sensor (CIS) with short exposure time and without carrier overflow. Such operation is demonstrated by experiments with a 6 µm size 400 × 400 pixels SPAD-based CIS designed with this methodology."

ST Unveils ToF Sensor for Multi-Object Ranging

STMicro extends its FlightSense ToF sensors with the VL53L3CX device featuring histogram algorithms that allow measuring distances to multiple objects as well as increasing accuracy.

The VL53L3CX measures object ranges from 2.5cm to 3m, unaffected by the target color or reflectance, unlike conventional infrared sensors. This allows designers to introduce powerful new features to their products, such as enabling occupancy detectors to provide error-free sensing by ignoring unwanted background or foreground objects, or reporting the exact distances to multiple targets within the sensor’s field-of-view.

The ST patented histogram algorithms increase cover-glass crosstalk immunity and allow real-time smudge compensation preventing external contamination from adversely affecting the ranging accuracy of, for example, vacuum cleaners or equipment that may be used in a dusty industrial environment. Ranging under ambient lighting is also improved.

In addition, the VL53L3CX has high linearity that increases short-distance measurement accuracy enhancing wall tracking, faster cliff detection, and obstacle avoidance in equipment such as service robots and vacuum cleaners, markets in which ST has already enjoyed considerable commercial success. Like all FlightSense sensors, the VL53L3CX features a compact, all-in-one package design that eases integration in customer devices, as well as low power consumption that helps extend battery runtime.

The VL53L3CX is available now, priced from $1.70.

Adafruit introduces the new ST sensor:

ADAS Cameras Overview

Amkor, a packaging company, publishes "A Look Inside ADAS Modules" on various camera configurations found in different cars:

Online Training on Color Pipeline of a Camera

Framos announces an "Online Training: Colour Pipeline of a Camera" by be delivered by Albert Thuwissen on July 6-7, 2020.

The training will start with a short overview of the sensor and the lens, and will then dive into the details of a “standard” colour pipeline that is used to make a colour image out of the raw sensor signal. The following topics will be discussed:

• Auto White Balancing: The human eye is adapting easily and quickly to the spectrum of a light source, the image sensors do not adapt at all!
• Lens-Vignetting: Lenses have a strong fall-off of intensity and sharpness towards the edges. On top of that, also the image sensor will add an extra fall-off of intensity. Is correction possible?
• Colour Matrixing: Nobody is perfect, neither are the imagers that suffer from optical cross-talk and from imperfections when it comes to the transmission characteristics of the colour filters. Colour matrixing takes care about these issues. Question is how to find to optimum correction matrix coefficients?
• Contouring: This is a technique to „regain“ details, edges and sharpness in an image. But quite often not only the details are enhanced, but the noise in the image as well.
• Colour Interpolation: The Bayer pattern sampling is extensively used in digital imaging, but the sampling is only half of the story. The other half is the demosaicing or interpolation. Several methods will be discussed and compared with each other.
• Dark Current Compensation: The average value of the dark current can be corrected by the use of dark reference lines/pixels. Fixed-pattern noise can be corrected by means of dark frame subtraction. How efficient are these techniques? What is their influence on signal-to-noise performance and what about temperature effects?
• Noise Filtering: A very important issue in data processing is the filtering of any remaining noise. This can be done in a non-adaptive or an adaptive way. What are the pros and cons of the various techniques?
• Defect Correction: How can defect pixels be corrected without any visible effect? Can similar techniques also be applied to correct defect columns?

Although not really part of the colour pipeline, the following aspects of a digital camera will be discussed in the training as well:

• Auto-exposure: How can the data of the image sensor itself being used to optimize the exposure time of the imager?
• Auto-focusing: How can the data of the image sensor itself being used to activate the auto-focusing function?

Facial Recognition Adoption Around the Globe

VisualCapitalist publishes a summary of facial recognition approved in different countries:

• In the US, 59% of Americans are in favor of implementing facial recognition technology for use in law enforcement, according to a Pew Research survey.
• The US Department of Homeland Security plans to conduct facial recognition of 97% of all air travellelrs by 2023
• In South America, Facial Recognition is used by 92% of the countries
• 80% of Europeans are not keen on sharing facial data with authorities

HDR Pixels Review and Comparison

Dana Diezemann published her presentation "High Dynamic Range Imaging, A short summary" at Image Sensors Europe held in London in March 2020. Few slides from the presentation: