Color sensing for nano-vision-sensors

Ningxin Li et al. from Georgia State University have published a new article titled "Van der Waals Semiconductor Empowered Vertical Color Sensor" in ACS Nano.

Abstract

Biomimetic artificial vision is receiving significant attention nowadays, particularly for the development of neuromorphic electronic devices, artificial intelligence, and microrobotics. Nevertheless, color recognition, the most critical vision function, is missed in the current research due to the difficulty of downscaling of the prevailing color sensing devices. Conventional color sensors typically adopt a lateral color sensing channel layout and consume a large amount of physical space, whereas compact designs suffer from an unsatisfactory color detection accuracy. In this work, we report a van der Waals semiconductor-empowered vertical color sensing structure with the emphasis on compact device profile and precise color recognition capability. More attractive, we endow color sensor hardware with the function of chromatic aberration correction, which can simplify the design of an optical lens system and, in turn, further downscales the artificial vision systems. Also, the dimension of a multiple pixel prototype device in our study confirms the scalability and practical potentials of our developed device architecture toward the above applications.

Article: https://pubs.acs.org/doi/10.1021/acsnano.1c09875

“This work is the first step toward our final destination­–to develop a micro-scale camera for microrobots,” says assistant professor of Physics Sidong Lei, who led the research. “We illustrate the fundamental principle and feasibility to construct this new type of image sensor with emphasis on miniaturization.”

Press: https://scitechdaily.com/new-electric-eye-neuromorphic-artificial-vision-device-developed-using-nanotechnology/

Samsung making a new larger ISOCELL camera sensor?

Samsung is the world’s second-biggest mobile camera sensor maker, and its sensors are used by almost every smartphone brand. Over the past couple of years, the South Korean firm has launched various big-sized camera sensors, including the ISOCELL GN1 and the ISOCELL GN2. This year, it has made one more giant ISOCELL camera sensor.

The company has developed the ISOCELL GNV camera sensor, and it will be used in a Vivo smartphone. It is being reported that the ISOCELL GNV is custom-made for Vivo phones, and it has a size of 1/1.3-inch. It is most likely a 50MP sensor, similar to the ISOCELL GN1, ISOCELL GN2, and the ISOCELL GN5. It will act as the Vivo X80 Pro+’s primary camera and features a gimbal-like OIS system.

The ISOCELL GNV is likely a slightly modified version of Samsung’s ISOCELL GN1. The Vivo smartphone has three other cameras, including a 48MP/50MP ultrawide camera (Sony IMX sensor), a 12MP telephoto camera with 2x optical zoom and OIS, and an 8MP telephoto camera with 5x optical zoom and OIS. The phone can record 8K videos using the primary camera and up to 4K 60fps videos using the rest of its cameras. On the front, it could have a 44MP selfie camera.

The phone also uses Vivo’s custom ISP (Image Signal Processor) named V1+, which has been made in close collaboration with MediaTek. The new chip brings 16% higher brightness and 12% better white balance to images in low-light conditions. Prominent sections of an image can see up to 350% better brightness for lower noise and better colors.

The rest of the phone’s specifications include a 6.78-inch 120Hz Super AMOLED LTPO display, Snapdragon 8 Gen 1 processor, 8GB/12GB RAM, 128GB/256GB storage, 4,700mAH battery, 80W fast wired charging, 50W fast wireless charging, stereo speakers, and an IP68 rating for dust and water resistance.

https://www.sammobile.com/news/samsung-isocell-gnv-camera-sensor-coming/

90-min Tutorial on Single Photon Detectors

Krister Shalm of National Institute of Standards and Technologies presented a tutorial: Single-photon detectors at the 2013 QCrypt Conference in August. http://2013.qcrypt.net

This is from a while back but an excellent educational resource nevertheless!

The video is roughly 90-minutes long but has several gaps that can be skipped ahead. Or play it at >1x speed!

Full video here: https://www.youtube.com/watch?v=Vt84rSJa7VI&t=541s

Embedded Vision Summit 2022

The Edge AI and Vision Alliance, a 118-company worldwide industry partnership is organizing the 2022 Embedded Vision Summit, May 16-19 at the Santa Clara Convention Center, Santa Clara, California.

The premier conference and tradeshow for practical, deployable computer vision and edge AI, the Summit focuses on empowering product creators to bring perceptual intelligence to products. This year’s Summit will attract more than 1,000 innovators and feature 90+ expert speakers and 60+ exhibitors across four days of presentations, exhibits and deep-dive sessions. Registration is now open.

Highlights of this year’s program include:

• Keynote speaker Prof. Ryad Benosman of University of Pittsburgh and the CMU Robotics Institute will speak on “Event-based Neuromorphic Perception and Computation: The Future of Sensing and AI”
• General session speakers include:
• Zach Shelby, co-founder and CEO of Edge Impulse, speaking on “How Do We Enable Edge ML Everywhere? Data, Reliability, and Silicon Flexibility”
• Ziad Asghar, Vice President of Product Management at Qualcomm, speaking on “Powering the Connected Intelligent Edge and the Future of On-Device AI”
• 90+ sessions across four tracks—Fundamentals, Technical Insights, Business Insights, and Enabling Technologies
• 60+ exhibitors including Premier Sponsors Edge Impulse and Qualcomm, Platinum Sponsors FlexLogix and Intel, and Gold Sponsors Arm, Arrow, Avnet, BDTi, City of Oulu, Cadence, Hailo, Lattice, Luxonis, Network Optics, Nota, Perceive, STMicroelectronics, Synaptics and AMD Xilinx
• Deep Dive Sessions — offering opportunities to explore cutting-edge topics in-depth — presented by Edge Impulse, Qualcomm, Intel, and Synopsys
• “We are delighted to return to being in-person for the Embedded Vision Summit after two years of online Summits,” said Jeff Bier, founder of the Edge AI and Vision Alliance. “Innovation in visual and edge AI continues at an astonishing pace, so it’s more important than ever to be able to see, in one place, the myriad of practical applications, use cases and building-block technologies. Attendees with diverse technical and business backgrounds tell us this is the one event where they get a complete picture and can rapidly sort out the hype from what’s working. A whopping 98% of attendees would recommend attending to a colleague.”

Registration is now open at https://embeddedvisionsummit.com.

The Embedded Vision Summit is operated by the Edge AI and Vision Alliance, a worldwide industry partnership bringing together technology providers and end-product companies to accelerate the adoption of edge AI and vision in products. More at https://edge-ai-vision.com.

EETimes Article

EETimes has published a "teaser" article written by the general chair of this year's summit.

Half a billion years ago something remarkable occurred: an astonishing, sudden increase in new species of organisms. Paleontologists call it the Cambrian Explosion, and many of the animals on the planet today trace their lineage back to this event.

A similar thing is happening in processors for embedded vision and artificial intelligence (AI) today, and nowhere will that be more evident than at the Embedded Vision Summit, which will be an in–person event held in Santa Clara, California, from May 16–19. The Summit focuses on practical know–how for product creators incorporating AI and vision in their products. These products demand AI processors that balance conflicting needs for high performance, low power, and cost sensitivity. The staggering number of embedded AI chips that will be on display at the Summit underscores the industry’s response to this demand. While the sheer number of processors targeting computer vision and ML is overwhelming, there are some natural groupings that make the field easier to comprehend. Here are some themes we’re seeing. 

Founded in 2011, the Edge AI and Vision Alliance is a worldwide industry partnership that brings together technology providers who are enabling innovative and practical applications for edge AI and computer vision. Its 100+ Member companies include suppliers of processors, sensors, software and services.

First, some processor suppliers are thinking about how to best serve applications that simultaneously apply machine learning (ML) to data from diverse sensor types — for example, audio and video. Synaptics’ Katana low–power processor, for example, fuses inputs from a variety of sensors, including vision, sound, and environmental. Xperi’s talk on smart toys for the future touches on this, as well.

Second, a subset of processor suppliers are focused on driving power and cost down to a minimum. This is interesting because it enables new applications. For example, Cadence will be presenting on additions to their Tensilica processor portfolio that enable always–on AI applications. Arm will be presenting low–power vision and ML use cases based on their Cortex–M series of processors. And Qualcomm will be covering tools for creating low–power computer vision apps on their Snapdragon family.

Third, although many processor suppliers are focused mainly or exclusively on ML, a few are addressing other kinds of algorithms typically used in conjunction with deep neural networks, such as classical computer vision and image processing.  A great example is quadric, whose new q16 processor is claimed to excel at a wide range of algorithms, including both ML and conventional computer vision.

Finally, an entirely new species seems to be coming to the fore: neuromorphic processors. Neuromorphic computing refers to approaches that mimic the way the brain processes information. For example, biological vision systems process events in the field of view, as opposed to classical computer vision approaches that typically capture and process all the pixels in a scene at a fixed frame rate that has no relation to the source of the visual information. The Summit’s keynote talk, “Event–based Neuromorphic Perception and Computation: The Future of Sensing and AI” by Prof. Ryad Benosman, will give an overview of the advantages to be gained by neuromorphic approaches. Opteran will be presenting on their neuromorphic processing approach to enable vastly improved vision and autonomy, the design of which was inspired by insect brains.

Whatever your application is, and whatever your requirements are, somewhere out there is an embedded AI or vision processor that’s the best fit for you. At the Summit, you’ll be able to learn about many of them, and speak with the innovative companies developing them.  Come check them out, and be sure to check back in 10 years — when we will see how many of 2032’s AI processors trace their lineage to this modern–day Cambrian Explosion!

—Jeff Bier is the president of consulting firm BDTI, founder of the Edge AI and Vision Alliance, and the general chair of the Embedded Vision Summit.

About the Edge AI and Vision Alliance

The mission of the Alliance is to accelerate the adoption of edge AI and vision technology by:

• Inspiring and empowering product creators to incorporate AI and vision technology into new products and applications
• Helping Member companies achieve success with edge AI and vision technology by:
• Building a vibrant AI and vision ecosystem by bringing together suppliers, end-product designers, and partners
• Delivering timely insights into AI and vision market research, technology trends, standards and application requirements
• Assisting in understanding and overcoming the challenges of incorporating AI in their products and businesses

Perspective article on solar-blind UV photodetectors

A research group from the Indian Institute of Science has published a perspective article titled "The road ahead for ultrawide bandgap solar-blind UV photodetectors" in the Journal of Applied Physics.

Abstract:

This perspective seeks to understand and assess why ultrawide bandgap (UWBG) semiconductor-based deep-UV photodetectors have not yet found any noticeable presence in real-world applications despite riding on more than two decades of extensive materials and devices’ research. Keeping the discussion confined to photodetectors based on epitaxial AlGaN and Ga2O3, a broad assessment of the device performance in terms of its various parameters is done vis-à-vis the dependence on the material quality. We introduce a new comprehensive figure of merit (CFOM) to benchmark photodetectors by accounting for their three most critical performance parameters, i.e., gain, noise, and bandwidth. We infer from CFOM that purely from the point of view of device performance, AlGaN detectors do not have any serious shortcoming that is holding them back from entering the market. We try to identify the gaps that exist in the research landscape of AlGaN and Ga2O3 solar-blind photodetectors and also argue that merely improving the material/structural quality and device performance would not help in making this technology transition from the academic realm. Instead of providing a review, this Perspective asks the hard question on whether UWBG solar-blind detectors will ever find real-world applications in a noticeable way and whether these devices will be ever used in space-borne platforms for deep-space imaging, for instance.

Full text: https://aip.scitation.org/doi/full/10.1063/5.0082348

The chain of UWBG detector technology development: A general status.

State-of-art n-type (right axis) and p-type (left axis) conductivity values in epitaxial AlGaN as a function of the bandgap of the ternary alloy, as reported in the literature.

Scatter plot of the product of UV-to-visible rejection ratio and gain of various types of AlGaN solar-blind photodetectors, as published in the literature, benchmarked with a Hamamatsu commercial-grade solar-blind photomultiplier tube.

A possible blown-up schematic of deep-UV imaging assembly based on AlGaN photodetector, which can significantly cut down on weight, footprint, and complexities such as high voltage requirement.

A qualitative plot of the current status of solar-blind UV photodetectors vis-à-vis their approximate TRL levels for AlGaN, β-Ga2O3, α-Ga2O3, and ɛ-Ga2O3.

Videos du jour - CICC, PhotonicsNXT and EPIC

IEEE CICC 2022 best paper candidates present their work

Solid-State dToF LiDAR System Using an Eight-Channel Addressable, 20W/Ch Transmitter, and a 128x128 SPAD Receiver with SNR-Based Pixel Binning and Resolution Upscaling

Shenglong Zhuo, Lei Zhao,Tao Xia, Lei Wang, Shi Shi, Yifan Wu, Chang Liu, et al.

Fudan University, PhotonIC Technologies, Southern Univ. of S&T

A 93.7%-Efficiency 5-Ratio Switched-Photovoltaic DC-DC Converter

Sandeep Reddy Kukunuru,Yashar Naeimi, Loai Salem

University of California, Santa Barbara

A 23-37GHz Autonomous Two-Dimensional MIMO Receiver Array with Rapid Full-FoV Spatial Filtering for Unknown Interference Suppression

Boce Lin, Tzu-Yuan Huang,Amr Ahmed, Min-Yu Huang, Hua Wang

Georgia Institute of Technology

PhotonicsNXT Fall Summit keynote discusses automotive lidar

This keynote session by Pierrick Boulay of Yole Developpement at the PhotonicsNXT Fall Summit held on October 28, 2021 provides an overview of the lidar ecosystem and shows how lidar is being used within the auto industry for ranging and imaging.

EPIC Online Technology Meeting on Single Photon Sources and Detectors

The power hidden in one single photon is unprecedented. But we need to find ways to harness that power. This meeting will discuss cutting-edge technologies paving the way for versatile and efficient pure single-photon sources and detection schemes with low dark count rates, high saturation levels, and high detection efficiencies. This meeting will gather the key players in the photonic industry pushing the development of these technologies towards commercializing products that harness the intrinsic properties of photons.

Wide field-of-view imaging with a metalens

A research group from Nanjing University has published a new paper titled "Planar wide-angle-imaging camera enabled by metalens array" in the recent issue of Optica.

Abstract:

Wide-angle imaging is an important function in photography and projection, but it also places high demands on the design of the imaging components of a camera. To eliminate the coma caused by the focusing of large-angle incident light, traditional wide-angle camera lenses are composed of complex optical components. Here, we propose a planar camera for wide-angle imaging with a silicon nitride metalens array mounted on a CMOS image sensor. By carefully designing proper phase profiles for metalenses with intentionally introduced shifted phase terms, the whole lens array is capable of capturing a scene with a large viewing angle and negligible distortion or aberrations. After a stitching process, we obtained a large viewing angle image with a range of >120 degrees using a compact planar camera. Our device demonstrates the advantages of metalenses in flexible phase design and compact integration, and the prospects for future imaging technology.

Metalens array mounted directly on a CMOS camera

Schematic diagram of the principle and device architecture. (a) Schematics of wide-angle imaging by MIWC. Zoom-in figure shows the imaging principle with each part of the wide-angle image clearly imaged separately by each metalens. (b) Photograph of MIWC. The metalens array can be seen in the middle of the enlarged figure on the right. (c) Architecture of MIWC. The metalens array is integrated directly on the CMOS image sensor (DMM 27UJ003-ML) and fixed by an optically clear adhesive (OCA) tape (Tesa, 69402).

Experimental wide-angle imaging results by MIWC. (a) Projected “NANJING UNIVERSITY” on the curved screen covers a viewing angle of 120° and then is imaged by MIWC. (b) Imaging results and corresponding mask functions of lenses with designed angles of −57.5∘, 0°, 57.5°. (d) Imaging result of a traditional metalens showing limited field of view. (e) Final imaging result of MIWC by processing with mask functions and sub-images, which shows three times larger FOV compared with the traditional lens.

Paper [open access]: https://doi.org/10.1364/OPTICA.446063

Press release: https://phys.org/news/2022-04-miniature-wide-angle-camera-flat-metalenses.html

PhD Thesis on Analog Signal Processing for CMOS Image Sensors

The very first PhD thesis that came out of Albert Theuwissen's group at TU Delft is now freely available as a pdf. This seems like a great educational resource for people interested in image sensors.

Direct download link: https://repository.tudelft.nl/islandora/object/uuid:2fbc1f51-7784-4bcd-85ab-70fc193c5ce9/datastream/OBJ/download

Title: Analog Signal Processing for CMOS Image Sensors

Author: Martijn Snoeij

Year: 2007

Abstract: 

This thesis describes the development of low-noise power-efficient analog interface circuitry for CMOS image sensors. It focuses on improving two aspects of the interface circuitry: firstly, lowering the noise in the front-end readout circuit, and secondly the realization of more power-efficient analog-to-digital converters (ADCs) that are capable of reading out high-resolution imaging arrays. 

Chapter 2 provides an overview of the analog signal processing chain in conventional, commercially-available CMOS imagers. First of all, the different photo-sensitive elements that form the input to the analog signal chain are briefly discussed. This is followed by a discussion of the analog signal processing chain itself, which will be divided into two parts. Firstly, the analog front-end, consisting of in-pixel circuitry and column-level circuitry, is discussed. Second, the analog back-end, consisting of variable gain amplification and A/D conversion is discussed. Finally, a brief overview of advanced readout circuit techniques is provided.

In chapter 3, the performance of the analog front-end is analyzed in detail. It is shown that its noise performance is the most important parameter of the front-end. An overview of front-end noise sources is given and their relative importance is discussed. It will be shown that 1/f noise is the limiting noise source in current CMOS imagers. A relatively unknown 1/f noise reduction technique, called switched-biasing or large signal excitation (LSE), is introduced and its applicability to CMOS imagers is explored. Measurement results on this 1/f noise reduction technique are presented. Finally, at the end of the chapter, a preliminary conclusion on CMOS imager noise performance is presented. 

The main function of the back-end analog signal chain is analog-to-digital conversion, which is described in chapter 4. First of all, the conventional approach of a single chip-level ADC is compared to a massively-parallel, column-level ADC, and the advantages of the latter will be shown. Next, the existing column-level ADC architectures will be briefly discussed, in particular the column-parallel single-slope ADC. Furthermore, a new architecture, the multiple-ramp single-slope ADC will be proposed. Finally, two circuit techniques are introduced that can improve ADC performance. Firstly, it will be shown that the presence of photon shot noise in an imager can be used to significantly decrease ADC power consumption. Secondly, an column FPN reduction technique, called Dynamic Column Switching (DCS) is introduced.

Chapter 5 and 6 present two realisations of imagers with column-level ADCs. In chapter 5, a CMOS imager with single-slope ADC is presented that consumes only 3.2µW per column. The circuit details of the comparator achieving this low power consumption are described, as well as the digital column circuitry. The ADC uses the dynamic column switching technique introduced in chapter 4 to reduce the perceptional effects of column FPN. Chapter 6 presents an imager with a multiple-ramp single-slope architecture, which was proposed in chapter 4. The column comparator used in this design is taken from a commercially available CMOS imager. The multiple ramps are generated on chip with a low power ladder DAC structure. The ADC uses an auto-calibration scheme to compensate for offset and delay of the ramp drivers.

Google AI Blog article on Lidar-Camera Fusion

A team from Google Research has a new blog article on fusing Lidar and camera data for 3D object detection. The motivating problem here seems to be the issue of misalignment between 3D LiDAR data and 2D camera data.

Full article: https://ai.googleblog.com/2022/04/lidar-camera-deep-fusion-for-multi.html

The blog discusses the team's forthcoming paper titled "DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection" which will be presented at the IEEE/CVF Computer Vision and Pattern Recognition (CVPR) conference in June 2022. A preprint of the paper is available here.

Some excerpts from the blog and the associated paper:

LiDAR and visual cameras are two types of complementary sensors used for 3D object detection in autonomous vehicles and robots. To develop robust 3D object detection models, most methods need to augment and transform the data from both modalities, making the accurate alignment of the features challenging.

Existing algorithms for fusing LiDAR and camera outputs generally follow two approaches --- input-level fusion where the features are fused at an early stage, decorating points in the LiDAR point cloud with the corresponding camera features, or mid-level fusion where features are extracted from both sensors and then combined. Despite realizing the importance of effective alignment, these methods struggle to efficiently process the common scenario where features are enhanced and aggregated before fusion. This indicates that effectively fusing the signals from both sensors might not be straightforward and remains challenging.

In our CVPR 2022 paper, “DeepFusion: LiDAR-Camera Deep Fusion for Multi-Modal 3D Object Detection”, we introduce a fully end-to-end multi-modal 3D detection framework called DeepFusion that applies a simple yet effective deep-level feature fusion strategy to unify the signals from the two sensing modalities. Unlike conventional approaches that decorate raw LiDAR point clouds with manually selected camera features, our method fuses the deep camera and deep LiDAR features in an end-to-end framework. We begin by describing two novel techniques, InverseAug and LearnableAlign, that improve the quality of feature alignment and are applied to the development of DeepFusion. We then demonstrate state-of-the-art performance by DeepFusion on the Waymo Open Dataset, one of the largest datasets for automotive 3D object detection.

We evaluate DeepFusion on the Waymo Open Dataset, one of the largest 3D detection challenges for autonomous cars, using the Average Precision with Heading (APH) metric under difficulty level 2, the default metric to rank a model’s performance on the leaderboard. Among the 70 participating teams all over the world, the DeepFusion single and ensemble models achieve state-of-the-art performance in their corresponding categories.

Quantum Dot Photodiodes for SWIR Cameras

A research team from Ghent University in Belgium  has published an article titled "Colloidal III–V Quantum Dot Photodiodes for Short-Wave Infrared Photodetection".

Abstract: Short-wave infrared (SWIR) image sensors based on colloidal quantum dots (QDs) are characterized by low cost, small pixel pitch, and spectral tunability. Adoption of QD-SWIR imagers is, however, hampered by a reliance on restricted elements such as Pb and Hg. Here, QD photodiodes, the central element of a QD image sensor, made from non-restricted In(As,P) QDs that operate at wavelengths up to 1400 nm are demonstrated. Three different In(As,P) QD batches that are made using a scalable, one-size-one-batch reaction and feature a band-edge absorption at 1140, 1270, and 1400 nm are implemented. These QDs are post-processed to obtain In(As,P) nanocolloids stabilized by short-chain ligands, from which semiconducting films of n-In(As,P) are formed through spincoating. For all three sizes, sandwiching such films between p-NiO as the hole transport layer and Nb:TiO2 as the electron transport layer yields In(As,P) QD photodiodes that exhibit best internal quantum efficiencies at the QD band gap of 46±5% and are sensitive for SWIR light up to 1400 nm.

a) Normalized absorbance spectra of the three QD batches (red) measured in tetrachloroethylene (TCE) before and (blue) dimethylformamide (DMF) after phase transfer. For each set of spectra, the vertical line indicates the maximum absorbance of the band-edge transition at 1140, 1270, and 1400 nm, respectively. The spectra after ligand exchange have been offset for clarity. b) (top) Photograph of the extraction of QDs from (top phase) octane to (bottom phase) DMF and (bottom) representation of the phase transfer chemistry when using 3-mercapto-1,2-propanediol (MPD) and butylamine (n-BuNH2) as phase transfer agents, indicating several reactions that bring about the replacement of the as-synthesized ligand shell of chloride and oleylamine by deprotonated MPD and n-BuNH2. c) X-ray photoelectron spectra (red) before and (blue) after ligand exchange in different energy ranges, showing the disappearance of chloride, the appearance of sulfide and the preservation of the In:As ratio after ligand exchange.

a) Schematic of the In(As, P) QD field effect transistor, consisting of a spincoated film of ligand exchanged QDs on top of cross-fingered source and drain electrodes and separated from the gate electrode by a thermally grown oxide. b) Transfer characteristics of the field effect transistor at a source–drain voltage of 5 V.

a) (top) Energy level diagram of the In(As,P) QDPD stack used here. The diagram was constructed by combining UPS results for the 1140 In(As,P) QD film and literature data for the contact materials.[39-43] (bottom) Schematic of the QDPD stack. b–d) Dark and photocurrent densities under white-light illumination of In(As,P) QDPDs for specific absorber layers as indicated. e) Photocurrent density as a function of white light illumination power in log–log scale. The reference power is 114.7 mW cm^−2.

a–c) External quantum efficiency spectra for the different In(As,P) QDPDs as indicated, recorded at a reverse bias of −2, −3, and −4 V. The absorbance spectrum of the corresponding In(As,P) QD batch is added in each graph for comparison. d) Normalized transient photocurrent response of the different In(As,P) QDPDs following a 400 µs step illumination. Rise and fall times have been indicated by the dominant fast time constant obtained from a multi-exponential fit of the transient.

Full article (open access): https://onlinelibrary.wiley.com/doi/10.1002/advs.202200844