"...there exist disadvantages to the process of generating a HDR image from multiple independently captured images. For example, changes may occur when images are captured successively such that a composite HDR image generated therefrom may not be completely aligned. This may generate motion artifacts in the composite HDR image. Further, the images may be affected by local motion in the image scene, e.g., trees swaying in the wind, people and faces shifting slightly, etc. Additionally, the time required for a HDR image to be processed may be delayed based on the images to be captured."
So, the solution is to read same row several times, each time with different exposure. Then the several reads are combined into a single HDR row in an image processor:
"To generate a HDR image during a single exposure of the frame (i.e., fixed amount of time t during which the rolling shutter reset 80 moves across a frame), multiple reads of the same row of pixels of the image sensor may occur. For example, a first data read 86, of the data stored in a row of pixels, may be undertaken at a time n, where n is a fixed fractional time of time t. This time n may be, for example, 1/2, 1/3, 1/4, 1/5, 1/10, 1/20, or another value of the frame time t. This time n may be represented as line 88 in FIG. 5. That is, the first data read 86 may occur at a time n subsequent to the reset of a row of pixels by the rolling shutter reset 80.
Accordingly, as the rolling shutter reset 80 passes downwards along line 78, the first data read 86 may trail the rolling shutter reset 80 by time n. In this manner, data stored in the pixels for each row of the frame may be read at a time n after the rolling shutter reset 80 of that row of pixels. Thus, each row of pixels read as the first data read 86 passes across the image sensor will have been exposed to light for the same time n, which may be referred to as an exposure time or integration time."
While the exact reference escapes me, I've heard about a similar technique being used by a major security camera vendor for more than 10 years now. It might be new in a smartphone world though.
While we are at Apple, Terry Gilton, Senior Director of Sensor Technology at Apple, posted a note on his LinkedIn page:
"I am seeking some more ultra-motivated, highly innovative sensor engineers. Some opportunities in characterization, validation, analog design, and architecture of all kinds of sensors. If you are passionate about sensing stuff and feel like you want to change the world, let me know."
Multiple exposures with different exposure times is a well-known approach for HDR/WDR, as far as I know... Unless the implementation is really new...
ReplyDeleteNow something I had not seen is HDR with multiple exposures with different apertures and the same exposure time, as this paper proposes:
http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1485297
Yes, multiple apertures is an interesting approach, albeit quite computing intensive in DoF management part. May be a simpler one would be to use different ND filters instead of different apertures.
DeleteAs for Apple, they are trying to patent a rolling shutter where each row exposed few times in an interleaved manner. This is not new as a stand alone idea, but in combination with smartphone it might pass the patent office, who knows.
Seems so ... familiar.
ReplyDeleteFrom 1995: US Patent No. 6,115,065 for example. I believe it is already used in some products.
I think the surveillance camera design you are referring to is based on Pixim (recently bought by sony) technology.
ReplyDeleteNo, Pixim uses a different technology.
DeleteBut the basic principle is the same. Except that Pixim puts a simple ADC inside the pixels, so they can read data out at high speed. More frames can be extracted for HDR reconstruction.
DeletePixim shows better image simply becuase they use more exposures and more frames. But this approach is hard to make megapixels sensors.
Multiple exposure pixels for image sensors are also described in patent US2010/0090091
ReplyDelete2010 haha !!!!
ReplyDeleteHere's my description of the idea, back in April 2007, when I thought it was original: http://ambivalentengineer.blogspot.com/2007/04/better-video-ad-converters.html
ReplyDeleteThe maths behind this method are described in books ... I have referenced several similar papers in my recently published HDR book.
ReplyDeleteI can only hope that you took the time to do a thorough literature search all the way back to CCD days as well search the patent data bases. When you write a book, you have a much greater onus to be thorough.
DeleteLooking forward to your book's release. I will be buying one for my lab and my students for sure.
Can I patent the Apple in the USA? Then i sue Apple of being Apple.
ReplyDeleteHow is the multiple non destructive readout achieved？In rolling shutter the readout is destructive as the reset and readout happen at same time if my memory serves me correctly...
ReplyDelete