"Through this work we will specifically consider CMOS sensors that place intelligent circuits directly in the pixel. Coupling sensors with processors allows us to report new visual quantities as processing can interact, in a closed loop, with sensing. Besides, deporting processing closer to the sensor alleviates classical bottleneck issues in conventional vision sensors, as most of the information can stay and be processed directly on the sensor while only “relevant" visual quantities can be output. Indeed, blurring the frontier between sensing and processing using such sensor-processors on the hardware side calls for new algorithms that can run directly in the pixels and hence must be distributed, parallel and eventually triggered upon the arrival of an event.
Considering these first three principles 1) the joint inference of visual quantities, 2) “starting" by the observation of other visual information than direct correlates of illuminance 3) thanks to sensors that have more “intelligence” in-pixel suggests 4) to design vision systems more holistically. This ultimately means: from the physics of the transduction of a photon in the device up to its contribution to the inference of high-level visual quantities. A red-thread in this work has been the design of algorithmic solutions that try to exploit the peculiarities of the devices we used: not only this aims at creating high-performance systems, but also aims at exploring a new design space, building, for instance, systems that process visual information as exposure is happening.
This thesis demonstrates through various examples the practice of those unconventional sensors coupled with in-pixel circuitry and the kind of new algorithms it is possible to design for those. Examples include applications such as high dynamic range imaging and tone mapping, depth reconstruction from focal sweeps as well as tracking in the focal plane. These examples show how such sensor-processors simplify the design of systems that report other visual quantities than illuminance.
Then, this work takes a step back and suggests processing frameworks that can help guiding us in the design of algorithms for these parallel, distributed and eventually event-based devices. Specifically, we instantiate the idea of jointly inferring visual quantities in a more formal computing framework. Finally, considering vision systems holistically a need is to build tools that allow the use of these new kind of unconventional sensing and processing devices."
No comments:
Post a Comment
All comments are moderated to avoid spam and personal attacks.