"Image denoising and demosaicking are the first two crucial steps in digital camera pipelines. In most of the literature, denoising and demosaicking are treated as two independent problems, without considering their interaction, or asking which should be applied first. Several recent works have started addressing them jointly in works that involve heavy weight neural networks, thus incompatible with low power portable imaging devices. Hence, the question of how to combine denoising and demosaicking to reconstruct full color images remains very relevant: Is denoising to be applied first, or should that be demosaicking first? In this paper, we review the main variants of these strategies and carry-out an extensive evaluation to find the best way to reconstruct full color images from a noisy mosaic. We conclude that demosaicking should applied first, followed by denoising. Yet we prove that this requires an adaptation of classic denoising algorithms to demosaicked noise, which we justify and specify."
If we're using neural networks, we should just lump these two steps together.
ReplyDeleteBefore NNs are cheap enough for high throughput real-time applications, applying appropriate denoising before demosaicking may have several benefits.
The paper showed that, assuming white noise,
ReplyDeleteDM first, then DN could be a very simple change of the noise parameter of the denoiser DN coped with the structure of
demosaicked noise, and led to efficient denoising after demosaicking
As I know,
For some(maybe most?) of middle to high-end ISP chip, in conventional ISP HW pipeline:
the democasic HW logic were implemented based on Residual interpolation(RI-based) algorithm,
the main 2D denoise HW logic were implementation based on color channels(CMB3D) or spatial(NLM) patch-based correlation(similarity) algorithm in Bayer domain.
that is DN first, then DM.
it is not only cost-driven consideration but also architecture-level consideration due to other ISP function in Bayer domain (ex: HDR, WB,..)
And, base on conventional ISP pipeline,
it is possible using DL-based algorithm to replace some parts of conventional ISP pipeline
ex: DL-based SR to replace kernel function of RI-interpolation or DL-based similarity decision on Bayer domain.
But, I think, if not considering cost,
as paper state that end-to-end joint demosaicking and denoising DL methods would win the SNR and also visual quality game, no matter white noise or color noise
DL-based ISP scheme for human eye-ball prefer,high visual quality is not so critical in real applications.
Cause the current conventional ISP did it well enough.
But light(simple) SW-based ISP with basic ISP functions or DL-based application driving ISP architecture might be another directions.