One of the headline features of the Mate 30 Pro is its super slow motion capabilities. Most smartphones available on the market today, if they offer slow motion, do it at either 120 fps, 240 fps, or up to 960 fps. This typically comes in the form of a sensor and system that can take 120 frames per second video and do clever interpolation to appear as if it is a higher frame rate. We’ve seen some good and bad solutions here.
The way that traditional slow motion cameras work is on a sliding scale – the trade-off between frame rate is resolution. If you reduce the area of the sensor that needs to take the image by one quarter, then technically the fast memory storing the video data can store 4x as much – as long as you can poll data from the sensor at 4x the speed, it should be good to go.
It seems that Huawei is doing a mix of things here to get 7680 fps. The camera offers several settings, which comes with a slider from 2x to 256x. Any setting 32x or lower gives a 1080p video, while 64x and higher gives a 720p video:
- 1080p at 120 fps = 4x speedup
- 1080p at 240 fps = 8x speedup
- 1080p at 960 fps = 32x speedup
- 720p at 1920 fps = 64x speedup
- 720p at 7680 fps = 256x speedup
I do not see where is their attempt.
ReplyDeleteIn my opinion, the sensor is used in a higher-order interlace (that is, not just 2 fields are regular interlaced systems work, but 4 or 8 fields). I have recently seen this feature in some datasheets. The source data is stored into memory, which limits the duration, and deinterlaced afterwards with a motion adaptive process.
In the lipstick movie, significant ghosting/blurring occurs in the moving parts.
This could be a part of explanation. Still, 720p at 7680 fps means about 200ns per row. If we assume 8b per pixel, the generated data would be 60 Gbps. For 0.12s recording, they need 1 GB of memory, 8x of IMX400 capacity. Any guess how do they achieve all of that?
DeleteI guess it's ML.
ReplyDeleteIndeed, they are using AI to pick the best moment for the recording. Since the recording time at maximal speed is only 120ms, it's hard to do it manually.
DeleteAs for the possibility of time interpolation just with ML, it's hard to believe they can do it with such a high quality. But this could be a combination of, for example, 3,800 fps physical recording with a further 2x interpolation with ML.
It's motion interpolated from native 1920fps capture.
DeleteInteresting... Do they interpolate only in time domain? Or use spatial interpolation too? Like capturing in QVGA and upsampling it to 720p? Then, the whole system would be much more manageable.
DeleteOne can see some trailing when the mouse trap moves over the rough background. As if the background is stretched along with the moving subject.
DeleteFast JPEG-like compression on the fly (on GPU)?
ReplyDeleteLike this one:
https://www.fastcompression.com/blog/fast-raw-compression.htm
I would imagine, for many application, a high resolution at super high speed may not be required.
This is a great tool for fun and science projects!