Privacy-Aware Cameras for Human Pose Recognition

Carlos Hinojosa, Juan Carlos Niebles and Henry Arguello published an article titled "Learning Privacy-preserving Optics for Human Pose Estimation" in the 2021 International Conference on Computer Vision which was held virtually in October 2021. This is a collaboration between Universidad Industrial de Santander (Colombia) and Stanford University (USA).

The widespread use of always-connected digital cameras in our everyday life has led to increasing concerns about the users’ privacy and security. How to develop privacy-preserving computer vision systems? In particular, we want to prevent the camera from obtaining detailed visual data that may contain private information. However, we also want the camera to capture useful information to perform computer vision tasks. Inspired by the trend of jointly designing optics and algorithms, we tackle the problem of privacy-preserving human pose estimation by optimizing an optical encoder (hardware-level protection) with a software decoder (convolutional neural network) in an end-to-end framework. We introduce a visual privacy protection layer in our optical encoder that, parametrized appropriately, enables the optimization of the camera lens’s point spread function (PSF). We validate our approach with extensive simulations and a prototype camera. We show that our privacy-preserving deep optics approach successfully degrades or inhibits private attributes while maintaining important features to perform human pose estimation.

They take a "deep-optics" approach --- a learning-based approach where a neural network is used not only to recognize the human pose, but also to train a privacy-preserving point-spread-function (PSF). The neural network is trained to strike a balance between two competing requirements: (a) hiding scene information so that people's faces are not recognizable in the RGB images (even after image deblurring), while ensuring that (b) the PSF distortions aren't so strong that the pose-estimation task becomes impossible.

Their results look quite promising, and they even built a proof-of-concept hardware prototype using a wavefront modulator. Notice that the human faces are not recognizable in the RGB images, but the "match-stick" skeletons are still reliably picked out by the algorithm.

More details are in the open access paper and accompanying supplementary document and video available here: https://carloshinojosa.me/project/privacy-hpe/