Researchers at the University of Maryland have made strides in transforming eye reflections into 3D scenes, albeit with limitations. Building upon Neural Radiance Fields (NeRF), an AI technology capable of reconstructing environments from 2D photos, the team delved into the possibility of extracting information about a person’s surroundings from subtle light reflections in their eyes. While this eye-reflection approach is far from practical applications, the study provides an intriguing glimpse into a technology that could one day unveil environments based on simple portrait photos.
To achieve their goal, the researchers utilized consecutive images captured from a single sensor, focusing on the reflections of light in human eyes. By analyzing high-resolution photos of a person looking towards a fixed camera position, they isolated and examined the eye reflections to determine where the eyes were directed in the images.
The results revealed a discernible reconstruction of the environment based on human eye reflections in a controlled setting. Additionally, the team experimented with a synthetic eye, which produced an even more impressive dreamlike scene. However, their attempt to model eye reflections from music videos featuring Miley Cyrus and Lady Gaga only yielded vague blobs, indicating the technology’s current distance from real-world use.
The researchers faced substantial challenges in reconstructing even crude and blurry scenes. The cornea, for instance, introduced inherent noise that made it difficult to separate the reflected light from the complex textures of the iris. To address this, they implemented cornea pose optimization to estimate the position and orientation of the cornea, as well as iris texture decomposition to extract unique features from individual irises during the training process. Additionally, they employed radial texture regularization loss, a machine learning technique that simulated smoother textures than the original material, further enhancing the isolated reflected scenery.
Despite the progress made and the clever workarounds devised, significant obstacles remain. The researchers noted that their current results were obtained in a laboratory setup, which involved zooming in on a person’s face, illuminating the scene with area lights, and controlling the person’s movements. They acknowledged the challenges of achieving similar results in more unconstrained settings, such as video conferencing with natural head movements, due to factors like lower sensor resolution, dynamic range, and motion blur. Furthermore, the team recognized the need to develop less simplistic assumptions about iris texture, as real-world scenarios involve wider rotation of the eyes compared to the controlled environment of their study.
Nonetheless, the researchers view their progress as a milestone that can pave the way for future breakthroughs. They hope to inspire further exploration of unexpected visual signals that can unveil information about the world around us, expanding the horizons of 3D scene reconstruction. While more advanced versions of this technology may raise concerns about privacy intrusion, it is important to note that the current iteration can only vaguely discern details, such as a Kirby doll, even under the most favorable conditions.
