Researchers at Cornell University have developed sonar glasses that have the ability to track facial movements for silent communication. The eyeglass attachment, created by Ph.D. student Ruidong Zhang, uses tiny microphones and speakers to read the words that a user mouths silently, allowing them to command actions such as pausing or skipping a music track, entering a passcode on their phone, or working on CAD models without a keyboard.
The system builds on a previous project by the team that used wireless earbuds and eliminates the need for a camera or an earpiece, making it more practical and feasible for everyday use. “Most technology in silent-speech recognition is limited to a select set of predetermined commands and requires the user to face or wear a camera, which is neither practical nor feasible,” explained Cheng Zhang, Cornell assistant professor of information science. “We’re moving sonar onto the body.”
The system requires only a few minutes of training data, such as reading a series of numbers, to learn a user’s speech patterns. Once trained, it uses sound waves sent and received across the user’s face to sense mouth movements and analyzes echo profiles in real time using a deep learning algorithm, achieving an accuracy of about 95%.
One of the key advantages of the system is its privacy features. The data processing is offloaded wirelessly to the user’s smartphone, ensuring that no data leaves the phone and eliminating privacy concerns. The current version of the system offers around 10 hours of battery life for acoustic sensing, making it practical for everyday use.
The potential applications for the sonar glasses are wide-ranging. They could be used to control music playback in quiet environments, dictate messages in noisy settings where other options would fail, or even assist people with speech disabilities by feeding dialogue into a voice synthesizer. The researchers are also exploring commercialization options through Cornell’s funding program and are looking into applications of smart glasses to track facial, eye, and upper body movements.
“We’re very excited about this system because it really pushes the field forward on performance and privacy,” said Cheng Zhang. “It’s small, low-power, and privacy-sensitive, which are all important features for deploying new, wearable technologies in the real world.” The team at Cornell’s Smart Computer Interfaces for Future Interactions (SciFi) Lab sees great potential in using glass as a personal computing platform to understand human activities in everyday settings.
