Facebook aims to give you perceptual superpowers through its new augmented reality glasses. A revolutionary Facebook experiment aims to demonstrate how sound could play an important role in augmented reality, to the point of giving people something that researchers from the social network call “perceptual superpowers.”
Facebook and the new Augmented Reality glasses
Facebook Reality Labs Research (FRL Research), the research laboratory of the social network, has been developing a project based on augmented reality and virtual reality for six years that seeks to improve human communication.
An interdisciplinary audio team, made up from research scientists, engineers, designers, among others, has the mission of creating virtual sounds that are perceptually indistinguishable from reality and redefining human hearing.
The new augmented reality glasses from Facebook could allow audio capture with microphones, using contextual clues that are able to measure what sounds are important and transmit them via a headset with noise cancellation.
Additionally, an improved spatial sound could project the voices of the participants in a telephone conversation or a video call, increasing the feeling that they are with another person or in the presence of audio.
To achieve this, the team has focused on offering two capabilities:
- Audio Presence: A feeling that the source of a virtual sound is physically present in the same place, with such high fidelity that is indistinguishable from the real world.
- Perceptual superpowers: technological advances to hear better in noisy environments, turning up the volume of the person sitting in front of us and lowering the audio for unwanted background noises.
Audio calibration
Facebook detailed the progress of the research of the team led by Ravish Mehra, which has developed an active audio calibration system, which allows you to listen to sounds with ultra-high fidelity through a pair of glasses equipped with headphones and 32 microphones.
According to the company, the mechanism uses an algorithm and software processing technologies that allow to recreate the acoustics of a room and make devices such as headphones transparent or imperceptible to people.7
- How to create a survey using Facebook Messenger?
- Messenger messages can only be forwarded to 5 people now
- Microsoft explains why it won’t allow you to disable Microsoft Defender
In 2017, the team developed virtual sounds that mimic the directions that sounds come from in real life and high-quality acoustic simulation technologies, which make virtual environments more believable. Technologies have driven the state of the art in spatial audio and power many of today’s experiences, such as the Oculus Quest virtual reality glasses and the Rift platform.
Improving spatial audio
One of the main tools that sustains the work is spatial audio, which offers the feeling of being present in the same place, even though they are actually miles apart.
Spatial audio mimics the directions that sounds come from in real life and ambient acoustics, something that could be a game changer for video calling.
“With a phone call today, the other person’s voice sounds like it’s coming from the phone itself (or the center of your head, if you’re wearing headphones), so your brain rejects the idea that the other person might be in the same location, “describes Lisa Brown, manager of Communications Technologies at Facebook.
The company believes that spatial audio can be combined with Codec Avatars, ultra-realistic renderings of people animated in real time. “This will allow breaking the true social presence. By allowing you to spend time with the important people in your life in meaningful places, we can radically transform the way you live, work and play, ”adds Brown.
Perceptual superpowers
For example, It would be possible to carry on a conversation in a crowded restaurant or bar, without having to raise your voice to be heard or make an effort to understand what others are saying.
By using multiple microphones on the glasses, it is possible to capture the sounds of the environment. Then, with the pattern of the movements of the head and eyes, we can find out which of these sounds he is most interested in hearing.
“This allows us to enhance the appropriate sounds and attenuate others, ensuring that what you really want to hear is clear, even with loud background noise,” Mehra says.
The team’s goal is to cover the full range of sounds that humans can hear, from 20Hz to 20,000Hz.
Currently, they develop Special Internal Monitors (IEM), an output prototype, which allows the use of active noise cancellation technologies to effectively reduce the volume of unwanted background noise.
“When we combine this with FRL Research’s input prototype, including the microphone array, we can deliver the full hearing superpowers experience,” explains Brown.
