Binaural Audio

By Tom Green & Sofie Andersen


Bjork’s latest video was recently released online but it's initial release was at several 'live' installations this spring at MoMA’s PS1 in Long Island City and at a Rough Trade shop in New York or in London. If you did make it to one of these locations you would’ve experienced what it was like to go inside the video by wearing a newly developed freefly Virtual Reality headset. It was shot in 360°, meaning you can look in any direction at any time, as though you were actually there on an Icelandic beach with Bjork. Now, if you didn't make it you can get some semblance of the experience on the YouTube app on your phone.

It’s not the first 360° music video. But what is particularly interesting about this project is its use of sound. Created by REWIND and Two Big Ears, the experience uses real-time binaural audio mixing to put the user at the centre of an orchestra, meaning that as they look around, they can hear the different musicians in different directions.

Binaural audio is a recording or post production technique that recreates sounds as though they were heard by the human ear. Unlike ’straight stereo’ it takes into account the minute delays between a sound reaching each ear if it’s to one side of the head, and the way a sound changes as it passes through the skull from one ear to the other. The result is an ‘ultra immersive’ recording which can place a listener right inside an audio environment.

Where binaural gets really exciting is when it’s combined with real-time processing, such as in the Bjork video, so that it takes into account the direction that the user is looking in - it becomes the audio equivalent of virtual reality. The combination of the technique with the emotive impact of watching Bjork’s performance on virtual reality headsets and hearing her music at the same time goes beyond the thrills of technology and high production values, and enters into a much more interesting realm of an experiential art form. Seeing Bjork perform as if only for you is one thing, but to be able to follow her (or one of her as she splits into three) and hear her voice change as it disappears behind you, gives an intense sense of being there with the artist, on the chilly Icelandic beach and carried into her intimate universe. It’s no wonder that staff at PS1 reported visitors staying to watch it again and again, many moved to tears.

Use of dynamic binaural audio applications is a growing trend. Pioneered by the critically acclaimed cult  ‘video games with no video’ Papa Sangre & Papa Sangre 2 (full disclaimer: in a previous life Tom was game director and producer of Papa Sangre 2), there are now a number of deeply immersive audio games that make use of this technology.

Sofie experiences Stonemilker at MoMA's PS1

Sofie experiences Bjork's Stonemilker at MoMA's PS1

The applications of this kind of audio for the cultural sector are huge. We recently experienced the incredible lunch counter simulator designed by the Rockwell Group at the Center for Human and Civil Rights in Atlanta. The installation invites visitors to seat themselves at the counter and pick up headsets to listen. ‘How long can you last’ the sign invites you to consider, while a stop watch tracks your time. Once you’re wearing the headsets, a voice calmly tells you just to mind your business and stay seated. But the reality of why most stand up again after 30, 45 or 60 seconds quickly sets in. The soundtrack is devastating. You are harassed verbally, then the ground literally thumps and shakes with the shear force of the sound of the men shoving and hitting. And make no mistake it feels unmistakably like someone is literally breathing down your neck. This isn’t history told in words, pictures or any kind of traditional narrative. It’s theatre of the mind for one.

Lunch Counter at the Center for Civil and Human Rights, Atlanta.

Lunch Counter at the Center for Civil and Human Rights, Atlanta.

So taking it one step further, what if in your next museum visit you could explore with your ears, with different statues, or portraits, talking to you from different directions? What if we could extend the art inside a gallery to have an audio dimension that fitted with the physical space? What if you encountered historical events as if you are truly in their time and space? No more creating interpretation as a version of the didactics, but creating a truly lived experiences. Now that's an exciting prospect.

- author