Description
The art of music has always been incredibly immersive, capable of taking us on emotional journeys to different times and places and with the emergence of XR technology, the potential for music to create even more immersive experiences has skyrocketed. XR combines virtual reality (VR), augmented reality (AR), and mixed reality (MR) to create experiences that blur the lines between the physical and virtual worlds. For musical XR, this means creating environments where users can not only hear the music but also see and interact with it in innovative ways. The integration of extended reality (XR) technologies in the realm of music are bringing about a new level of immersive and interactive experiences for artists and audiences alike. This article explores a few ways XR enhances musical performances and experiences.
What is Immersion?
Immersion is a state of being deeply engaged in an experience. It could shift perception, blurring the lines between what is experienced and what is real. This is a technique much older than any technology.
The act of creating immersive experiences is innate. Kids do this all the time; you know how as kids, we used to imagine different scenarios, play different roles, and just conjure fantastic ideas and narratives from the endless stream in our minds? We used to pretend we were singers, doctors, teachers, superheroes, or even beings from the outside earth. A little cardboard box could have been a spaceship; our dolls and action figures were very much real and alive in our make-believe worlds. Games in different forms also strengthened these experiences by allowing us to become totally engrossed in the ideas behind them.
Our ancestors who lived in caves and rock shelters left us some evidence of how they imagined or reimagined their reality — they painted animals, plants, other objects from their natural world, and extraterrestrial beings on their walls and grounds using vibrant mineral pigments like ochre, limonite, charcoal and whatever else was available to them. The paintings would have been visually striking due to the vivid colours and bold lines against whatever surface they were made on, and they must have had to work In the darkness using fire lamps or torches, which would have flickered and made the art seem animated as they moved around. Although we have no recordings of what they must have sounded like, combining this with music, dancing, and cueing the other sounds and smells from their environment must have created a multi-sensory immersive experience.
Today we have access to a wide range of mediums and technology that help us further explore the depths of our imagination and create highly immersive experiences for ourselves. We are able to engage with scenarios that are not possible in the real world, constantly shaping the future of how we relate, think, and connect in new and exciting ways. Our immersive experiences and those of our ancestors are distinct, but they are important expressions of conscious human creativity and our longing to interact with our environment and blend it with
Musical XR
The intersection between XR technology and music has grown over time; we now have better and faster digital creation, virtual events, and performances; this has been grounds for constant debate regarding use and definition. Throughout history, humans have used materials readily available to create music and musical instruments. The digital age and the internet have been a catalyst in the creation and dissemination of music.
With the advent of cognitive technologies such as Artificial intelligence (AI) and Extended-reality (XR) faster processing speed on newer and smaller devices, integrating these technologies and music is unavoidable. In our post-pandemic world, these technologies enable musicians and creators to tell their stories, convey emotions, and engage audiences in a very interactive manner, whether present or remotely, through novel musical experiences and captivating soundscapes. As more musicians explore the application of XR in concerts, creating and teaching music education will revolutionize the industry in ways never thought possible.
Defining what constitutes a Musical XR experience requires looking at it on a spectrum considering sound and music’s significance. In Musical XR projects, the way in which sound and music are manipulated, such as by changing their pitch, volume, spatial, and tonal properties, must be tied to the user’s actions within the set environment. Musical XR projects can take many forms, from virtual musical instruments to DJ sets in immersive concerts, generative music systems, and musical games. The use of sound and music in Musical XR systems can vary greatly. It can be integrated as part of the story or as background elements. It can be pre-recorded, played back as is, or generated interactively or procedurally. It can have a stereo sound that is tied to the user’s head or body movements or spatial audio that changes based on the user’s location. A fundamental aspect of technology should be the organization of sound. This can govern the structure of the virtual environment or its events, and the user’s interaction with the environment can be informed by or facilitate musical expression.
Examples of Musical XR Experiences
SoundSelf — a technodelic, meditative experience that combines chanting, meditation, and virtual reality. In SoundSelf, players use their own voices to create beautiful harmonic tones and experience a calm, hypnotic, and meditative journey. As the user speaks or sings, their voice is processed and transformed into an audio-visual feedback loop. The game uses real-time audio analysis to create visuals that are synchronized with the player’s voice, creating a unique and immersive experience. SoundSelf is designed to help players achieve a state of trance, relaxation, and inner peace through music and technology.
Electronauts — This provides a new way for musicians to create and share music. Rather than physical instruments and 2D software, users have a range of virtual synthesizers, drum machines, and samplers. Multiple users worldwide can jam, spin tracks, and build on each other’s ideas with few limits. They can share finished tracks or live shows, get feedback, gain exposure, and rise on leaderboards. By combining VR and music, Electronauts enables new forms of creativity, collaboration, and entertainment. The software lets anyone passionate about music fully immerse in music, transcend physical limits, and achieve a musical expression that would otherwise be impossible.
Fract OSC — A truly innovative musical exploration game that combines puzzle-solving with music creation. In FRACT OSC, players explore a virtual world, solve puzzles, and create their own music by manipulating the many unique virtual instruments and ambient soundscapes placed throughout the environment. virtual instruments and soundscapes. Players are transported to a mysterious virtual world filled with hidden secrets waiting to be uncovered.
Tonandi — is a unique mixed-reality experience that combines music, nature, and technology to create an immersive interactive world. Created by the Icelandic post-rock band Sigur Rós in collaboration with Magic Leap, a company developing augmented and mixed reality technologies, Tónandi is a surrealistic musical XR experience showcasing the possibilities of creatively mixing music and technology.
TheWave VR: This virtual reality platform was created using Unity, a popular game engine, and is designed to offer a fully immersive experience for users who want to DJ and perform music in a virtual environment. The platform allows users to create and customize their avatars, interact with other users in real time, and experience music in a new and exciting way. TheWave VR also offers a social aspect, as users can invite friends or strangers from around the world to join their virtual parties, creating a unique sense of community and connection.
These are a few examples of Musical XR experiences, which combine various technologies to create immersive and interactive musical environments. These experiences typically make use of virtual reality, augmented reality, or mixed reality technology, as well as 3D graphics and spatial audio. In all cases, these experiences make use of cutting-edge technology to create new and exciting ways for people to engage with music, whether by creating it themselves, exploring virtual soundscapes, or performing for others in a shared virtual space.
Performance formats had already been evolving for some time before the outbreak of the COVID-19 pandemic. The Sound of One Hand by Jaron Lanier in 1993 is one of VR’s earliest and most referenced musical performances. Lanier performed live while wearing a virtual reality headset and a single instrumented glove. The performance occurred on a plain stage, and Lanier was fully immersed in a virtual environment. Virtual concerts have also been held in computer-simulated environments since the early 2000s.
In 2003, the English rock band Duran Duran became the first to perform in a virtual world called Second Life, and many artists followed suit, providing their fans with the thrill that comes with interactive experiences. Post-pandemic, however, the music industry has had to be even more innovative in finding new ways of connecting with audiences. New, inventive formats for performances have had to be devised to ensure that the fans can still enjoy live performances in different forms wherever they are.
Artistes have performed live sets inside games; for example, rapper Travis Scott’s ‘Astronomical’ performance on the video game Fortnite in April 2020, which had about 27 million players watching live, and Lil Nas X’s performance on Roblox, which brought in about 33 million views in two days. MTV Video Music Awards, arguably the top music industry award, recently added a new category celebrating the use of immersive technology in music for artists like Justin Bieber, BlackPink, and Charli XCX, who hosted virtual concerts on platforms like Roblox, Minecraft, Wave, amongst others adding another block to the general acceptance of Musical XR.
Recording on Virtual Instruments
This involves capturing the audio generated by the virtual instruments in real-time, like an actual DAW, either by direct audio recording or Midi recording. The direct audio recording captures the sound as heard by the user, with any added effects or post-processing. The midi recording captures the virtual instruments’ output which can then be used to recreate or remix the performance. Midi recording allows you to edit further, manipulate the sound collected, and be used for live performances and collaborations. Examples of virtual instrument programs are PatchWorld which allows you to create and collaborate in VR, and Tribe VR, which allows you to DJ, create, and perform live in virtual reality.
Audience Engagement — The ability to view lyrics while a song is playing or being performed offers a more interactive and enjoyable experience for the user, especially during live performances. This allows the audience to be fully immersed in the experience, whether it’s seeing the lyrics in front of them or exploring the stage and environment from different angles.
Sound Engineering — XR provides a range of capabilities for mixing and manipulating audio in impossible ways for conventional tools. Virtual reality tools like Dear Reality offer immersive audio controllers that virtualize real rooms. This gives us a unique approach to sound source visualization enabling sound engineers to control the sources directly and create realistic sound fields around the listener.
Education — Education using XR is an area of high impact; VR performance training has been used to reduce performance anxiety. Musicians can use this technology to train themselves in different real-world scenarios, helping them improve their skills. It has also been proposed for use to teach students about music theory, rhythm, acoustics, and other music-related fields.
Conclusion
Musical XR is a total game-changer for the music industry, it will change the way we experience music. Although still in its infancy, this technology is quickly becoming an industry standard, with more and more people adopting it. XR studio sessions and simulated reality are paving the way for a new era of spatial experiences. The possibilities are limitless, from virtual reality concerts that transport you to the front row to augmented reality music videos that come alive right before you. Musicians and artists now have new tools and media to create music that pushes past traditional constraints. They can experiment with new sounds and genres, leading to more innovative compositions. They can craft immersive concert experiences that go far beyond what is possible on stage, providing audiences with ground-breaking ways to engage with the music. With Musical XR, artists, and labels have the opportunity to monetize these novel immersive experiences. Musical XR is a total game-changer that will fundamentally disrupt how we experience music.
References
https://benspooner.medium.com/diegetic-non-diegetic-and-trans-diegetic-sound-in-xr-7d94f00f97a1
https://medium.com/pintsizedrobotninja/using-sound-in-xr-8fa566e84a54
