Tl;dr: Dan Deacon worked with Google’s latest music AI models to compose the preshow music. Check out the MusicLM demo in the AI Test Kitchen app. Read on for more details about our collaboration with Dan Deacon.
Dan Deacon’s I/O Performance
On several occasions, we have had the pleasure of working with musicians that perform at Google I/O. This is an opportunity for us to bring our latest creative machine learning tools out of the lab and into the hands of the musicians. In previous years, we have worked with YACHT and The Flaming Lips. With YACHT we explored custom symbolic music generation models tailored to the band, and with The Flaming Lips we explored an interaction to bridge the audience and performers.
This year’s I/O pre-show was performed by electronic musician and composer Dan Deacon. With Dan we explored how artists might interact with generative models of music audio and incorporate them into their artistic process. Check out his performance in the video below and read on to learn more about his process using Google’s latest music AI tools:
Dan used two of our new generative models in his performance: MusicLM (paper, demo), which produces music based on a text-based input prompt, and SingSong (paper), which will generate an accompaniment track for an audio-based singing input. Both of these models are part of the AudioLM (paper) family, and they directly produce audio based on the input conditioning (i.e., text or singing) by autoregressively predicting SoundStream (paper) tokens with one or more Transformer language models. SoundStream tokens can then be converted back to raw audio that can be used in conjunction with other audio editing software.
For his performance, Dan used MusicLM to create the chill, relaxing piano groove that’s heard behind his two meditations starring the Duck with Lips. Additionaly, Dan used both MusicLM and SingSong to create the Chiptune song. Most excitingly, Dan didn’t just use both SingSong and MusicLM, but actually extended their capabilities to put his performance together. We’ll discuss more of how Dan shaped the tools–and why it’s important that he did so–in the next section.
Working with Dan
As Dan discusses at around 7 minutes into his performance, he has always been excited by the promise that new technologies bring to the compositional process. Technology has a long and intertwined history with the art of making music. We might not think of things like flutes, violins, or trombones in the same way we think of computers now, but these were revolutionary new technologies when they were first introduced! They can also often seem disruptive at first–at one point in history, microphones caused quite a stir because they let vocalists sing much more softly (opposed to singing so loud they could be heard over the band). Yet in retrospect, microphones changed our relationship to music in many positive ways, enabling us to create, represent, and distribute music in ways that would have been inconceivable beforehand. Importantly, each new technological development expanded the creative palette of musicians, bringing with them new textures, new techniques, and sometimes new conceptions of music itself.
We view our new models as a continuation of music technology’s evolution. We’re incredibly inspired by the opportunity for these new tools to bring new creative capabilities to humanity, while remaining conscious of–and working hard to mitigate–their potential negative consequences. Our goal is and always has been to empower artists and musicians; a crucial piece of empowering musicians is understanding now these new tools situate themselves in different artists’ creative processes. With that in mind, collaborating with Dan was a great opportunity for us to work towards embodying our goals of empowering musicians in the era of generative modeling.
About a month before I/O, we had a workshop with Dan where we introduced him to MusicLM and SingSong. Initially, Dan found many interesting text prompts to our MusicLM such as “a 600ft trombone.” He started to push the tools past their limit by, for example, playing his synthesizer into SingSong, ignoring that the system was trained on only singing inputs. These initial experiments turned out to be really fun and promising!
As we kept working with Dan, he surprised us by pushing these tools even further. Inspired by “I Am Sitting in a Room” (click here to listen), he fed the output of the SingSong model back into itself… over and over and over. Again, Dan moved beyond the model’s design of accepting singing input; by feeding its own output back into itself, the input audio was out of the distribution that the model had seen during training and we weren’t sure if this would work at all. Yet, not only did it work, but the feedback loop tended to produce music that still accompanies the input; it has the same key, tempo, and style. This was the interaction that Dan designed to compose the Chiptune song, above.
Dan began with a handful of text prompts to MusicLM, and then used the generated audio as input to SingSong and that output back through SingSong for numerous iterations. He was able to create hundreds of audio clips that complemented each other. From these, he handpicked his favorite clips, edited them slightly, and performed them.
We’re very proud to have been a part of Dan’s amazing performance. We’re extremely excited for the direction that this research is headed, and we’re always looking for ways to give musicians new tools to interact with. Check out the Google Keyword blog post to learn more about MusicLM and you can try it yourself by signing up via the AI Test Kitchen app.
Acknowledgements
This year’s I/O pre-show was a huge collaborative effort. We would like to thank everyone involved in making the performance a success (in no particular order): Josh Christman, Daniel Chandler, Meghan Reinhardt, Carolyne De Bellefeuille, Adi Goodrich, Jon Barron, Meghan Reinhardt, Carolyne De Bellefeuille, Irina Blok, Spencer Sterling, Ruben Beddeleem, Ben Poole, Cadie Desbiens-Desmeules, Chris Donahue, Jorge Gonzalez Mendez, Noah Constant, Jesse Engel, Timo Denk, Andrea Agostinelli, Neil Zeghidour, Christian Frank, Mauricio Zuluaga, Hema Manickavasagam, Tom Hume, and Lynn Cherry.