Editorial Note: At I/O 2019, we announced collaborations with two musical artists during our talk session: YACHT and The Flaming Lips. You can read about these collaborations on the Google Blog, but we are also excited about revealing more details of this work on our blog in the coming months!

In this post, Deeplocal, a creative studio we partnered with, discusses the development of “Fruit Genie”, an AI-assisted instrument debuted by The Flaming Lips on stage at I/O. They also provide instructions for you to build your own Fruit Genie at home!

Fruit Genie

We (Deeplocal) had the opportunity to work with Google’s Magenta team and The Flaming Lips to create an AI-assisted performance as part of the headline concert at I/O 2019. The result was Fruit Genie: a real-time intelligent musical instrument which combines Magenta’s Piano Genie model with a physical interface consisting of fruit (or whatever else you can dream up)!

Here is a video summary of the experience, but keep reading to get into some of the nitty-gritty details and behind-the-scenes process!

The Beginning

From the beginning, we knew we wanted to integrate AI into the work of one of the most innovative bands in the music industry, The Flaming Lips. We were all psyched to be working on this, but also saw that the challenge was extremely open-ended and broad. Luckily, we had a great source of inspiration in the catalog of existing work by the Magenta team.

We explored some ideas around reinterpreting existing Flaming Lips songs through the lens of tools like MusicVAE, or extending the great work around NSynth to create a one-of-a-kind Flaming Lips instrument.

After a day of brainstorming with the Magenta team, Wayne and Steven from The Flaming Lips kept coming back to Piano Genie for both its novel input and AI-powered dynamic output. Piano Genie is a real-time system which translates inputs on a simple controller into plausible music. Among Magenta’s AI-powered music systems, it is one of the few that works in a real-time context, allowing actions to be translated to music instantaneously. This made it a natural pairing with band’s iconic live experience which consists of real-time musical and visual elements.

The Fruit Plate

We didn’t have the actual Piano Genie button-based hardware in our studio that day, but we did have an enterprising electrical engineer named Nico who said he could put something together. He quickly got the hardware up and running that could send MIDI to the Piano Genie software, although we didn’t have any large buttons on hand to create the original physical interface.

In a stroke of genius (or maybe desperation), Nico grabbed a plateful of fruit, inserted some leads into each piece, and used the natural capacitance to turn grapes, oranges, and pears into “buttons.” Now, whenever you touched a piece of fruit, it generated a MIDI trigger which the Piano Genie software then turned into a musical note. The end result was similar to a “fruit piano” (a popular Makey Makey project), but with the pitch of each fruit controlled by the Piano Genie model instead of mapping to a single pitch.

Quickly everyone was taking turns “playing” the fruit plate. As Steven rapidly became a virtuoso, you could see the wheels turning in Wayne’s head.

Apologies for the cellphone footage, but Adam’s embedded camera work really captures the excitement of the moment best.

The Song

With the new “Fruit Genie” instrument in their arsenal, The Flaming Lips went back to Oklahoma City and began experimenting and writing a new song. As they ran into roadblocks or made suggestions, they were able to get timely updates from the Magenta team. These updates included the addition of chord- and key-conditioning to the Piano Genie model (contributed by Chris Donahue) to allow the model to be integrated contextually into a musical performance. Soon we found ourselves at their studio watching them perform a brand new song with the help of a fruit plate and artificial intelligence. There’s a sentence that has probably never been written before.

The Performance

The Flaming Lips are known for having larger-than-life shows, so we knew that a small plate of fruit wasn’t going to cut it for their performance at I/O. We also wanted the crowd to be involved, so we took some inspiration from The Flaming Lips’ history of throwing out large inflatables to the crowd and ran with it.

Using some of Wayne’s designs…

…we created oversized, fruit-shaped balloons and outfitted them with lights, accelerometers, and wireless communication capabilities. Whenever someone in the crowd batted a balloon up into the air, it sent a signal to the Piano Genie software to play a note. Here’s what it looked like from a fan’s perspective (in 360!):

Conclusion

It was an incredible experience building Fruit Genie with The Flaming Lips and Magenta, and a pleasure to watch it come to life on stage at Google I/O. It was challenging to identify a meaningful way to integrate AI into a live musical experience. However, this challenge is justified by the potential rewards: such technology can redefine the concert experience by allowing the crowd to become part of the performance. We see Fruit Genie as an important step towards this goal.

Make Your Own

Want to make your own musical fruit plate? Check out the instructions on GitHub.