Editorial Note: Here we present a blog post from our friends at Google Arts & Culture, who built a fun musical experiment based on DDSP.
In June 2021, the Google Arts & Culture Lab released an online experiment called Paint With Music, in which the users could create music by painting on sensorial canvases. This experiment is the final outcome of an adventure that started months prior, with a one-week sprint that gathered the Magenta team together with the Lab. This article is a brief summary about the making of this experiment.
Music for everyone
The initial goal of Paint With Music was to make music accessible and fun for musicians and non-musicians alike. With that idea in mind, Paint With Music connects music with another major form of artistic expression: painting. Helped by Magenta’s DDSP library (Differentiable Digital Signal Processing), the drawings of the user are translated into musical notes, performed by a chosen instrument. To make the experience more unique and personalized, the user can paint on sensorial canvases, such as the sky, or the ocean, amongst others.
A glitch snippet as a starting point
The starting point of this project was an open-source prototype made by two Magenta researchers, Jesse Engel and Ian Simon. The prototype, called ddsp-drawing, allowed the user to draw a line on a canvas, and the Y position of the line would be translated into a pitch. High notes were obtained by drawing on the top of the canvas, whereas low notes were obtained by drawing towards the bottom. Paint With Music is built on top of this prototype, and it uses the same principle; except it has more whales and birds!
The models
The power of Magenta is that it comes with great examples, snippets, and even pre-trained models that are ready to use! With that collaboration, the Google Arts & Culture team could use Magenta DDSP’s pre-trained models for 4 different instruments: violin, saxophone, trumpet and flute.
A canvas, some birds, and voilà!
To flesh out the experiment, some additional sounds and graphical elements were also added; elements of nature, such as whales and birds sounds. The graphics of Paint With Music are generated in canvas and are meant to replicate natural elements. The goal of these atmospheres was to render the experiment very easily understandable and intuitive for everyone. To enrich the sounds generated by DDSP, the javascript library tone.js was used.
Closing thoughts
Paint With Music was released in Summer 2021 and it has been very interesting to see what users did draw with it! Below are some of our favourite pieces shared by users.
The DDSP library holds a strong potential for experiments that tie the audio to the visual. Paint With Music is one interpretation of this principle, but it could be forever expanded and improved! For example, what if the user could draw with many strokes at once and create chords? What if they could change the strokes to create reverb or vibrato? The possibilities are endless and it is definitely a very fun and exciting field to explore!
Some examples of creations
Here is a gallery of examples made with our experiment. Enjoy!
Special thanks: Paint With Music would not have been possible without Jesse Engel, Ian Simon, Ben Tan, Edwin Toh, Asheley Gao, Bastien Girshig, and Aurora Straton.