Tone Transfer lets you transform everyday sounds into musical instruments. Record and upload audio directly into the browser and hear our machine learning models re-render it into saxophones, flutes and more! Don’t fancy singing? Play around with a curated set of samples that will get your creative juices flowing!
Tone Transfer was born from a year-long collaboration between two teams within Google Research: Magenta and AIUX. AI Researchers, UX engineers and designers worked together to create an experience that opens up the magic of audio machine learning to a wider audience; from musicians to non-coders alike. Tone Transfer is built on a technology Magenta open-sourced earlier this year called Differentiable Digital Signal Processing or DDSP. At first, Magenta’s only demo was a technical colab notebook intended for folks with coding backgrounds. Through many iterations of design explorations and user research, the AIUX team developed and refined an experience that makes DDSP’s sound transformation approachable for everyone and more fun than ever to play with!
Tone Transfer is just the beginning. In making the leap from pure research to an approachable AI Experiment, we uncovered learnings on how people perceive music, machine learning, and their own practice, as well as designing flows that enable instruments from musical cultures beyond Western traditions to interact with the technology. On the technical side, Tone Transfer runs on an early version of DDSP in tensorflow.js, a fully in-browser deployment of DDSP that will become part of magenta.js. To extract pitch from audio, we use another model developed by Google Research, SPICE. We are excited with upcoming releases enabling you to easily train your own DDSP models and deploy them everywhere: a phone, an audio plugin or a website using the larger tensorflow lite and tensorflow.js ecosystem.
We can’t wait to hear what you create with Tone Transfer! Be sure to share with us using #ToneTransfer #MadeWithMagenta
Tone Transfer would not be possible without the help of many individuals within and beyond Google. Special thanks to: Doug Eck, Ricardo Prada, Katie Toothman, Sehmon Burnam, Kevin Malta, Divya Chandran, Ania Bogdanowicz, Hannes Widsomer, Ian Johnson, Ping Yu, David Zats, Hieu Dang and Greg Mikels.