One of the most interesting applications for Magenta models are real-time apps that have the user in the loop. Historically, most of the applications using our models have been “offline” – a musician would ask the model to do something, wait for a bit, then get back a result. While this is a useful approach for composing, one of the Magenta dreams has been a “jamming” app – one where you and a model play music together, at the same time.
Meet DrumBot, a web app that lets you play real-time music with a machine learning drummer that drums based on your melody.
In order for DrumBot to work seamlessly, it was really important to make sure that none of the data processing would block the audio of the musician playing. While Magenta.js lets us use models in the browser, it also allows them to be used in Node.js – on the server side. This means that we can do any of the expensive processing (that could block the main JavaScript thread) in the background, while the user is still playing music.
DrumBot uses the same GrooVAE model as the Magenta Studio Drumify plugin to generate drums that match a user’s melody. From then on…the world is your oyster. Jam away!
Play with DrumBot in your browser, or check out the code on GitHub.