Update (11/30/18): Piano Genie has been added to magenta.js, and has a shiny new web demo you can play with!

We introduce Piano Genie, an intelligent controller that maps 8-button input to a full 88-key piano in real time:

Piano Genie is in some ways reminiscent of video games such as Rock Band and Guitar Hero that are accessible to novice musicians, with the crucial difference that users can freely improvise on Piano Genie rather than re-enacting songs from a fixed repertoire. You can try it out yourself via our interactive web demo!

How does it work?

There are many ways one could map 8-button controller sequences to full piano performances. We restrict ourselves to 1-to-1 mappings between button presses and notes, giving the user precise control over timing and degree of polyphony but not which notes are played. Even given that restriction, there are many possible mappings; for example, the 8 buttons could map to a fixed scale over a single octave. Instead of using such a fixed mapping, we learn a time-varying mapping using a discrete autoencoder architecture trained on a set of existing piano performances:

A bidirectional LSTM encoder maps a sequence of piano notes to a sequence of controller buttons (shown as 4 in the above figure, 8 in the actual system). A unidirectional LSTM decoder then decodes these controller sequences back into piano performances. After training, the encoder is discarded and controller sequences are provided by user input.

You may have noticed in the above demo video that pitch contours in the piano performance closely mimic the contours of the button sequence. Such behavior is encouraged by a training loss term that penalizes the encoder for violating relative pitch ordering, e.g. if an ascending piano interval maps to a descending button interval. The below image shows (top) a pianoroll for a real piano performance, and (bottom) the 8-button sequence output by the contour-regularized encoder; you can see that the contours of the two sequences match closely:

We trained the model on a set of ~1400 virtuosic performances from the International Piano-e-Competition, the same dataset used in Performance RNN. This choice of dataset naturally affects the style of the generated performances; as we have open-sourced the model (see below), you are welcome to train Piano Genie on your own set of MIDI files.

Try it out!

You can try Piano Genie yourself with our web demo. For more details, check out our arXiv paper. The web demo code or model training code. Additional improvisation videos are available here.