Get Started Demos Blog Research Community

Full list of publications

Here is a link to all of our research publications.

Datasets

As part of the project, we open source some of the datasets that were used in our research.

Research highlights

Wave2Midi2Wave

A new process able to transcribe, compose, and synthesize audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s).

Music VAE

A hierarchical latent vector model for learning long-term structure in music

Onsets and Frames

We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames.

Latent Constraints

A method to condition generation without retraining the model, by post-hoc learning latent constraints, value functions that identify regions in latent space that generate outputs with desired attributes. We can conditionally sample from these regions with gradient-based optimization or amortized actor functions.

COCONET

An instance of orderlessNADE, Coconet uses deep convolutional neural networks to perform music inpaintings through Gibbs sampling.

Performance RNN

An LSTM-based recurrent neural network designed to model polyphonic music with expressive timing and dynamics.

Sketch RNN

A recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes.

overview of Sketch RNN

Papers

Blog Posts

NSynth

A powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform.

overview of NSynth

Papers

Blog Posts

Colab Notebooks