Today, we’re excited to launch Space DJ, a web application from Magenta that turns music exploration into an interactive journey through a constellation of sounds. You pilot a spaceship through a galaxy where each star represents a musical genre. As you navigate this universe, Space DJ uses the Lyria RealTime API to generate a continuous stream of music that reflects your position and selections in real-time.

We used the deploy app feature in AI Studio to make this available to everyone!
Try Space DJ now, or view and fork the source code in AI Studio.

volume_off

Fly Through Music

  • Explore a Musical Universe: Fly through a star constellation where each star is labeled with a music genre. This galaxy is a 3D projection of genre embeddings.
  • Generate Music in Real-Time: As you fly, the stars close to the spaceship light up and influence the music. Clicking on a star or a point in space anchors your selection. The Lyria Realtime model blends the prompts of nearby genres into a unique musical mashup that evolves dynamically as you move.
  • Uncover Hidden Connections: Similar genres appear close together in the 3D space. You can also enable “High-Dimensional Neighbors” to find genres that are semantically similar in the original high-dimensional embedding space, even if they aren’t visual neighbors in the projection.
  • Engage Auto-Pilot: Randomly drift through space for an ever-changing, generative soundscape.

How it Works

Space DJ combines several technologies to create an immersive experience:

  • Genre Embeddings: We start with text prompts for 300 musical genres out of a 1000 genre dataset. The text is converted into a rich numerical representation (embedding) using the open-source MagentaRT model’s MusicCoca embedder. These 768-dimensional embeddings are then reduced to 128 dimensions using Principal Component Analysis for efficiency.
  • 3D Projection: To render the embeddings in 3D, we use Uniform Manifold Approximation and Projection (UMAP), an algorithm that projects the data into 3D space while trying to preserve its high-dimensional structure. You can tweak UMAP parameters in the settings for different constellation shapes.
  • Interactive Rendering: The 3D space, spaceship, and stars are rendered in your browser using three.js. You can select how many stars to create and whether to randomize the selection.
  • Real-Time Audio Synthesis: Your interactions within the 3D space are translated into a set of weighted text prompts (i.e. Deep House: 0.7, Ambient Techno: 0.3) based on proximity. These prompts are sent to the Lyria RealTime API, which synthesizes the music you hear, responding instantly to the spaceship’s position.
  • Development and Deployment: We used AI Studio to develop the applet through its interactive code editor. We leveraged AI Studio’s Cloud Run integration to deploy the application. This approach simplifies the deployment process and helps protect the Gemini API key by securely proxying requests to the Lyria RealTime API.

A New Frontier for Musical Interaction

Space DJ is an exploration into new ways of interacting with generative AI models for music. We hope to inspire new forms of musical expression and discovery.

Ready to take flight? Try Space DJ Now!