Editorial Note: In August of 2020, Magenta partnered with Gray Area to host BitRate: a month-long series focused on experimenting with the possibilities of Music and Machine Learning. Here is a blog post from one of our top projects, explaining their project in more detail. Learn more about BitRate and view additional submissions on the event page.
In 2020, journaling has helped me survive. Playing music, singing, and writing songs has helped too, especially when life’s tragedies seem to overwhelm me. As a guitar coach, I teach my students how to get in touch with our shared musical sides. In my personal life, I write songs in part to help me process difficult emotions. These two worlds inspire questions I’ve been pondering lately: how can we support more people to feel the healing power of creating music? How can we lead them into deeper communion with their emotional life? How can we inspire them to learn more about themselves and their community?
Inspired by these questions, Stephen Haney, Suyash Joshi, and I (Devin Lane) built Dear Diary (GitHub repo here, it’s open source). Type some words—thoughts, feelings, poems, goals, stories, a to-do list—and you’ve created an original song. Even if you don’t consider yourself musical or creative, you can use journaling to dive deeper into your emotional state.
Open up DearDiary.ai and try it yourself. You might consider starting with the classic “Hello, world!”
You’ll notice that music plays as you type. You’ve created a song! You’re well on your way to musical stardom! As you delete letters, a chromatically descending scale plays; when you write punctuation marks, a satisfying chord plays. You can replay the song and share a unique link of your creation with others (entries are public to anyone with your link). Meanwhile, comforting sounds of birds and wind play in the background.
What else can you do with Dear Diary? You can set music to a story or poem you’ve written and create a cinematic audio/visual experience. You can add emotional resonance to something you’re feeling to share with someone you love. You can use the music of Dear Diary to mindfully pay attention to your feelings as you write and listen. While this is all inherently beneficial and enjoyable, research also shows impressive mental health benefits of both journaling and creating music.
How does Dear Diary work?
As you type, the Natural Language Processing library VADER.js analyzes the sentiment of your writing, giving it a score along a “happy” to “sad” spectrum. We then plug this sentiment analysis score into the MusicVAE tools by Magenta to interpolate between two pre-composed melodies.
What does it mean to interpolate between two melodies? MusicVAE —a machine learning model developed by the Magenta team—blends two musical inputs together, creating new melodies that “live between” the original two sounds. These new melodies could be thought of as the weighted averages between two points on a spectrum.
Here you can listen to the original happy and sad melodies I composed that form the foundation of music made with Dear Diary.
To summarize: VADER.js gives your phrases a sentiment score, which we feed into MusicVAE to create a new melody that exists somewhere on the spectrum between the pre-composed “happy” and “sad” melodies. Thus, you have a completely original song for every DearDiary entry!
Response from the public
Our project was featured on the front page of Product Hunt, which resulted in lots of inspiring user feedback. We heard excitedly from some who felt newly empowered to make music, such as Courtney above; others such as Kağan felt relief. Yet others used Dear Diary to share and process very powerful messages about their loved ones, such as this entry (shared with permission; TW: COVID-19).
What were some limitations and challenges of the project?
Achieving authentic, diverse emotional analysis and musically pleasing results remain challenging for creative AI projects such as Dear Diary. Our initial goal was to implement a multidimensional emotion ranking system, similar to something a mental health professional might use. However, within the roughly month-long timeframe of this project, we did not find a satisfactory open source solution that could run in the browser.
Additionally, achieving musical-sounding results proved challenging. Initially, we worked to create very believable happy and sad musical examples for the two poles of our spectrum to feed into MusicVAE. However, the interpolated melodies that resulted from use testing felt dissonant, tense, or difficult to classify emotionally, rather than happy or sad. We eventually arrived at the final result by creating happy and sad examples that were closely related musically, so that the resultant music created by Dear Diary would sound nice to the listener.
What are some potential new features for projects such as Dear Diary?
In our next steps, we would like to include more emotions, as described above. Future versions could also include more instruments such as guitars, violins, and sitars, as well as timbre transfer between instruments using DDSP. We would implement an audio mixer to control the volume and panning between these different musical tracks and the sound design.
Visually, this project could benefit from text to image using a GAN or CPPN, or drawing using SketchRNN.
Final Thoughts
We believe that projects like Dear Diary leverage the power of AI to enhance the human capacity to feel and express our emotions. Serving as a model for future tools, Dear Diary infuses our technological experience with creativity, music, and mindfulness.
We’d love for you to try it out and share your creations with us on Twitter with the #deardiaryAI and #madewithmagenta tags.
Special thanks to the Magenta and Gray Area teams for hosting the Bitrate hackathon. We had a great time and hope you enjoy our product.