
Overview
In the last decade, the size of digital music collections has increased dramatically. The capacity of MP3 Players has increased from a dozen songs to 40,000 songs or more. Online music stores offer millions of songs for sale at a dollar per song. Digital music subscription services offer unlimited access to millions of songs for a few dollars per month. Even though the size of music collections has grown, the tools offered to music consumers to find music have not changed much. Music consumers still browse by music genre or search for music by artist, album or song title just as they used to do in a record store. As music collections get larger it is getting a lot harder for people to find music, especially new music that they will like, using these primitive search tools. The goal of the 'Search Inside the Music' project is to explore new methods of categorizing, indexing and organizing large collections of music to allow more effective ways of searching through these collections. This project extends music search to search 'inside the music', that is, to search not just titles, keywords and artists, but to search music by music content and context. We want to help people find and organize their music based on all of of the properties of the music including such properties as acoustic similarity, mood, lyrics, musical theme, melody, tempo, rhythm, and instrumentation. We are currently focusing on two areas: using acoustic similarity to help people find music that 'sounds similar' to music that they already like, and using social data to recommend and organize music based upon the listening habits of people with similar musical tastes.Mapping Audio onto Words
One major goal of SITM is to build a machine learning model which can generate useful, descriptive words by "listening" to audio. The resulting word set can be used to measure similarity among songs and artists. Moreover, the words can be mixed with other descriptive word sets such as those garnered from social websites like last.fm.As an example consider an old version of a song I wrote 15 years ago, Keep the Change . (I use my music because the only time anyone listens to any of my music is when I link to it in scientific demos. Also my music was not used to build our models. Thus it's a fair test of the model.) Here are the most relevant results for genre and emotion ("emotion" here construed loosely):
| Genre words | Emotion words |
|---|---|
| 1 bluegrass | 1 beautiful |
| 2 irish | 2 sad |
| 3 slowcore | 3 gentle |
| 4 indie pop | 4 melancholy |
| 5 alt-country | 5 sexy |
| 6 americana | 6 relaxing |
Are these good? It's hard to say. Certainly we can find bad words a bit further down the list ("female vocalist"; I am offended!) but overall I'm pleased to see these words come out of our models.
So you want technical details? If you're really interested see these two papers:
- D. Eck, P. Lamere, T. Bertin-Mahieux, and S. Green. Automatic generation of social tags for music recommendation. In Neural Information Processing Systems Conference (NIPS) 20, 2007. [ bib | .pdf ]
- J. Bergstra, N. Casagrande, D. Erhan, D. Eck, and B. Kégl. Aggregate features and AdaBoost for music classification. Machine Learning, 65(2-3):473-484, 2006. [ bib | .pdf ]
You could also see my tech talk at Google on Automatically Tagging Audio Files Using Supervised Learning on Acoustic Features (Video)
Here's a one paragraph summary: We assign each word in our vocabulary its own machine learning model, specifically a large-margin ensemble learner (AdaBoost or FilterBoost) which maps features from 5-second segments of audio onto a label such as americana. During training we select positive examples for a given word from our audio database. Positive examples are decided via data mining results. In other words, we use the words from social taggers to generate our machine learning training sets. Specifically, the positive examples for say, the word americana is drawn from the songs most labeled americana by last.fm users. We select the audio for these songs from our lab database and train a classifier to find the relevant features in the audio necessary to "hear" whether a song is americana or not. Our training is done over 5-second segments of audio. To label an entire song, we take the average prediction over all 5 second segments in the song for a particular word. By training lots of individual word models we are able to build a set of relevant words for a song, album or artist. To summarize : a set of non-linear classifiers take audio as input and generate relevant words as output.
Below are the features used to train the model. Very broadly speaking
they are features sensitive to rhythm and meter (autocorrelation;
top), pitch (spectrum; middle), and musical timbre / instrumentation
(cepstrum; bottom)

Autotag similarity
ta We use the term "autotag" to describe one of the words generated by our machine learning models. Once we have generated a set of autotags for a music collection, we can use them as a means to measure song, album and artist similarity. Because autotags are words like any other word, we can easily blend autotags with evidence from other sources of data such as last.fm, wikipedia, etc.
Here we offer a few examples of what it sounds like to navigate the resulting space of similar artists. Thierry applied a dimensionality reduction technique called Isomap to create a nearest neighbor graph of artists. We could then find the shortest path joining two artists. I sampling 5-second segments from songs by these artists and chained them together in an Mp3 file. These demos are a bit rough because I sampled 5 seconds randomly from a random song selected from each artist. There are many better ways to do this by smartly sampling the song and also smartly choosing the 5 seconds. Click on graphics for larger versions.
![]() Beethoven to The Prodigy. |
![]() John Coltrane to System of a Down. |
![]() Mozart to Nirvana. |
People and Places
Search Inside the Music is a project of Sun Labs, Boston. I worked full time for six months on the project in 2007 and am now back at University of Montreal as associate professor. The project is still under active development, with (broadly speaking) the machine learning for music work being done in my lab and everything else, including the development of an open source recommendation framework, being done at Sun Labs. Here are some of the people involved in SITM:Current Team
- Paul Lamere is a Staff Engineer at Sun Labs, Boston and is the Principal Investigator of the project.
- Douglas Eck was a Visiting Professor at Sun Labs in 2007 and has since returned to University of Montreal. He remains involved in the project on the level of machine learning algorithm development.
- François Maillet was an intern at Sun Labs in Summer 2008 and currently a Master's student at University of Montreal with Douglas Eck.
We collaborate closely with the Advanced Search Technology group in Sun Labs that includes:
- Steve Green, Senior Staff Engineer at Sun Labs, Burlington, MA and Principal Investigator of the Advanced Search Technology group.
- Jeff Alexander, Sun Labs, Burlington, MA and member of the Advanced Search Technology group.
Alumni
- Sten Anderson is an independent contracto who contributed to the 3D visualizations of the music space.
- Chris West an intern at Sun Labs in 2005.
- Rebecca Fiebrink was an intern at Sun Labs in 2006 and now a Ph.D student at Princeton with Perry Cook.
- Thierry Bertin-Mahieux was an intern at Sun Labs in Summer 2007 and about to finish his Master's at University of Montreal with Douglas Eck.


