Is it a horror movie or a rom-com? AI can predict primarily based solely on music

Music is an indispensable factor in movie: it establishes environment and temper, drives the viewer’s emotional reactions, and considerably influences the viewers’s interpretation of the story.

In a current paper revealed in PLOS ONE, a analysis crew on the USC Viterbi School of Engineering, led by Professor Shrikanth Narayanan, sought to objectively look at the impact of music on cinematic genres. Their research aimed to find out if AI-based know-how might predict the genre of a movie primarily based on the soundtrack alone.

“By better understanding how music affects the viewer’s perception of a film, we gain insights into how film creators can reach their audience in a more compelling way,” mentioned Narayanan, University Professor and Niki and Max Nikias Chair in Engineering, professor {of electrical} and computer engineering and computer science and the director of USC Viterbi’s Signal Analysis and Interpretation Laboratory (SAIL).

The notion that completely different movie genres are extra seemingly to make use of sure musical components of their soundtrack is slightly intuitive: a lighthearted romance would possibly embrace wealthy string passages and luxurious, lyrical melodies, whereas a horror film would possibly as an alternative function unsettling, piercing frequencies and eerily discordant notes.

But whereas previous work qualitatively signifies that completely different movie genres have their very own units of musical conventions—conventions that make that romance movie sound completely different from that horror film—Narayanan and crew got down to discover quantitative proof that components of a movie’s soundtrack might be used to characterize the movie’s style.

Narayanan and crew’s research was the primary to use deep learning models to the music utilized in a movie to see if a pc might predict the style of a movie primarily based on the soundtrack alone. They discovered that these fashions had been in a position to precisely classify a movie’s style utilizing machine studying, supporting the notion that musical options could be highly effective indicators in how we understand completely different movies.

According to Timothy Greer, Ph.D. pupil at USC Viterbi within the division of pc science who labored with Narayanan on the research, their work might have worthwhile purposes for media companies and creators in understanding how music can improve different types of media. It might give manufacturing firms and music supervisors a greater understanding of the right way to create and place music in tv, films, ads, and documentaries in an effort to elicit sure feelings in viewers.

In addition to Narayanan and Greer, the analysis crew for the research included Dillon Knox, a Ph.D. pupil within the division {of electrical} and pc engineering, and Benjamin Ma, who graduated from USC in 2021 with a B.S. in pc science, a grasp’s in pc science, and a minor in music manufacturing. (Ma was additionally named one of many two 2021 USC Schwarzman Scholars.) The crew labored throughout the Center for Computational Media Intelligence, a analysis group in SAIL.

Predicting style from soundtrack

In their research, the group examined a dataset of 110 standard movies launched between 2014 and 2019. They used style classification listed on the Internet Movie Database (IMDb), to label every movie as motion, comedy, drama, horror, romance, or science-fiction, with lots of the films spanning multiple of those genres.

Next, they utilized a deep studying community that extracted the auditory info, like timbre, concord, melody, rhythm, and tone from the music and rating of every movie. This community used machine studying to research these musical options and proved able to precisely classifying the style of every movie primarily based on these options alone.

The group additionally interpreted these fashions to find out which musical options had been most indicative of variations between genres. The fashions did not give specifics as to which kinds of notes or devices had been related to every style, however they had been in a position to set up that tonal and timbral options had been most vital in predicting the movie’s style.

“Laying this groundwork is really exciting because we can now be more precise in the kinds of questions that we want to ask about how music is used in film,” mentioned Knox. “The overall film experience is very complicated and being able to computationally analyze its impact and the choices and trends that go into its construction is very exciting.”

Future instructions

Narayanan and his crew examined the auditory info from every movie utilizing a know-how referred to as audio fingerprinting, the identical know-how that allows companies like Shazam to determine songs from a database by listening to recordings, even when there are sound results or different background noise current. This know-how allowed them to take a look at the place the musical cues occur in a movie and for a way lengthy.

“Using audio fingerprinting to listen to all of the audio from the film allowed us to overcome a limitation of previous film music studies, which usually just looked at the film’s entire soundtrack album without knowing if or when songs from the album appear in the film,” mentioned Ma. In the longer term, the group is inquisitive about making the most of this functionality to check how music is utilized in particular moments in a movie and the way musical cues dictate how the narrative of the movie evolves over its course.

“With the ever-increasing access to both film and music, it has never been more crucial to quantitatively study how this media affects us,” Greer mentioned. “Understanding how music works in conjunction with other forms of media can help us devise better viewing experiences and make art that’s moving and impactful.”