Video-based Symbolic Music Generation

As more people publish video content on platforms such as YouTube, incorporating engaging soundtracks has become an important challenge. Current alternatives have significant drawbacks. Using royalty-free music requires time and effort to find a suitable piece, and many creators unknowingly use the same tracks, limiting novelty. Composing a soundtrack requires musical expertise, while hiring professionals is costly. Purchasing a suitable piece of music also incurs expenses and requires the effort to find an appropriate track.

This project addresses this issue. I develop a two-stage model, employing deep neural networks (DNNs), to analyze a given video, and then to automatically compose a suitable soundtrack. The soundtracks fits the video in two aspects. Both music and video conduct similar emotions, and they both have concurrent temporal boundaries.

In the video analysis stage, the model classifies the emotions of the video. For more details, check out my video project. The scene boundaries of the video are also extracted.

In my previous project, I developed an emotion-based symbolic music generator. The emotions extracted from the video drive music generator.

Secondly, the music model generates musical boundaries in the form of long chords, near the video scene boundaries, while maintaining the rhythm and harmony. This is made possible by a novel mechanism that incorporates “boundary offsets” into the transformer architecture, enabling the model to anticipate upcoming boundaries. The relevant paper is in review (Sulun et al., 2025), and the code and the demo will be released soon.

References

Video Soundtrack Generation by Aligning Emotions and Temporal Boundaries

Serkan Sulun, Paula Viana, and Matthew E. P. Davies

In revision, Feb 2025

Abs DOI Bib PDF

We introduce EMSYNC, a video-based symbolic music generation model that aligns music with a video’s emotional content and temporal boundaries. It follows a two-stage framework, where a pretrained video emotion classifier extracts emotional features, and a conditional music generator produces MIDI sequences guided by both emotional and temporal cues. We introduce boundary offsets, a novel temporal conditioning mechanism that enables the model to anticipate and align musical chords with scene cuts. Unlike existing models, our approach retains event-based encoding, ensuring fine-grained timing control and expressive musical nuances. We also propose a mapping scheme to bridge the video emotion classifier, which produces discrete emotion categories, with the emotion-conditioned MIDI generator, which operates on continuous-valued valence-arousal inputs. In subjective listening tests, EMSYNC outperforms state-of-the-art models across all subjective metrics, for music theory-aware participants as well as the general listeners.
@article{soundtrack, title = {Video Soundtrack Generation by Aligning Emotions and Temporal Boundaries}, author = {Sulun, Serkan and Viana, Paula and Davies, Matthew E. P.}, year = {2025}, month = feb, number = {arXiv:2502.10154}, eprint = {2502.10154}, primaryclass = {cs}, publisher = {arXiv}, doi = {10.48550/arXiv.2502.10154}, urldate = {2025-02-24}, archiveprefix = {arXiv}, journal = {In revision}, }