Abstract

Turning a photograph into a piece of music is an intriguing but difficult cross-modal task. Most existing image-to-music systems squeeze the entire image into one global feature vector and then hand that single vector to a music generator. The trouble is that this flattening step throws away the spatial layout of the scene. A guitarist in a quiet meadow gets the same treatment as an empty meadow, because the global vector averages everything together. We present SC- M2U, a framework that splits the image into foreground and background regions, encodes each region separately through a shared CLIP ViT-B/32 backbone, and routes the resulting embeddings to distinct music stems through a gated cross-modal attention module. The foreground drives the lead melody track while the background shapes the accompaniment. We evaluate SC-M2U on five diverse visual scenes using cosine similarity alignment, ablation over five crop ratios, and a cross-scene confusion analysis. On average, the matched region– track alignment reaches 0.2411, which sits above the 0.2387 of the global baseline, with a positive selectivity contrast ?? of +0.001 and a diagonal dominance of +0.0213 in the confusion matrix. These numbers are modest under frozen CLIP weights, but they show that spatially aware conditioning tightens the link between what is seen and what is heard, and they give a concrete starting point for controllable, multi-stem image-to-music generation.

Authors

Gurpreet Singh
Universiti Putra Malaysia, Malaysia

Keywords

Image-to-Music Generation, Cross-Modal Alignment, Spatial Decomposition, Gated Attention, Multi-Track Music Generation