Exploring visual directionality as a cross-modal cue in classical music
Visual representations of music are widely used in notation, animation, and pedagogical tools, especially in video-based media. While various approaches have explored relationships between sound and visual phenomena, including color-based and synesthetic systems, relatively little work has examined how visualizations grounded in musically authentic elements, such as performers’ movements or instrument mechanics, might illuminate structural features of classical art music or support listeners’ understanding of complex textures. This raises the question of how visually grounded cues can be designed to make musical structure more perceptible, particularly for audiences with limited experience of classical repertoire. To address this, we investigated whether directional visual movement could mark musical phrases. We selected video segments whose motion aligned with phrase boundaries and determined how these movements should traverse the screen to convey progression and closure. Treating the design space as open, rather than assuming a single correct mapping, allowed us to explore alternative visual strategies empirically. We created paired visualizations based on excerpts from Claude Debussy’s Images (Books 1 and 2), whose richly layered textures provide an ecologically valid test case. In one version, visual gestures moved rightward at phrase onset and leftward at phrase end; in a flipped version, these directions were reversed. This enabled us to assess whether viewers shared directional expectations when interpreting musical motion and how they reasoned about links between movement and musical form. Although most excerpts produced no uniform preference, one passage elicited a clear directional choice among participants. These results suggest that carefully designed visual cues can highlight structural features in complex polytextural music, while remaining sensitive to local musical context.
- Fröst J, Falkenberg K. Music as Script: A Novel Approach to Visualizing Musical Structure with Creative Reinterpretation. In: Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Springer Nature Switzerland; 2026:38-52. doi: 10.1007/978-3-032-26999-7_3
- Spelke E. Infants’ intermodal perception of events. Cogn Psychol. 1976; 8(4):553–560. doi: 10.1016/0010-0285(76)90018-9
- Platz F, Kopiez R. When the eye listens: A meta-analysis of how audiovisual presentation enhances the appreciation of music performance. Music Percept. 2012; 30(1):71–83. doi: 10.1525/mp.2012.30.1.71
- Petzold B, Zaeh MF, Faerber B, et al. A Study on Visual, Auditory, and Haptic Feedback for Assembly Tasks. Presence. 2004; 13(1):16–21. doi: 10.1162/105474604774048207
- Sigrist R, Rauter G, Riener R, Wolf P. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychon Bull Rev. 2013; 20(1):21–53. doi: 10.3758/s13423-012-0333-8
- Joy A, Sherry Jr, JF. Speaking of art as embodied imagination: A multisensory approach to understanding aesthetic experience. J Consum Res. 2003; 30(2):259–282. doi: 10.1086/376802
- Bigand E, Lalitte P, Tillmann B. Learning music: prospects about implicit knowledge in music, new technologies and music education. In: Sound to Sense, Sense to Sound. A State of the Art in Sound and Music Computing. Berlin: Logos Verlag; 2008:47–81. ISBN: 9783832516000. https://smcnetwork.org/s2s2.pdf
- Isaacson EJ. What you see is what you get: On visualizing music. In: ISMIR. 2005:389–395. Accessed XXX. https://ismir2005.ismir.net/proceedings/1129.pdf
- Lima HB, Santos CGD, Meiguins BS. A survey of music visualization techniques. ACM Comput Surv. 2021; 54(7):1–29. doi: 10.1145/3461835
- Jensenius AR. Sound Actions: Conceptualizing Musical Instruments. MIT Press; 2022. ISBN: 9780262544634. https://direct.mit.edu/books/book-pdf/2234810/book_9780262372206.pdf
- Morikawa K, McBeath M. Lateral motion bias associated with reading direction. Vision Res. 1992; 32(6): 1137–1141. doi: 10.1016/0042-6989(92)90014-A
- Rusconi E, Kwan B, Giordano BL, Umilta C, Butterworth B. Spatial representation of pitch height: the SMARC effect. Cognition. 2006; 99(2):113–129. doi: 10.1016/j.cognition.2005.01.004
- Spence C. Crossmodal correspondences: A tutorial review. Atten Percept Psychophys. 2011; 73(4):971–995. doi: 10.3758/s13414-010-0073-7
- Godøy RI, Leman M. Musical Gestures: Sound, Movement, and Meaning. Routledge; 2010. ISBN: 9781135183639. https://www.routledge.com/Musical-Gestures-Sound-Movement-and-Meaning/Godoy-Leman/p/book/9780415998871
- Porac C, Coren S. Lateral Preferences and Human Behavior. Springer New York, NY, 1981. doi: 10.1007/978-1-4613-8139-6
- Foulsham T, Fröst E, Sage L. Stable individual differences predict eye movements to the left, but not handedness or line bisection. Vision Res. 2018; 144:38–46. doi: 10.1016/j.visres.2018.02.002
- Kocka L. Left or Right? Directing Lateral Movement in Film. Vernon Press; 2021. ISBN: 9781648891885. https://vernonpress.com/book/1012
- Egizii ML, Denny J, Neuendorf KA, Skalski PD, Campbell R. Which way did he go? Directionality of film character and camera movement and subsequent spectator interpretation. In: International Communication Association conference, Phoenix, AZ. 2012. https://www.researchgate.net/profile/Kimberly-Neuendorf/publication/228448619
- Hertz R. La prééminence de la main droite: étude sur la polarité religieuse. Rev Philos France Let. 1909; 68:553–580. https://www-jstor.org/stable/41079022
- Eitan Z, Granot RY. How music moves: Musical parameters and listeners’ images of motion. Music Percept. 2006; 23(3):221–248. doi: 10.1525/mp.2006.23.3.221
- Hidaka S, Manaka Y, Teramoto W, et al. Alternation of sound location induces visual motion perception of a static object. PLoS ONE. 2009; 4(12):e8188. doi: 10.1371/journal.pone.0008188
- Pankhurst T. SchenkerGUIDE: A Brief Handbook and Website for Schenkerian Analysis. Routledge; 2008. ISBN: 9781135871031. Accessed XXX. https://www.routledge.com/SchenkerGUIDE-A-Brief-Handbook-and-Website-for-Schenkerian-Analysis/Pankhurst/p/book/9780415973984
- Malawey V. Review of Open Music Theory Version 2 by Mark Gotham; Kyle Gullings; Chelsey Hamm; Bryn Hughes; Brian Jarvis; Megan Lavengood; and John Peterson. Journal of Music Theory Pedagogy. 2023; 37(1):6. doi: 10.71156/2994-7073.1442
- Dai S, Zhang H, Dannenberg RB. The interconnections of music structure, harmony, melody, rhythm, and predictivity. Music Sci. 2024; 7:20592043241234758. doi: 10.1177/20592043241234758
- Huron D. Sweet Anticipation. Published online 2006. doi: 10.7551/mitpress/6575.001.0001
- Huron DB. Voice Leading: The Science Behind a Musical Art. MIT Press; 2016. ISBN: 9780262335454. https://mitpress.mit.edu/9780262537384/voice-leading/
- Karwoski TF, Odbert HS. Color-music. Psychological Monographs. 1938;50(2):i-60. doi: 10.1037/h0093458
- Thompson KM. Optical poetry: The life and work of Oskar Fischinger. Film Q. 2005; 59(1):65. doi: 10.1525/fq.2005.59.1.65
- Rogers H. The musical script: Norman McLaren, animated sound, and audiovisuality. Animation. 2014; 22:68–84. https://www.academia.edu/11957147
- Mulina RF. Music Animation Machine Demonstration Reel by Stephen Malinowski. Leonardo (Oxf). 1991; 24(5):632. https://muse.jhu.edu/pub/6/article/606836
- Ciufo T. Real-time sound/image manipulation and mapping in a performance setting. In: Proc. MAXIS Festival of Sound and Experimental Music. 2002. Accessed XXX. https://ciufo.org/media/sound_image.pdf
- Muriago A. Re-envisioning the Piano Recital: An Audiovisual Alternative [Doctoral Dissertation]. Connecticut, United States: University of Hartford; 2017. https://www.proquest.com/openview/4bcef2f2a86c37a4138398bc135ea0a6/
- Greckel W. Visualization in the performance of classical music: A new challenge. Visions of Research in Music Education. 2021; 16:35. https://opencommons.uconn.edu/vrme/vol16/iss3/35
- Björkman M, Jakobsson K. Switched-On Malmö Live: Exploring directions for the future of classical music streaming experiences. In: LUP Student Papers. Lund, Sweden, 2024. https://lup.lub.lu.se/student-papers/search/publication/9159100
- Debussy C. Images: 1re série pour piano à 2 mains. Paris: Durand & Fils; 1905. https://books.google.se/books/about/Images.html?id=5rki0gEACAAJ&redir_esc=y
- Debussy C. Images: 2e série pour piano seul. Paris: Durand & Fils; 1908. https://books.google.se/books/about/Images.html?id=xu4i0gEACAAJ&redir_esc=y
- Elliot AJ, Maier MA. Color psychology: Effects of perceiving colour on psychological functioning in humans. Annu Rev Psychol. 2014; 65(1):95–120. doi: 10.1146/annurev-psych-010213-115035
- Kliuchko M, Heinonen-Guzejev M, Monacis L, et al. The association of noise sensitivity with music listening, training, and aptitude. Noise Health. 2015; 17(78):350–357. doi: 10.4103/1463-1741.165065
- Tran US, Stieger S, Voracek M. Evidence for general right-, mixed-, and left-sidedness in self-reported handedness, footedness, eyedness, and earedness, and a primacy of footedness in a large-sample latent variable analysis. Neuropsychologia. 2014; 62:220–232. doi: 10.1016/j.neuropsychologia.2014.07.027
