AccScience Publishing / AC / Online First / DOI: 10.36922/AC026130015
Cite this article
2
Download
70
Views
Related Info Links
More by Authors Links
Journal Browser
Volume | Year
Issue
Search
News and Announcements
View All
ORIGINAL RESEARCH ARTICLE

Exploring visual directionality as a cross-modal cue in classical music

Johan Fröst1,2* ,  Kjetil Falkenberg2
Show Less
1 Department of Classical Music, Royal College of Music (KMH), Stockholm , Sweden
2 Media Technology and Interaction Design, KTH Royal Institute of Technology, Stockholm , Sweden
Received: 25 March 2026 | Revised: 22 September 2026 | Accepted: 21 October 2026 | Published online: 20 November 2026
© 2026 by the Author(s). This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution -Noncommercial 4.0 International License (CC-by the license) ( https://creativecommons.org/licenses/by-nc/4.0/ )
Abstract

Visual representations of music are widely used in notation, animation, and pedagogical tools, especially in video-based media. While various approaches have explored relationships between sound and visual phenomena, including color-based and synesthetic systems, relatively little work has examined how visualizations grounded in musically authentic elements, such as performers’ movements or instrument mechanics, might illuminate structural features of classical art music or support listeners’ understanding of complex textures. This raises the question of how visually grounded cues can be designed to make musical structure more perceptible, particularly for audiences with limited experience of classical repertoire. To address this, we investigated whether directional visual movement could mark musical phrases. We selected video segments whose motion aligned with phrase boundaries and determined how these movements should traverse the screen to convey progression and closure. Treating the design space as open, rather than assuming a single correct mapping, allowed us to explore alternative visual strategies empirically. We created paired visualizations based on excerpts from Claude Debussy’s Images (Books 1 and 2), whose richly layered textures provide an ecologically valid test case. In one version, visual gestures moved rightward at phrase onset and leftward at phrase end; in a flipped version, these directions were reversed. This enabled us to assess whether viewers shared directional expectations when interpreting musical motion and how they reasoned about links between movement and musical form. Although most excerpts produced no uniform preference, one passage elicited a clear directional choice among participants. These results suggest that carefully designed visual cues can highlight structural features in complex polytextural music, while remaining sensitive to local musical context.

Keywords
Directional movement
Visualization
Art music
Visual music representation
Musical phrasing
Funding
The authors gratefully acknowledge financial support from the Royal College of Music in Stockholm (Kungl. Musikhögskolan, KMH), Sweden, which supported the research and dissemination of this work.
Conflict of interest
The authors declare they have no competing interests.
References
  1. Fröst J, Falkenberg K. Music as Script: A Novel Approach to Visualizing Musical Structure with Creative Reinterpretation. In: Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Springer Nature Switzerland; 2026:38-52. doi: 10.1007/978-3-032-26999-7_3
  2. Spelke E. Infants’ intermodal perception of events. Cogn Psychol. 1976; 8(4):553–560. doi: 10.1016/0010-0285(76)90018-9
  3. Platz F, Kopiez R. When the eye listens: A meta-analysis of how audiovisual presentation enhances the appreciation of music performance. Music Percept. 2012; 30(1):71–83. doi: 10.1525/mp.2012.30.1.71
  4. Petzold B, Zaeh MF, Faerber B, et al. A Study on Visual, Auditory, and Haptic Feedback for Assembly Tasks. Presence. 2004; 13(1):16–21. doi: 10.1162/105474604774048207
  5. Sigrist R, Rauter G, Riener R, Wolf P. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychon Bull Rev. 2013; 20(1):21–53. doi: 10.3758/s13423-012-0333-8
  6. Joy A, Sherry Jr, JF. Speaking of art as embodied imagination: A multisensory approach to understanding aesthetic experience. J Consum Res. 2003; 30(2):259–282. doi: 10.1086/376802
  7. Bigand E, Lalitte P, Tillmann B. Learning music: prospects about implicit knowledge in music, new technologies and music education. In: Sound to Sense, Sense to Sound. A State of the Art in Sound and Music Computing. Berlin: Logos Verlag; 2008:47–81. ISBN: 9783832516000. https://smcnetwork.org/s2s2.pdf
  8. Isaacson EJ. What you see is what you get: On visualizing music. In: ISMIR. 2005:389–395. Accessed XXX. https://ismir2005.ismir.net/proceedings/1129.pdf
  9. Lima HB, Santos CGD, Meiguins BS. A survey of music visualization techniques. ACM Comput Surv. 2021; 54(7):1–29. doi: 10.1145/3461835
  10. Jensenius AR. Sound Actions: Conceptualizing Musical Instruments. MIT Press; 2022. ISBN: 9780262544634. https://direct.mit.edu/books/book-pdf/2234810/book_9780262372206.pdf
  11. Morikawa K, McBeath M. Lateral motion bias associated with reading direction. Vision Res. 1992; 32(6): 1137–1141. doi: 10.1016/0042-6989(92)90014-A
  12. Rusconi E, Kwan B, Giordano BL, Umilta C, Butterworth B. Spatial representation of pitch height: the SMARC effect. Cognition. 2006; 99(2):113–129. doi: 10.1016/j.cognition.2005.01.004
  13. Spence C. Crossmodal correspondences: A tutorial review. Atten Percept Psychophys. 2011; 73(4):971–995. doi: 10.3758/s13414-010-0073-7
  14. Godøy RI, Leman M. Musical Gestures: Sound, Movement, and Meaning. Routledge; 2010. ISBN: 9781135183639. https://www.routledge.com/Musical-Gestures-Sound-Movement-and-Meaning/Godoy-Leman/p/book/9780415998871
  15. Porac C, Coren S. Lateral Preferences and Human Behavior. Springer New York, NY, 1981. doi: 10.1007/978-1-4613-8139-6
  16. Foulsham T, Fröst E, Sage L. Stable individual differences predict eye movements to the left, but not handedness or line bisection. Vision Res. 2018; 144:38–46. doi: 10.1016/j.visres.2018.02.002
  17. Kocka L. Left or Right? Directing Lateral Movement in Film. Vernon Press; 2021. ISBN: 9781648891885. https://vernonpress.com/book/1012
  18. Egizii ML, Denny J, Neuendorf KA, Skalski PD, Campbell R. Which way did he go? Directionality of film character and camera movement and subsequent spectator interpretation. In: International Communication Association conference, Phoenix, AZ. 2012. https://www.researchgate.net/profile/Kimberly-Neuendorf/publication/228448619
  19. Hertz R. La prééminence de la main droite: étude sur la polarité religieuse. Rev Philos France Let. 1909; 68:553–580. https://www-jstor.org/stable/41079022
  20. Eitan Z, Granot RY. How music moves: Musical parameters and listeners’ images of motion. Music Percept. 2006; 23(3):221–248. doi: 10.1525/mp.2006.23.3.221
  21. Hidaka S, Manaka Y, Teramoto W, et al. Alternation of sound location induces visual motion perception of a static object. PLoS ONE. 2009; 4(12):e8188. doi: 10.1371/journal.pone.0008188
  22. Pankhurst T. SchenkerGUIDE: A Brief Handbook and Website for Schenkerian Analysis. Routledge; 2008. ISBN: 9781135871031. Accessed XXX. https://www.routledge.com/SchenkerGUIDE-A-Brief-Handbook-and-Website-for-Schenkerian-Analysis/Pankhurst/p/book/9780415973984
  23. Malawey V. Review of Open Music Theory Version 2 by Mark Gotham; Kyle Gullings; Chelsey Hamm; Bryn Hughes; Brian Jarvis; Megan Lavengood; and John Peterson. Journal of Music Theory Pedagogy. 2023; 37(1):6. doi: 10.71156/2994-7073.1442
  24. Dai S, Zhang H, Dannenberg RB. The interconnections of music structure, harmony, melody, rhythm, and predictivity. Music Sci. 2024; 7:20592043241234758. doi: 10.1177/20592043241234758
  25. Huron D. Sweet Anticipation. Published online 2006. doi: 10.7551/mitpress/6575.001.0001
  26. Huron DB. Voice Leading: The Science Behind a Musical Art. MIT Press; 2016. ISBN: 9780262335454. https://mitpress.mit.edu/9780262537384/voice-leading/
  27. Karwoski TF, Odbert HS. Color-music. Psychological Monographs. 1938;50(2):i-60. doi: 10.1037/h0093458
  28. Thompson KM. Optical poetry: The life and work of Oskar Fischinger. Film Q. 2005; 59(1):65. doi: 10.1525/fq.2005.59.1.65
  29. Rogers H. The musical script: Norman McLaren, animated sound, and audiovisuality. Animation. 2014; 22:68–84. https://www.academia.edu/11957147
  30. Mulina RF. Music Animation Machine Demonstration Reel by Stephen Malinowski. Leonardo (Oxf). 1991; 24(5):632. https://muse.jhu.edu/pub/6/article/606836
  31. Ciufo T. Real-time sound/image manipulation and mapping in a performance setting. In: Proc. MAXIS Festival of Sound and Experimental Music. 2002. Accessed XXX. https://ciufo.org/media/sound_image.pdf
  32. Muriago A. Re-envisioning the Piano Recital: An Audiovisual Alternative [Doctoral Dissertation]. Connecticut, United States: University of Hartford; 2017. https://www.proquest.com/openview/4bcef2f2a86c37a4138398bc135ea0a6/
  33. Greckel W. Visualization in the performance of classical music: A new challenge. Visions of Research in Music Education. 2021; 16:35. https://opencommons.uconn.edu/vrme/vol16/iss3/35
  34. Björkman M, Jakobsson K. Switched-On Malmö Live: Exploring directions for the future of classical music streaming experiences. In: LUP Student Papers. Lund, Sweden, 2024. https://lup.lub.lu.se/student-papers/search/publication/9159100
  35. Debussy C. Images: 1re série pour piano à 2 mains. Paris: Durand & Fils; 1905. https://books.google.se/books/about/Images.html?id=5rki0gEACAAJ&redir_esc=y
  36. Debussy C. Images: 2e série pour piano seul. Paris: Durand & Fils; 1908. https://books.google.se/books/about/Images.html?id=xu4i0gEACAAJ&redir_esc=y
  37. Elliot AJ, Maier MA. Color psychology: Effects of perceiving colour on psychological functioning in humans. Annu Rev Psychol. 2014; 65(1):95–120. doi: 10.1146/annurev-psych-010213-115035
  38. Kliuchko M, Heinonen-Guzejev M, Monacis L, et al. The association of noise sensitivity with music listening, training, and aptitude. Noise Health. 2015; 17(78):350–357. doi: 10.4103/1463-1741.165065
  39. Tran US, Stieger S, Voracek M. Evidence for general right-, mixed-, and left-sidedness in self-reported handedness, footedness, eyedness, and earedness, and a primacy of footedness in a large-sample latent variable analysis. Neuropsychologia. 2014; 62:220–232. doi: 10.1016/j.neuropsychologia.2014.07.027
Share
Back to top
Arts & Communication, Electronic ISSN: 2972-4090 Published by AccScience Publishing