Communicating the dialog between painting and music with a mobile application
Contemporary visual artists often collaborate with musicians and composers to enrich their artworks and provide multisensory viewing experiences by leveraging the power of digital technology. In this paper, we present our approach to follow in their footsteps by creating a mobile application that provides “mixed-media interpretations” for a variety of visual artworks and music pieces, spanning over different time eras, with the objective to reflect the creative dialog and multifaceted interaction between their creators through time. Targeting the broader public, we attempt to capture a variety of interconnections between works of visual arts and music that may be of interest and are comprehensible to people with varying levels of knowledge on art or music, including not only inspirational but also personal and social relationships among the artists. We provide a six-layered conceptual model to ease the description of the content produced to that end while reporting several concrete examples. Then, we focus on how to communicate and digitally mediate the intertwined and often intricate interconnections in a concise way, proposing a “three-act” story scheme to progressively display short narratives on screen, combined with music reproduction on the mobile device. Finally, we explore the potential of the proposed approach through a user study with 15 participants and report a series of promising key findings, whereas also discussing future directions and pointing out open issues.
- Karel L. Comparative theories of visual art and music: May I play you a picture? McNair Scholars J. 2012;16(1):7.
- Cristiá C. On the Interrelationship between Music and Visual Art in the Twentieth and Twenty-first Centuries: A Possible Typology Derived from Cases Originated in Argentinean Artistic Field.·Trans-Revista Transcultural de Música/ Transcultural Music Review 16. Available from: https://www. sibetrans.com/trans/public/docs/trans_16_06.pdf [Last accessed 2024 Oct 19].
- Buckley A. Music iconography and the semiotics of visual representation. Music in Art. 1998;23(1/2):5-10.
- Blazekovic Z. Music and iconography. In: The SAGE International Encyclopedia of Music and Culture. United States: SAGE Publications; 2019. doi: 10.4135/9781483317731.n356
- Miu AC, Pițur S, Szentágotai-Tătar A. Aesthetic emotions across arts: A comparison between painting and music. Front Psychol. 2016;6:1951. doi: 10.3389/fpsyg.2015.01951
- Blacking J. The value of music in human experience. Yearb Int Folk Music Counc. 1969;1:33-71. doi: 10.2307/767634
- Loureiro SM, Lima F. The Background Music at Art Galleries Experience in Lisbon. In: 2017 Global Fashion Management Conference; 2017. p. 531-535. doi: 10.15444/GFMC2017.06.07.01
- Loureiro SM, Roschk H, Lima F. The role of background music in visitors’ experience of art exhibitions: Music, memory and art appraisal. Int J Arts Manag. 2019;22:4-24.
- Fekete A, Specker E, Mikuni J, Trupp MD, Leder H. When the painting meets its musical inspiration: The impact of multimodal art experience on aesthetic enjoyment and subjective well-being in the museum. Psychol Aesthet Creat Arts. 2023. doi: 10.1037/aca0000641
- Brenner B. Does Music Matter to Museum Visitors?: Understanding the Effect of Music in An Exhibit on the Visitor Experience. Thesis. University of Washington; 2016. Available from: https://hdl.handle.net/1773/36368 [Last accessed 2024 Oct 19].
- Chen CL, Tsai CG. The influence of background music on the visitor museum experience: A case study of the Laiho Memorial Museum, Taiwan. Visit Stud. 2015;18(2):183-195. doi: 10.1080/10645578.2015.1079098
- Antoniou A, O’Brien J, Bardon T, Barnes A, Virk D. Micro- Augmentations: Situated Calibration of A Novel Non-tactile, Peripheral Museum Technology. In: Proceedings of the 19th Panhellenic Conference on Informatics; 2015. p. 229-234. doi: 10.1145/2801948.2801959
- Aaron J. Integrating music with core subjects. Music Educ J. 1994;80(6):33-36. doi: 10.2307/3398709
- Seeger C. Reflections upon a given topic: Music in universal perspective. Ethnomusicology. 1971;15(3):385-398. doi: 10.2307/850639
- Semenza DC. Feeling the beat and feeling better: Musical experience, emotional reflection, and music as a technology of mental health. Sociol Inq. 2018;88(2):322-343. doi: 10.1111/soin.12194
- Crites S. The narrative quality of experience. J Am Acad Relig. 1971;39(3):291-311. doi: 10.1093/jaarel/XXXIX.3.291
- Kramer L. Music as Cultural Practice, 1800-1900. Berkeley, CA: University of California Press; 1990. Available from: https://ark.cdlib.org/ark:/13030/ft7j49p1r5 [Last accessed 2024 Oct 19].
- Levent N, Pascual Leone A, editors. The Multisensory Museum: Cross-Disciplinary Perspectives on Touch, Sound, Smell, Memory, and Space. New York: Rowman and Littlefield; 2014. doi: 10.1080/10645578.2016.1220192
- Classen C. The Museum of the Senses. 1st ed. United Kingdom: Bloomsbury Publishing; 2017. Available from: https://www. perlego.com/book/808421/the-museum-of-the-senses-experiencing-art-and-collections-pdf [Last accessed 2024 Oct 19].
- Howes D, Clarke E, Macpherson F, Best B, Cox R. Sensing art and artifacts: Explorations in sensory museology. Senses Soc. 2018;13(3):317-334. doi: 10.1080/17458927.2018.1516024
- Dudley SH. Museum Materialities: Objects Engagements Interpretations. 1st ed. England, UK: Routledge; 2010. doi: 10.4324/9780203523018
- Howes D. Introduction to sensory museology. Senses Soc. 2014;9(3):259-267. doi: 10.2752/174589314X14023847039917