Enabling embodied metaphors in new media arts: a multi-sensory augmented reality approach
DOI:
https://doi.org/10.47989/ir31iConf64161Keywords:
Embodied metaphors, Human-information interaction, Multi-sensory augmented realityAbstract
Introduction. This study examines how multi-sensory augmented reality (MSAR), and embodied metaphor theory (EMT) can enhance immersion and comprehension in new media art. The installation, the moon knows, embodies four moon metaphors—romance, time, reunion, and self-discovery—to explore embodied interaction and audience perception.
Method. The system combined a tangible moon interface, Microsoft HoloLens 2, projection, and sensors linked to Unity3D and Arduino. Fifteen participants experienced four interactive chapters and provided feedback through questionnaires and semi-structured interviews.
Analysis. Quantitative data measured immersion, engagement, and understanding, while qualitative coding of interviews identified themes in multisensory interaction and metaphor perception.
Results. It proved possible to integrate MSAR with EMT in a new media artwork and to demonstrate that immersion and engagement were enhanced through embodied interaction. Results also revealed variations in how different metaphors were understood, with some concepts more intuitively grasped than others.
Conclusion(s). MSAR combined with EMT effectively enhances immersion and engagement in new media art. While abstract metaphors need clearer articulation, The moon knows shows how embodied multisensory design can foster emotional resonance and broaden interpretive possibilities. The potentials of MSAR being used in libraries and museums are discussed.
References
Antle, A. N., Corness, G., & Droumeva, M. (2009). What the body knows: Exploring the benefits of embodied metaphors in hybrid physical digital environments. Interacting with Computers, 21(1–2), 66–75. https://doi.org/10.1016/j.intcom.2008.10.005
Antle, A. N., Marshall, P., & Van Den Hoven, E. (2011). Workshop on embodied interaction: Theory and practice in HCI. CHI ’11 Extended Abstracts on Human Factors in Computing Systems, 5–8. https://doi.org/10.1145/1979742.1979592
Bakker, S., Antle, A. N., & van den Hoven, E. (2012). Embodied metaphors in tangible interaction design. Personal and Ubiquitous Computing, 16(4), 433–449. https://doi.org/10.1007/s00779-011-0410-4
Bilbow, S. (2021a). Developing Multisensory Augmented Reality As A Medium For Computational Artists. Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction, 1–7. https://doi.org/10.1145/3430524.3443690
Bilbow, S. (2021b). Developing Multisensory Augmented Reality As A Medium For Computational Artists. Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction, 1–7. https://doi.org/10.1145/3430524.3443690
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Cantoni, V., Dondi, P., Lombardi, L., Nugrahaningsih, N., Porta, M., & Setti, A. (2018). A Multi-Sensory Approach to Cultural Heritage: The Battle of Pavia Exhibition. IOP Conference Series: Materials Science and Engineering, 364(1), 012039. https://doi.org/10.1088/1757-899X/364/1/012039
Crofton, E. C., Botinestean, C., Fenelon, M., & Gallagher, E. (2019). Potential applications for virtual and augmented reality technologies in sensory science. Innovative Food Science & Emerging Technologies, 56, 102178. https://doi.org/10.1016/j.ifset.2019.102178
Droumeva, M., & Corness, G. (2008). Playing with the sound maker: Do embodied metaphors help children learn? Proceedings of the 7th International Conference on Interaction Design and Children, 178–185. https://doi.org/10.1145/1463689.1463754
Gere, C. (2008). New media art and the gallery in the digital age. New Media in the White Cube and beyond: Curatorial Models for Digital Art, 13–25.
Gibbs, R. W., Costa Lima, P. L., & Francozo, E. (2004). Metaphor is grounded in embodied experience. Journal of Pragmatics, 36(7), 1189–1210. https://doi.org/10.1016/j.pragma.2003.10.009
Grodecki, K., & Goulding, A. (2023). Digital Experiences in Physical Spaces: Virtual Reality and Public Libraries in Aotearoa New Zealand. In I. Sserwanga, A. Goulding, H. Moulaison-Sandy, J. T. Du, A. L. Soares, V. Hessami, & R. D. Frank (Eds.), Information for a Better World: Normality, Virtuality, Physicality, Inclusivity (pp. 157–170). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-28032-0_13
Hu, X., & Ng, J. T.-D. (2025). Low tech-barrier virtual reality content creation for cultural heritage education: Learning outcomes and pedagogy. Technology, Pedagogy and Education, 1–28. https://doi.org/10.1080/1475939X.2025.2503768
Hu, X., Ng, J. T.-D., & Liu, R. (2021). Development and Evaluation of a Digital Museum of a National Intangible Cultural Heritage from China. In K. Toeppe, H. Yan, & S. K. W. Chu (Eds.), Diversity, Divergence, Dialogue (Vol. 12646, pp. 493–501). Springer International Publishing. https://doi.org/10.1007/978-3-030-71305-8_42
Kiefer, M., & Trumpp, N. M. (2012). Embodiment theory and education: The foundations of cognition in perception and action. Trends in Neuroscience and Education, 1(1), 15–20. https://doi.org/10.1016/j.tine.2012.07.002
Kim, J., & Maher, M. L. (2019). Metaphors, Signifiers, Affordances, and Modalities for Designing Mobile and Embodied Interactive Systems. Proceedings of the 31st Australian Conference on Human-Computer-Interaction, 542–545. https://doi.org/10.1145/3369457.3369527
Kwan, J., Chu, J. H., Harley, D., McBride, M., & Mazalek, A. (2016). Grasping Cultural Context through Multisensory Interactions. Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, 482–487. https://doi.org/10.1145/2839462.2856537
Kwastek, K. (2016). Immersed in Reflection? The Aesthetic Experience of Interactive Media Art. In Immersion in the Visual Arts and Media (pp. 66–85). Brill. https://doi.org/10.1163/9789004308237_005
Lakoff, G., & Johnson, M. (2008). Metaphors we live by. University of Chicago press. https://books.google.com/books?hl=zh-CN&lr=&id=r6nOYYtxzUoC&oi=fnd&pg=PR7&dq=Metaphors+We+Live+By&ots=Lpqabq4A4X&sig=x33mQck9umXLHhY-y4obzCDhr-I
Lindeman, R. W., & Noma, H. (2007). A classification scheme for multi-sensory augmented reality. Proceedings of the 2007 ACM Symposium on Virtual Reality Software and Technology, 175–178. https://doi.org/10.1145/1315184.1315216
Liu, Y., Luo, Q., Zhou, Z., & Tian, Q. (2024). Empathy in Motion: A Case Study of “Dance of Youth” in Visual Art. Advances in Education, Humanities and Social Science Research, 12(1), 305. https://doi.org/10.56028/aehssr.12.1.305.2024
Marto, A., Melo, M., Gonçalves, A., & Bessa, M. (2021). Development and Evaluation of an Outdoor Multisensory AR System for Cultural Heritage. IEEE Access, 9, 16419–16434. https://doi.org/10.1109/ACCESS.2021.3050974
Samuel J., K. (2025). The Intersection of Technology and Arts: New Mediums for Expression. RESEARCH INVENTION JOURNAL OF RESEARCH IN EDUCATION, 5(1), 45–52. https://doi.org/10.59298/RIJRE/2025/514552
Schraffenberger, H., & Van Der Heide, E. (2016). Multimodal augmented reality: The norm rather than the exception. Proceedings of the 2016 Workshop on Multimodal Virtual and Augmented Reality, 1–6. https://doi.org/10.1145/3001959.3001960
Shapiro, L., & Stolz, S. A. (2019). Embodied cognition and its significance for education. Theory and Research in Education, 17(1), 19–39. https://doi.org/10.1177/1477878518822149
Shin, D. (2019). How does immersion work in augmented reality games? A user-centric view of immersion and engagement. Information, Communication & Society, 22(9), 1212–1229. https://doi.org/10.1080/1369118X.2017.1411519
Smith, S. (2016). Ask the Audience. Electronic Visualisation and the Arts. https://doi.org/10.14236/ewic/EVA2016.50
Sun, J. (2012). Pulse-Width Modulation. In F. Vasca & L. Iannelli (Eds.), Dynamics and Control of Switched Electronic Systems: Advanced Perspectives for Modeling, Simulation and Control of Power Converters (pp. 25–61). Springer. https://doi.org/10.1007/978-1-4471-2885-4_2
Takala, G. B. (2023). The Interactive Creativity of the Digital Era: Exploring How Media Art Redefines the Relationship Between Audience and Artwork. Studies in Art and Architecture, 2(3), 28–44.
Tribe, M., Jana, R., & Grosenick, U. (2006). New media art. Taschen Cologne. http://llrc.mcast.edu.mt/digitalversion/Table_of_Contents_128114.pdf
Vi, C. T., Ablart, D., Gatti, E., Velasco, C., & Obrist, M. (2017). Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. International Journal of Human-Computer Studies, 108, 1–14. https://doi.org/10.1016/j.ijhcs.2017.06.004
Wang, F., Zhang, Z., Li, L., & Long, S. (2024). Virtual Reality and Augmented Reality in Artistic Expression: A Comprehensive Study of Innovative Technologies. International Journal of Advanced Computer Science and Applications, 15(3). https://doi.org/10.14569/IJACSA.2024.0150365
Zamanifard, S., & Freeman, G. (2023). A Surprise Birthday Party in VR: Leveraging Social Virtual Reality to Maintain Existing Close Ties over Distance. In I. Sserwanga, A. Goulding, H. Moulaison-Sandy, J. T. Du, A. L. Soares, V. Hessami, & R. D. Frank (Eds.), Information for a Better World: Normality, Virtuality, Physicality, Inclusivity (pp. 268–285). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-28032-0_23
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Weiyue Lin, Xiao Hu
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
