Border: A Live Performance Based on Web AR and a Gesture-Controlled Virtual Instrument

Kiyu Nishida (1) , Akishige Yuguchi (2) , Jo Kazuhiro (3) , Paul Modler (4) , Markus Noisternig (5)

1 School of Design
2 NAIST - Nara Institute of Science and Technology - Graduate School of Information Science
3 Kyushu University
4 University of Arts and Design Karlsruhe
5 EAC - Espaces acoustiques et cognitifs

Recent technological advances, such as increased CPU/GPU processing speed, along with the miniaturization of devices and sensors, have created new possibilities for integrating immersive technologies in music and performance art. Virtual and Augmented Reality (VR/AR) have become increasingly interesting as mobile device platforms, such as up-to-date smartphones, with necessary CPU resources entered the consumer market. In combination with recent web technologies, any mobile device can simply connect with a browser to a local server to access the latest technology. The web platform also eases the integration of collabora-tive situated media in participatory artwork. In this paper , we present the interactive music improvisation piece 'Border,' premiered in 2018 at the Beyond Festival at the Center for Art and Media Karlsruhe (ZKM). This piece explores the interaction between a performer and the audience using web-based applications-including AR, real-time 3D audio/video streaming, advanced web audio, and gesture-controlled virtual instruments-on smart mobile devices.

Mots clés: Applied computing → Performing arts; Web-based interaction; Human-centered computing → Mixed / augmented reality; Collaborative Media; Gesture Control; Web Augmented Reality

Domaines: Traitement du signal et de l'image [eess.SP]; Informatique [cs]

Kiyu Nishida, Akishige Yuguchi, Jo Kazuhiro, Paul Modler, Markus Noisternig. Border: A Live Performance Based on Web AR and a Gesture-Controlled Virtual Instrument. International Conference on New Interfaces for Musical Expression (NIME), Jun 2019, Porto Alegre, Brazil, Brazil. ⟨hal-02469942⟩