Hugo Scurto
email: hugo.scurto@ensad.fr

An Online VR Installation for Critical Reflection on Machine Learning and Audiovisual Media

est destinée au dépôt

INSTALLATION DESCRIPTION

In terre, visitors are virtually put inside a cube, whose six faces successively display experimental music videos. Visitors can freely move inside the cube while watching and listening to these music videos using a VR display-or using keyboard and mouse with a personal computer and screen. While the installation is intended to be experienced linearly-watching and listening to each music video sequentially as they automatically start-, visitors can jump non-linearly between music videos using a pop-up menu displayed at the bottom of the web page. Lastly, visitors can experience a ``B-side'' version of terre-listening to six supplementary music tracks while moving outside the cube-using the same pop-up menu. terre is implemented with A-Frame; as such, a live demo of its current version is experienceable online 2 . 1 terre exists as the first part of an online ML installation, ciel = terre, whose second part, ciel, was presented at NIME2020 (Scurto 2020).

2 http://wolfgang.wang/terre By having VR visitors simply stare at movie screens and listen to recorded music, terre brings the notion of interactivity back to the experience of traditional audiovisual media. As such, it intends to raise three critical points related to HCI and ML research and design with sound and image: (1) the material affordances of audiovisual media, (2) the culturally-dependent perception of digital environments, and (3) the structural intelligence of interactive storytelling.

The material affordances of audiovisual media

In HCI, as well as in ML applied to sound, interactivity is often considered as the main feature to be designed in an innovative system. Yet, it does not always account for how audiovisual media themselves could be perceived as interactive by users. For example, recent works on movement sonification for rehabilitation found that sound and music design was deemed as relevant as ML-based movement-sound mappings to motivate users to perform a task [START_REF] Bevilacqua | Exploring different movement sonification strategies for rehabilitation in clinical settings[END_REF].

I was interested in using traditional audiovisual media within the VR environment to make these material affordances more palpable for visitors. Specifically, I used flat screens instead of three-dimensional virtual objects to underline how moving images may already be perceived as interactive by users, following a cinema approach [START_REF] Manovich | The language of new media[END_REF]. Also, I used binaural recordings as basis for music composition to account for the spatial immersion abilities of natural soundscapes [START_REF] Truax | Acoustic communication[END_REF], to which I added rhythmic synthetic sounds to account for the expressive affordances of music itself [START_REF] Leman | The expressive moment: How interaction (with music) shapes human empowerment[END_REF]. Lastly, I displayed these media into an elementary cube to reduce the VR space's complexity and emphasize material affordances of sound and image.

The culturally-dependent perception of digital environments

In HCI, especially in the course of current deep learning applications, many creativity support tools based on sound and image started being conceptualized, developed, and experimented, typically providing users with automatic media generation abilities-e.g., music [START_REF] Briot | Deep learning techniques for music generation--a survey[END_REF], or image [START_REF] Davis | Empirically studying participatory sense-making in abstract drawing with a co-creative cognitive agent[END_REF]). Yet, these tools are often built with minor or tacit cultural analyses of digital data used to engineer them-see the musicological and ethical issues raised by the one-million song and lyric dataset recently scraped by OpenAI (2020). As a result, images and sounds generated by such tools tend to be standardized, or else, characterized by some form of new aesthetics typical of Artificial Intelligence (AI) [START_REF] Pachet | Assisted music creation with Flow Machines: towards new categories of new[END_REF].

I was interested in exploring contemporary digital aesthetics to inquire this fast-paced quest for innovative AI applications. Specifically, I intentionally altered images and soundscapes using raw editings to evoke graphics and soundtracks typical of video games from the fifth generation era, following a post-digital approach [START_REF] Cramer | What is 'Post-digital'?[END_REF]. I also decided to reuse personal visual archives captured with my smartphone for video making to account for how mobile web technology is reconfiguring our contemporary society and its images [START_REF] Manovich | Instagram and contemporary image[END_REF]. The fact that different categories of users-e.g., non-video game players, or non-social network userswould perceive these digital environments with different familiarity suggests that cultural and aesthetic factors should be taken into account when designing AI-based HCI tools.

The structural intelligence of interactive storytelling

Many HCI systems developed within this wave of ML applications have focused on enabling users to interact more directly with audiovisual media. For example, interactive machine learning techniques for mapping allowed users to use their movement to interact with sound [START_REF] Françoise | Probabilistic Models for Designing Motion and Sound Relationships[END_REF]. Deep learning models enabled users to easily navigate through large amounts of pictures by computing smooth interpolations (RunwayML, 2020). Despite their strong potential for innovation, most of these models still lack interactive workflows enabling richer user interaction with sound and image over time.

I was interested in using interactive storytelling as a high-level form of structural adaptation, or intelligence (Mateas et al. 2003), which could inspire the design of the so-called intelligence of ML models for HCI with sound [START_REF] Scurto | Designing deep reinforcement learning for human parameter exploration[END_REF]) and image [START_REF] Koch | May AI? Design ideation with cooperative contextual bandits[END_REF]. Specifically, I used a pop-up menu-freely inspired by a CD's tracklist and a video game pause menu-to let visitors create their unique story based on the six music videos. Also, I created a B-side version of terre not only to make another nod to recorded music formats, but also to push visitors to experience flaws of the interactive installation in their personal way [START_REF] Cardoso | Breaking the game: The traversal of the emergent narrative in video games[END_REF]. While familiar to HCI researchers and video games designers-as well as to music producers to some extent-, such narrative aspects could be re-injected in ML models applied so as to empower users in their creative process with sound and image [START_REF] Parke-Wolfe | Sound Control: Supporting custom musical interface design for children with disabilities[END_REF].

ENVISIONED PERSPECTIVES

At time of writing, terre was not shown in public yet. Current perspectives include having the broad public, including researchers and practitioners in HCI and ML, experiment the installation so as to collectively discuss its raised issues. As terre will also be featured in SIIDS Art track, the symposium would constitute a first opportunity to examine both the installation and its practice-based approach. In the end, future work will include reflexive analysis of how such a practice-based approach to interdisciplinary research in HCI and ML may complement qualitative and quantitative methods usually employed in interaction design for sound and image.

Figure 1 :

 1 Figure 1: Image excerpts of terre: pop-up menu, cube (inside view), cube (outside view).


Hugo Ciel Scurto

Keywords: HUGO SCURTO, Re ective Interaction group, EnsadLab deep neural networks, noise, streaming culture, sustainable musical expression

est destinée au dépôt

INSTALLATION DESCRIPTION

ciel is an online AI installation o ering a critical meditation on AI technology and sustainable musical expression. Its noise aesthetics results from three artistic choices related to AI and music: (1) the recycling of low-quality, personal audiovisual data, (2) the cracking of deep neural networks, and (3) the allegorical displaying in an online "all-sky" map.

The recycling of low-quality, personal audiovisual data Most of AI applications to music often rely on high-quality, large musical datasets. These datasets often dematerializes the links between musical data and musicians, which raises ethical and legal issues-e.g., the one-million song and lyric dataset recently scraped by OpenAI [START_REF]OpenAI: Jukebox[END_REF]. To reappropriate AI, musicians and researchers explored data customisation practices, for example building accessible interfaces for machine learning-e.g., Rebecca Fiebrink's Wekinator [START_REF] Fiebrink | A Meta-Instrument for Interactive, On-the-Fly Machine Learning[END_REF]-, or situating data in cultural contexts-e.g., Holly Herndon's collecting of traditional singing recordings in PROTO [START_REF] Herndon | Proto. 4AD[END_REF].

I was interested in recycling low-quality, personal audiovisual data as an accessible and material data customisation practice for AI in music. I searched my BlackBerry, a mobile productivity device that let me capture poor audio recordings and pictures over eight years of my time-a somehow coherent data collection protocol for building a personal archival dataset. More than their individual qualities as experimental music media, it is their collective aspect that I wanted to explore, to counter AI applications and their perpetuation of digital dematerialisation and high-quality standards [START_REF] Steyerl | In Defence of the Poor Image[END_REF].

The cracking of deep neural networks

From the mid-twentieth century, many musicians and artists have searched for experimental sounds and images by hijacking music technology, from the manipulation of programming languages and web interfaces-e.g., Sang Won Lee's live coding of YouTube [START_REF] Lee | Live coding youtube: organizing streaming media for an audiovisual performance[END_REF]-to the breaking of physical apparatuses generating media-e.g., Fluxus's Wolf Vostell and its TV Burying [START_REF] June | Exposition of Music-Electronic Television[END_REF]. As an intermediary practice, Caleb Kelly introduced cracking as the exploitation of undesirable behaviors of playback media for artistic expression, often in a critique of the recording music industry [START_REF] Kelly | Cracked media: The sound of malfunction[END_REF].

I was interested in hijacking deep neural networks to challenge their promoted applications to music production.

Rather than manipulating the parameters of fully-trained nets-e.g., Mario Klingemann's neural glitch approach [START_REF] Klingemann | Neural Glitch[END_REF]-, I wanted to crack the learning process of deep neural nets so that they would fail to learn musical representations-while not breaking enough to still generate sound and image. I thus deliberately trained the networks over my low-quality datasets. The generated neural noise reveals the powerlessness of state-of-the-art AI applied to idiosyncratic media.

The allegorical displaying in an online "all-sky" map Automatic music generation is often framed as the main issue to be addressed with deep neural networks [START_REF] Briot | Deep learning techniques for music generation-a survey[END_REF]. While more and more studies attempt to evaluate its application to music [START_REF] Bob | Machine learning research that matters for music creation: A case study[END_REF], the application case-music generation-remains barely debated across research communities. In our times where streaming platforms weaken economic bonds between music and the public [START_REF] Eriksson | Spotify teardown[END_REF], will a new creativity support tool genuinely help musicians sustain their musical expression?

Eventually, could we think of debating such political and infrastructural issues within AI music technology research? I was interested in summoning the sky symbol-ciel in French-to frame these political issues, as an allegory of the modern myth of unlimited progress detached from its material conditions. I thus displayed the generated images inside an oval map, in reference to all-sky maps used for scienti c representation of sky radiations [START_REF]Cosmic microwave background[END_REF]. The online broadcasting of the installation eventually makes the dystopian horizon of streaming platforms continually producing AI-generated songs and sleeves accessible for musicians, research communities, and the broad public to debate collectively [START_REF] Bianchini | Towards Organogenesis: For an Instrumental Approach in Research in Art[END_REF].

Dataset specifications

I used data collected with a BlackBerry Curve 9320, which let me capture a heterogenous set of casual, descriptive, performative, private and public situations within human and nonhuman environments between 2011 and 2018. The smartphone's microphone supported mono recording with a 16,000 Hz sampling frequency. Its 3.15 megapixel camera rendered a 2048 x 1536 pixel resolution with low-quality grain as of mid-light setups. I extracted nearly 3 hours of audio from all voice notes and video records, and cropped all 2,615 camera pictures to a 256 x 256 format for preprocessing. I decided not to label the resulting datasets to cede interpretation of these experimental media to the networks alone.

Deep neural network training

I used a Variational Auto-Encoder (VAE) for audio generation [START_REF] Chemla-Romeu-Santos | Manifold representations of musical signals and generative spaces[END_REF] and a Generative Adversarial Network (GAN) for image generation [START_REF] Goodfellow | Generative adversarial nets[END_REF]. I used vschaos, a library for VAE sound synthesis1 , and a GAN implementation for image generation2 . As my intention was to only rely on my dataset to crack the networks' learning, I decided to take the networks' architecture and hyperparameters provided in the libraries' templates as granted. Training was led on my laptop's 2.8 GHz Intel Core i7 CPU 3 , and was interrupted regardless of the networks' losses. The two resulting latent spaces, exhibiting light learning artifacts, enable generation of noise sounds and images-which I propose to call neural noise. Postprocessing solely consisted in normalising sound amplitude to prevent hearing damage due to high-frequency artifacts of the VAE.

Online exhibition

Online implementation was done with A-Frame, a web framework for virtual reality experiences 4 . Broadcasting currently consists in playing a new sound and projecting a new image into the oval map every thirty seconds-in reference to the 30-second shortening of songs caused by streaming platforms and their norms of music consumption [START_REF]How streaming a ects the lengths of songs[END_REF]-from a folder of short sounds and images randomly sampled from the networks' latent spaces. Real-time generation from latent spaces may possibly be implemented by end of July-based on my experience with machine learning and SSH protocol [START_REF] Scurto | Entrain: encouraging social interaction in collective music making[END_REF]. Online visitors may use their laptop to experience the installation, using headphones or loudspeakers, setting their volume with care in relation to noise. They may click the grey circle at the bottom of page to read a program note.

FEASIBILITY

If accepted, ciel would be premiered at this year's NIME conference. As such, no documentation of previous deployment is available. Still, a live demo of its current implementation is provided in the next section to demonstrate feasibility.

  are provided details on ciel's dataset speci cations, deep neural network training, and online exhibition.

•

  Online installation: http://wolfgang.wang/ciel ARTIST'S BIO Hugo Scurto (1993, Marseille, Fr) is a researcher, musician, and designer. His research employs art, design, and science approaches to inquire, develop, and study machine learning technology within an ecology of music. His practice consists in creating interactive sound and music dispositifs in which machine learning seeks to reveal and recon gure the entanglement of individuals and their material environments. Hugo is currently postdoctoral researcher at EnsadLab, and co-founding member of wolfgang, a Paris-based music design collective. Before this, he completed a doctoral thesis in Machine Learning and Music Interaction at IRCAM (supervised by Frédéric Bevilacqua), and was visiting researcher at Goldsmiths University of London (supervised by Rebecca Fiebrink). He graduated in Physics from Ecole Normale Supérieure Paris-Saclay, received the MSc in Engineering from Sorbonne Université, and trained in Popular Music at Cité de la Musique de Marseille. With the support of a diversity of people, Hugo has publicized works and co-organised workshops in international conferences and academic reviews such as NIME, SIGGRAPH, or PLOS One, as well as in cultural places such as Friche la Belle de Mai, Le Cube, or Lutherie Urbaine.

https://github.com/domkirke/vschaos_package

https://github.com/eriklindernoren/PyTorch-GAN

The technical inadequacy of standard CPU-based computers to train deep neural networks-compared to massive GPUs required by AI applicationsmakes cracking a straightforward practice for artists wanting to inquire the economic and ecological assumptions over which AI is built.

[START_REF] Bianchini | Towards Organogenesis: For an Instrumental Approach in Research in Art[END_REF] This choice was made for aesthetic coherence with the rst part of the installation, terre, created for VR: https://hugoscurto.com/portfolio/ciel-terre/

ACKNOWLEDGMENTS

The author would like to thank Carine Le Malet and Le Cube, Axel Chemla-Romeu-Santos and wolfgang, and Frédéric Bevilacqua and IRCAM for their caring support in prototyping, implementing, and submitting this installation.

ETHICAL STANDARDS

ciel was entirely funded by the author. The author certi es that no person appearing in the original sounds and images are recognisable in the sounds and images generated for the online installation. The author knows of no way to reproduce the original sounds and images from the trained networks.