Marc Chemillier

Shlomo Dubnov

Ke Chen

Mikhail Malt

Gérard Assayag

Assayag Gérard

Toots Thielemans Toots Thielemans

A posthumous improvisation by Toots Thielemans

la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

A posthumous improvisaton by Toots Thielemans

Marc Chemillier (EHESS), Shlomo Dubnov (USCD), Ke Chen (UCSD), Mikhail Malt (IRCAM), Gérard Assayag (IRCAM) In jazz, the presence of the musician is of such central importance that the recordings considered to be the best were ofen live recordings, preferably made in small intmate clubs. The evoluton of live performance technologies tends to blur this noton, frst with the gigantsm of concerts, and more recently, with the appearance of simulacra in rap and pop music in the form of holograms (Tupac), virtual stars (Hatsune Miku), video game avatars (Travis Scot), and the most recent example based on voice synthesis (Drake and The Weeknd). Toots Thielemans was one of those jazzmen whose presence lef an indelible mark on a concert or a recording session, so strong was his musical personality and his harmonica playing immediately recognizable. We propose here to try an experiment consistng in creatng a musical avatar of Toots Thielemans.

We use a music improvisaton sofware developed at IRCAM in collaboraton with the CAMS (EHESS) in Paris (Assayag et al. 2006). This machine learning system allows to capture the phrases played by an instrumentalist and to extend them by a virtual improvisaton system restoring the sound of the musician, his phrasing and his accents, but playing something diferent. The system is able to synchronize with the pulse of a human orchestra and can also follow a given chord progression.

The overall sofware architecture is based on two main components : the corpus preprocessing and the machine improvisaton. Corpus preprocessing deals with extracton of Toots solos from mixed recordings by learning the sound characteristcs of Toots harmonica from a short solo example, as will be explained below. The machine Improvisaton component consists of three main modules. A frst one built in Max, the body, containing the listening machine, the sound memory unity, the tempo following functons and allowing to interface external MIDI controllers. The mind, an external set of algorithms, made in Common Lisp and embedded in the OpenMusic computed aided compositon sofware [START_REF] Assayag | Computerassisted compositon at IRCAM: From PatchWork to OpenMusic[END_REF], in charge of calculatng new improvisaton sequences (according to harmonic and other constraints) sent to the main interface. And the third piece, the Antescofo Max object [START_REF] Cont | A coupled duraton-focused architecture for realtme music to score alignment[END_REF][START_REF] Echeveste | Operatonal semantcs of a domain specifc language for real tme musician-computer interacton[END_REF], that is the clock and the bridge allowing to synchronize the body and mind of our sofware, by messages. Currently, this last module is also able to infer tempo, adaptng past recorded sound slices to actual improvisaton sequences calculated by our OpenMusic mind taking into consideraton swing and tempo changes.

To realize the experiment and to make the machine learn Toots' playing, it was necessary to have Toots' solos in separate tracks. We used a system of separaton of the audio sources based on state of the art machine learning that was developed by the authors [START_REF] Chen | Zeros-Shot Audio Source Separaton through Query-based Learning from Weakly Labeled Data[END_REF], which comprises of a three-component pipeline including a sound event detector, an embedding processor, and a query-based source separator. The system was able to extract both the leading instrument and the remaining accompaniment of the jazz music for the machine's improvisaton. The advantage of this model is that it only requires a small audio clip of the source to extract it from the audio mixture without being pre-trained for solely separatng this source. We used the only solo part, a two-sec harmonica clip around 5:56-5:58 in the jazz song "Body & Soul" (from the album Afnity with Toots and Bill Evans, 1979), to extract the whole harmonica piece of this song. And we used an existng accompaniment part, about an 80-sec clip around 2:44-4:09, to extract the whole accompaniment. The separaton results are correct enough to be used for the following machine improvisaton.

The unique interest of this experiment lies in the high barrier of acceptability raised by trying to simulate improvisatons of a legendary human musician by the machine [START_REF] Chemillier | Étrangement musical » : les jugements de goût de Bernard Lubat à propos du logiciel d'improvisaton ImproteK[END_REF]. This goes beyond the "deep fake" challenge where the goal is to convert a sound signature of one voice to another. The task for Toots avatar combines both aspects of his unique harmonica instrument sound and his idiomatc style of improvisaton in the specifc interpretaton of jazz on the harmonica. Will an amateur who is well familiar with the playing of Toots Thielmans be musically "taken in" by listening to this fake harmonica player? Moreover, it is possible to set the artfcial improviser to produce solos that are more or less similar to the human it tries to model. At what threshold of departure from exact replicaton will the Toots fan consider that these improvisatons are not real Toots? To address these challenges, we designed a test that involved listening to Toots' original solo on "Body & Soul" and our avatar, and trying to identfy which one was the real one and which one was the fake one. A multmedia animaton showing this test is available online (Improvisaton and the computer 2023, htp://digitaljazz.fr/multmedia/improvisatonordinateur/). Notce that a similar test was presented in the exhibiton devoted to Toots Thielemans at the KBR Royal Library in Brussels, but not involving a computer avatar. The curators set up an atractve interactve terminal where one could listen to audio clips of pieces with harmonica by Toots and other players, such as Stevie Wonder, Bob Dylan, or bluesmen, and try to identfy whether they were Toots or not (fg. 1).

Figure 1. Screen display of the interactve terminal at KBR exhibiton on Toots Thielemans.

Our experiment was designed in the following way. The harmonica part extracted from Toots was provided as input to the learning process of our improvisaton sofware Djazz. We then improvised using Djazz trained on these data with the accompaniment of the piano-bassdrums part extracted from the recording. But the remaining harmonica notes that were not completely eliminated by the source separaton algorithm conficted with the harmonica notes generated by Djazz. So, to ensure clear accompaniment, we made by ear a MIDI transcripton of the piano-bass-drums part of the original recording (piano by Bill Evans, bass by Mark Johnson, and drums by Eliot Zigmund). Finally, our tests focused on an audio avatar played by Djazz with this MIDI accompaniment which was compared to the original version. The experiment was limited to an AA secton of the song which, in its complete form, has an AABA structure.

The two examples (avatar and original recording) were tested during a seminar at EHESS in Paris, and then they were sent to Hugo Rodriguez in Brussels, curator of the Toots exhibiton. He played them to various people some of them being well aware of Toots' style. They all distnguished the real Toots, but said the avatar was quite impressive. Hugo Rodriguez also made his own comments: "Overall, this is very good, and impressive! […] In terms of style: between 00:38 and 00:48, in my opinion, Toots would not have repeated this chromatc motf so many tmes in a loop. I suppose that the sofware must have been inspired by what happens between 02:08 and 02:13 in the original version. There Toots does indeed repeat something, but the patern is shorter, melodically unidirectonal (he does this more ofen, rather than a loop with descending-ascending alternaton, sometmes he also does repeated notes "turning" around a note), fts into a larger melodic phrase (so it is not the leading motf, nor an isolated patern)" (Rodriguez 2022a). This comment reveals some aspects of the Djazz process. The solo generated by Djazz takes motfs from the original solo and recombines them with respect to the pulsaton and the harmonies. The notes played by Djazz are thus notes that were originally played by Toots (Hugo Rodriguez notced that Djazz plays motfs that are "inspired by" Toots). In additon, the user can modify the output manually through an interface by adding some efects such as looping or acceleratng [START_REF] Chemillier | The smile of Velonjoro. Bi-musicality and the use of artfcial intelligence in the analysis of Madagascar zither music[END_REF]. In this interface, the yellow butons on the right in the lower part are used for looping and those on the lef are used for acceleratng. The upper part is a visual representaton of the structure of the song which in this case is two lines corresponding to AA, with the current bar lit in red (fg. 2). Hugo Rodriguez mentons a chromatc motf occurring at the beginning of the second A in Djazz solo which was extended by adding loops (loops are underlined below) :

Db-C-A#-B-Bb-A / Db-C-A#-B-Bb-A-Bb-A / Db-C-A#-B-Bb-A / Db-C-A#-B-Db-C-A#-B-Bb-A
The corresponding motf of the original solo that was picked by Djazz algorithm appeared at the beginning of the third A of the AABA form. It was repeated by Toots and extended by chromatc transpositons as follows (extensions are underlined below) :

Db-C-A#-B-A-Bb / Db-C-A#-B-A-Bb / Db-C-A#-B-A-Bb-G#-A / Db-C-A#-B-A-Bb-G#-A-G-Ab
In Toots' improvisaton, the process is quite clear : notes are added progressively at the end of the motf A-Bb, then G#-A, then G-Ab and so on. In Djazz improvisaton, the process is more obscure. Hugo Rodriguez concludes this comparison by pointng out "the absence of a feeling that the music was "saying something" in the Toots fake" (Rodriguez 2022b). This narratve aspect of music has been studied by him for other musical repertoires [START_REF] Rodriguez | Sémantque et pragmatque de la musique. Une approche cognitve basée sur les travaux de Philippe Schlenker et sur les oeuvres de Franz Liszt[END_REF] People who can distnguish the real Toots are generally convinced that Toot's desire, which includes multple factors related to his creatve intent, emotons, artstc imaginaton and interpretaton of the jazz language, are not fully present in the avatar. This is what it means to say: "Toots wouldn't have done that". It is the personality of Toots, that we try to emulate here in additon to his technical and musical skills. But in reality, the Djazz user also has his or her own desire when improvising the avatar. This makes a fundamental diference in terms of judgment of the result compared to common approaches, such as Turing test, where success (or failure) is defned in terms of producing a computer response that is indistnguishable (or notceably artfcial) from that of a human, but this could be any human and not a specifc person, and even less a hybrid response that combines user's intent with that of a cloned musical virtuoso. Using the interface connected to the Djazz sofware, the user tries to fnd manual efects that might match Toots' phrases. These are not statstcal fake efects added to the recombinaton process, but rather they express the user's own desire. The idea behind Djazz is not just to provide clones of the original material, but to ofer opportunites to improvise with it yourself. One of Hugo Rodriguez's interviewees put it this way: "In the AI solo, there are some prety fast fragments (which made me think of the bebop period), which I think Toots wasn't going to do in a piece like "Body & Soul". But, I think that's just the point?" (Rodriguez 2022a, quotaton from a musicologist who worked on Toots exhibiton). Furthermore, you can admit it's not the real Toots, but stll love the avatar. Identfying the avatar doesn't mean you think it's bad. Another person interviewed by Hugo Rodriguez pointed to the avatar's personality: "Wow, this is great! Afer a non-analytcal listening, it seems to me that he is a litle more "nervous", that he makes more notes, that he takes less tme, but I don't know enough the style of Toots to know! Maybe in his other solos Toots is like that. It is blufng in any case" (Rodriguez 2022a, quotaton from a musicologist who knows litle about Toots).

During the conference dedicated to Toots Thielemans at KBR on May 10, 2022, we had the opportunity to test it out in front of a live audience. Unfortunately, as expected from this panel of Toots specialists, they were unanimous in fnding the right Toots and we were disappointed that not a single person thought the real one was the avatar...! Jonathan De Souza from the University of Western Ontario in Canada who was invited speaker at the conference proposed a very interestng idea. Instead of using the original recording as reference, we should use the extracted harmonica part mixed with the MIDI transcripton of the piano-bass-drums trio. In this way we could avoid bias in the comparison induced by the fact that the sound of the original recording is much beter than the sound of our own mixing. Since then, we've carried out this new test on various occasions (and you can do it online as indicated above). Although there remains a majority of people fnding the real Toots, there is a signifcant number of persons who think the real one is the avatar. Almost inadvertently, this experience leads one to think about the next research challenge of cloning Toots' persona, and not only his way of making and improvising music.

This also inspires us the perspectve of a general frame for creatve musical AI, linked to the emerging feld of general intelligence. AI systems proceed by some form of imitatons at the surface and formal level. This can be reasonably convincing as has been seen above, but will always lack the power of creatve mind. It seems that what's missing now is not so much the technical depth of structural comprehension by the model than its hypothetcal capacity of being a "desiring machine" according to the concept created by philosophers Deleuze and Guatari in their book Ant-OEdipus [START_REF] Deleuze | Ant-OEdipus. Capitalism and schizophrenia[END_REF]. For these authors, desire is not so much the expression of a lack, as Lacan puts it in his famous claim that "human desire is the desire of the other", where the other is understood as the locus of symbolic formaton (Lacan 1953, 407;[START_REF] Cléro | Concepts lacaniens[END_REF], but a creatve force rooted in refexivity and interacton ("coupling machines", Deleuze and Guatari 1971, 5). Modern generatve AI methods are partcularly interestng considering the parallels of the imaginary and symbolic Lacanian concepts to problems of learning generatve music representatons and problems of modeling symbolic language-like aspects, respectvely. Problems of fnding optmal symbolizaton and identfying the symbolic units partcular to a specifc style or cultural expression seem to be emerging as a research topic important for general musical AI (Dubnov and Geer 2023;[START_REF] Dubnov | Towards Cross-Cultural Analysis using Music Informaton Dynamics[END_REF]. On this subject, it is notceable that as soon as 1954, in the wake of cybernetcs, Lacan initated a refexion on the machine and its relaton to the subject and the symbolic order (Lacan 1978, 63, 350). As pointed out by Patrick Juignet, for Lacan "the cybernetc machine is identfed with a structure detached from the subject's actvity, so that "the symbolic is the world of the machine"" [START_REF] Juignet | Lacan, le symbolique et le signifant[END_REF]. Indeed, refexivity, self motvaton, creatve intent, interacton are major challenges on the horizon of musical AI and creatve agentvity. A synthesis between a noton of autonomous agency rooted in the "ego" as inaugurated by Freud and later developed by Lacan into his theory of the subject on one hand, and the idea of co-creatve emergence where the subject dissolves into the mere interacton of producton machines as in Deleuze and Guatari is stll missing, and beyond the scope of this artcle. We might just say here as a possible research route that the imaginary and symbolic orders are on the side of the subject, thus on the side of desire as motvaton, whereas the real order (that complements Lacan's real-imaginarysymbolic trilogy, Lacan 1953) would be a host to co-creatve interacton of machines, thus on the side of desire as creatve force, breaking free from the transcendance induced by the language/symbolisaton order. This dichotomy is at the very heart of the REACH project that underlies the work presented here [START_REF] Reach | Raising CoCreatvity in Cyber-Human Musicianship[END_REF]. Now to back to Toots, where is his personality in all this? Obviously we shouldn't pretend that our AI models in any way represents Toots's personality as a creatve source. If we feel a subject here, it is as Deleuze and Guatari rightly say, a "residuum" in the interplay of machines subjected to real producton (Deleuze and Guatari 1971, 19). This is partcularly true in the case of the complex experience we described, where we actually intervened as a musician ourselves! However we can fugitvely feel the presence of Toots, due the process of using his real sound, and reproducing paterns statstcally.

What we can say is that whatever remains of Toots' musical personality is not present as a cause, but as an efect (not as a deep structure, but as a surface structure produced by the later). It is just a recording, an image, that we reenact in a smart way, and of course there are traces of his unique way of playing that persist. So the queston is: "Is there more to a musical personality, than just its shallow structure as it can be recorded, imitated, and even transformed?" The answer could be : for a dead musician no, as they can be predictable in any aspects of their producton, since they will not surprise us anymore. A dead musician's musical personality is an archive, and manipulable as such. In the interacton process (s.a. when somebody "plays Toots" with the pad) we introduce a novel causal thread (our own desire), and reenact the archive thus creatng a strange mixture of our actng personality and images of a past personality. Archive personalites are animated (like voodoo) to create an efect without cause, their personality is an image of personality like an actor seen in the cinema. So it's literally "fcton." This is consistent with Auslander's view that musicians on stage appear before us under a dual identty: as real people playing music (like actors playing their roles in the theater), and as musicians involved in narratves, and, therefore, implicitly dramatc (Auslander 2013, 56). In their positon against psychoanalysis Deleuze and Guatari have indeed vowed to avoid transcendance and interiority, and are as such a real precursor of machine interacton. An old debate is thus receiving new light in the development of AI.

Figure 2 .

 2 Figure 2. Interface of the Djazz sofware.

  ; see also for recent developments McAuley et al. 2021).

Figure 3 .

 3 Figure 3. Comparison of the original AABA solo by Toots and the AA solo by Djazz

Fundings: This research is supported by the European Research Council (ERC) REACH project, under Horizon 2020 programme (Grant 883313) and by Agence Natonale de la Recherche (ANR) project MERCI (Grant ANR-19-CE33-0010).

Assayag, Gérard, Georges Bloch, Marc Chemillier, Arshia Cont and Shlomo Dubnov. 2006. "OMax brothers: a dynamic topology of agents for improvizaton learning." AMCMM '06:

Audio and Music Researcher. Ke Chen now is a Ph.D student in Computer Music at UC San Diego, California, USA. He is co-advised by Prof. Shlomo Dubnov and Prof. Taylor Berg-Kirkpatrick. His research interest lies in the interdisciplinary between music and computer science, focusing on: Music Generatve System and Music Informaton Retrieval (including singing melody extracton, music source separaton, and music recommender system). He proposed a music generatve framework, Music SketchNet, to allow the user to specify their own ideas in automatc music generaton. He also proposed a zero-shot audio separator to allow a more efcient separaton pipeline to separate all possible sources. Ke Chen obtained his Bachelor's degree in Sofware Engineering at Fudan University, China. He is also the website maintainer of China Conference on Sound and Music Technology (CSMT) and New Interfaces for Musical Expression (NIME).

Mikhail Malt, having a twofold training, scientfc and musical (Engineer, composer and musical conductor) started out his musical career in Brazil as both futst and orchestral conductor. He has a PHD grade with a thesis at the "Ecole des Hautes Etudes en Sciences Sociales" dedicated to the use of Mathematcal models in Computer Assisted Compositon, and a HDR (Habilitaton à diriger des recherches) degree with the dissertaton : "Representaton in Computer Aided Compositon and Computatonal Musicology". Nowadays he is researcher in the Music Representatons Team -UMR 9912 STMS (IRCAM, CNRS, UPMC), associated researcher at IReMus -Sorbonne Universités and Computer Music Designer Teacher in the Educatonal Department at Ircam, Paris-France. He is currently pursuing his research and compositon actvites in the felds of musical modeling and discovery, computer assisted performance, and musical representaton epistemology.

Shlomo