This thesis explores the application of Mathematical Morphology to the analysis and generation of music, focusing on two time-frequency representations: spectrograms and piano rolls. Mathematical Morphology is a nonlinear image processing tool that serves to consider topological notions of the image. We present three applications. The first is to analyze spectrograms with morphological tools to obtain parameters with which to synthesize a musical instrument sound. The second is to generate piano rolls with two musical parameters, texture and harmony, by arranging them through morphological dilation. The third is to apply morphological operators to analyze piano rolls using graph theory. The thesis thus proposes new approaches for problems in sound analysis and computational musicology.

The completion of this PhD represents the culmination of years of dedication and study. Although the journey of writing a PhD is often a lonely one, it would not have been possible without the support of numerous individuals who stood by my side. Here is the list of those I wish to thank for their unwavering support.

But before that, I want to thank the institutions that made possible me becoming doctor: Sorbonne Université, which I consider my alma mater, and IRCAM, which is the place I want to be. It is marvelous that these two institutions work together and that I was able to be part of them through the ATIAM master and the EDITE doctoral school.

I would like to thank now all the people that helped me along the way. First of all, I want to thank my two supervisors, Carlos Agón and Isabelle Bloch for believing in me and supporting me during all this journey. I could not have dreamed with better supervisors; together, they form the necessary Yin and Yang.

Secondly, I want to thank Moreno Andreatta and Gérard Assayag for having introduced me into IRCAM's RepMus team. It has been a real pleasure to be part of such an amazing team and to finally found people that explore the very specific domain I dreamed all my life: computational musicology.

This PhD lies at the intersection of three domains: mathematics, computation and music. In the process of learning these topics, I was helped by several professors that marked me. I would like to thank all of them.

In particular, I want to thank my teachers from the Real Conservatorio Superior de Música de Madrid; special thanks to Alicia Díaz de la Fuente, who taught me analysis, Alejandro Román, who encouraged me to visualize music, Zulema de la Cruz, who introduced me to sound synthesis, and Enrique Rueda, who explained to me what harmony means.

Also, I want to thank those from Universidad Complutense de Madrid, in particular Daniel Azagra Rueda, the supervisor of my first research work, and the one who encouraged me to come to Paris and to integrate Sorbonne Université.

And finally, the teachers of ATIAM's Master, from whom I learned computing and signal processing. In particular, I want to thank Roland Badeau, the one who taught me all I know about Signal Processing, and Philippe Esling, the one who taught me all I know about Machine Learning.

And now is the time to thank those that did contribute to my PhD in nonacademic ways. I prefer to express my gratitude to them in the language I speak with them, so I will use either Spanish of French.

Me gustaría empezar por mi familia. Esas personas que me apoyaron indefectibleiii mente y tendieron la oreja cada vez que quise contarles mis extravagancias matemusicales. Me siento muy afortunado de tenerlos a mi

Résumé

Cette thèse explore l'application de la Morphologie Mathématique à l'analyse et à la génération de musique, en se concentrant sur deux représentations temps-fréquence : les spectrogrammes et les piano roll. La Morphologie Mathématique est un outil de traitement d'images non linéaire qui sert à exploiter des notions topologiques de l'image. Nous présentons ici trois applications : la première consiste à analyser les spectrogrammes avec des outils morphologiques pour obtenir des paramètres avec lesquels synthétiser un son d'instrument musical. La seconde est de générer des piano roll avec deux paramètres musicaux, la texture et l'harmonie, en les agençant avec la dilatation morphologique. La troisième consiste à appliquer des opérateurs morphologiques pour analyser les rouleaux de piano en utilisant la théorie des graphes. Ainsi, la thèse propose de nouvelles approches pour les problèmes d'analyse sonore et de musicologie computationnelle.

ii partager des soirées entières à parler informatique et musique.

Et puis le temps passe et tu rencontres des gens qui rentrent dans ta vie avec une puissance particulière : Daniel Bedoya, camarade de COSMOS, Thomas Borsoni, camarade de maths, Christophe Weis, camarade d'analyse musicale, Pierre Ludmann, camarade d'informatique, Timothée Chambery, camarade de discussions, Mohsen Mechichi, camarade de jeux. Je suis très heureux de vous avoir connus.

Et, avant de finir avec le français, je voudrais remercier les nouveux amis que je me suis fait ici pour l'accueil chaleureux qu'ils m'ont réservé. Merci de m'avoir intégré dans votre clan. De plus, je ne voudrais pas oublier Victoria, avec qui j'ai passé une très bonne semaine à l'IRCAM et qui m'a invité à son séminaire à Strasbourg, ni Paul Lascabettes, avec qui j'ai partagé cette aventure de Morphologie Mathématique Musicale.

Y ya va siendo hora de volver a las raíces: Madrid y Valencia. Y a aquellos que llevan conmigo desde la uni y el liceo. Empecemos por los de Madrid.

En aquellos tiempos, acababa de llegar a Madrid y estaba ansioso de saber qué me depararía la vida. Y fue en ese momento que conocí a personas que llevo y llevaré siempre en mi corazón. En el orden cronológico, el primero fue Jorge, que me recibió con un « hola, me llamo Jorge », seguido de Alberto, uno de esos amigos que siempre te alegras de ver y con el que sientes que la conversación podría durar para siempre; pocos hay en el mundo tan listos y tan cultos. Un gran hallazgo fue Adrián Pineda; desde ese día en el ascensor y esa conversación en los cien montaditos, me abrió las puertas de sus círculos de par en par. Tardes enteras de conversaciones de música y noches enteras de juerga madrileña. Me gustaría también mencionar a los que llegaron al final: Xabi, con el que terminé la carrera (aún recuerdo ese trabajo de topología algebraica), y Darío, aquel que ha pasado de alumno a maestro.

Y para terminar de enrraizar, volvamos a Valencia, esa tierra tórrida de la que vengo, lo quiera o no. Allí pasé mi infancia y adolescencia, y guardo de esos tiempos muchos amigos. Pero de todos ellos, me gustaría resaltar cuatro: Jorge, Pedro, Louis y Cristóbal. Cada uno diferente al otro. Cada cual en lo suyo. Y, a día de hoy, cada uno en un sitio. Del último, me gustaría resaltar hasta qué punto ha sido central en mi desarrollo; tanto hemos hablado que podría decirse que hemos formado una mente común. Gran visionario, mejor persona. Pero no quisiera olvidarme de dos personas de Valencia que han contribuido directamente a esta tesis: Marina Delicado, con la que grabé los samples de piano que utilicé al principio de mi tesis, y Carlos García Pagán, que ha diseñado la figura 2.8d.

Hay una persona que he dejado deliberadamente para el final: Bérengère Denizeau. No voy a mencionar los diferentes apelativos que utilizo para dirigirme a ella, pues podrían llenar una tesis entera. Ma compagne d'aventures, mi media v naranja, aquella que hace que cada día sea un regalo. La personne qui m'a soutenu du début jusqu'à la fin, en passant par le milieu, et encore après. Siempre ahí; intelligente, subtile, mais surtout drôle. Sagace et stratégique, douce et ferme, avec un oeil d'aigle et un coeur de lion. Es músico y escritora, baila y canta, et jongle avec le langage haciendo trucos de ilusionista. Heureusement que je t'ai connu. C'est à toi que je dédie cette thèse. vi Notations Set theory 1. P(X): the power set of X.

2. B A : the set of functions from A to B, i.e., B A = {f ∈ P(A × B) :

a = b ⇒ f (a) = f (b)} 3. |X|: if X is
a set, the cardinal of X, i.e., the number of elements of X. If the number of elements of X is infinite, we can notate |X| = ∞ and, if needed, use the usual ℵ 0 , ℵ 1 , ... for different cardinals.

4. P F (X): the finite subsets of X. We can write P F (X) = {A ∈ P(X) : ∃n ∈ N such that |A| = n}.

Arithmetic

We will use several common number sets. We will use the dot symbol for decimals since the comma symbol is already used to enumerate elements.

1. N: the set of natural numbers with the zero included2 . We will call them the natural numbers.

2. N * : the set of natural numbers without the zero. We will call them the positive natural numbers.

3. Z: the set of integer numbers.

4. Q: the set of rational numbers.

Functions

1. A ∈ {0, 1} X : the characteristic function of subset A of X, i.e.,

A : X → {0, 1}

x →

1 if x ∈ A 0 if x / ∈ A .
viii 2. We will use often the canonical bijection between the subsets of a set X and its characteristic function, i.e., : P(X) → {0, 1} X A → A .

3. supp(f ): the support of the function f , i.e., if f : X → K, supp(f ) = {x ∈ X :

f (x) = 0}
, where K is a field.

Functional spaces

1. L 1 (R; C): the space of functions from R to C that are integrable i.e., such that

||f || 1 = R |f (t)| dt < ∞.
2. L 2 (R; C): the space of functions from R to C that are square-integrable i.e., such that ||f || 2 = R |f (t)| 2 dt < ∞.

3. L ∞ (R; C): the space of functions from R to C that are bounded i.e., such that

||f || ∞ = sup{|f (t)| : t ∈ R} < ∞.
4. C ∞ (R; R): The space of smooth functions from R to R, i.e., the functions f : R → R such that f (k) exists and is continuous ∀k ∈ N.

Order theory

1. ∨: the supremum symbol as a binary operation (for instance, a ∨ b).

2. ∧: the infimum symbol as a binary operation (for instance, a ∧ b).

3. : the supremum symbol as a unary operation (for instance, A).

4. : the infimum symbol as a unary operation (for instance, A).

Spaces for music

These spaces are presented in Chapter 2.

1. T : a time space. ix 2. T s : the time space measured in seconds.

3. T 1 : the time space measured in samples. It is equal to Z.

T p q

 : the time space measured in wholes with a p q time signature. 5. F: a frequency space.

Introduction

Music encompasses a variety of activities, among which two stand out: the generation of music and the analysis of music. These two activities are the focus of this thesis. What sets this work apart lies in our approach to the generation and analysis of music through a specific mathematical and computational discipline: Mathematical Morphology.

Mathematical Morphology (MM) is a domain that lies at the intersection of Mathematics and Computer Science. It was originally formulated in the mid-1960s by [START_REF] Matheron | An expert system for harmonic analysis of tonal music[END_REF], 1965, 1975[START_REF] Serra | Image analysis and mathematical morphology[END_REF] Historically, this was the first consistent nonlinear image analysis theory [START_REF] Najman | Mathematical morphology: From theory to applications[END_REF], and it remains an indispensable tool in today's field of image analysis.

Although originally developed for image analysis, particularly in the context of porous media, mathematical morphology has transcended its initial scope. This evolution into a broader discipline is highlighted by the development of a mathematical formalism based on lattice theory.

The main challenge of this thesis is to explore the applications of MM to music. Our chosen methodology involves examining the different musical representations to which MM can be applied, and determining their musical relevance. To achieve this, we focus on a specific aspect of MM that employs structuring elements. This requires a space of the form

(T E , )
where (T, ≤) is a complete lattice and E is a set endowed with a notion of neighborhood, that is, for each point p ∈ E, there is a neighborhood V(p) associated. The order is the pointwise order induced by ≤, i.e., ∀f, g ∈ T E , f g ⇔ ∀p ∈ E, f (p) ≤ g(p).

Multiple methods exist for associating a neighborhood with a point, but in MM, a prevalent approach is to use what is called a structuring element B. The neighborhood of p is then defined as the translation p + B. One way to accomplish this is by having a group (G, +) acting on E, and consider B as a subset of G.

Introduction

Group theory has proven extremely useful in musical analysis and generation [START_REF] Lewin | Generalized musical intervals and transformations[END_REF][START_REF] Andreatta | On group-theoretical methods applied to music: Some compositional and implementational aspects[END_REF][START_REF] Papadopoulos | Mathematics and group theory in music[END_REF]. Given this, we posit that MM can be meaningfully applied to music from a group-theoretical perspective. More precisely, we propose a particular space that we call a musical space and that is of the form M = A T ×F where T represents time, F represents frequency, together constituting the timefrequency plane T × F , and A represents the set of possible amplitudes. The timefrequency plane, T × F , is accompanied by a group (G T ×F , +) that acts on it. Additionally, the amplitude set is endowed with a complete lattice structure (A, ≤).

These musical spaces capture what are commonly referred to as time-frequency representations of music. Traditionally, musical representations are categorized into two major families: signal-based and symbolic. Within both of these families, multiple time-frequency representations exist. Our objective is to establish a unified framework within which MM can be applied. Due to its algebraic nature, we believe that MM can be adapted to accommodate both families of representations, despite their seemingly distinct natures.

Among the various time-frequency representations available, we have selected one from each major family: spectrograms for the signal-based family and piano rolls for the symbolic family. These chosen representations are well-studied and provide a robust foundation for the application of MM. Figure 1 showcases an example of each representation, featuring the initial five bars of Bach's Toccata and Fugue in D minor, BWV 565.

Several questions naturally arise within this context: what types of lattices should we employ for spectrograms and piano rolls? Which groups are most suitable? What is the musical significance of the structuring elements? Addressing these questions constitutes one of the primary focuses of this thesis. Throughout this work, we aim to answer these questions while ensuring musical relevance.

Although the primary goal of this thesis is to offer tools for both generating and analyzing music, considerable effort is devoted to establishing a rigorous mathematical framework. To this end, this thesis is replete with definitions, propositions, examples, and other standard mathematical terminology. Aside this mathematical rigor, every theoretical contribution has a computational counterpart. We have made the code available in a public repository at https://github.com/Manza12/MMM where one can find the figures, algorithms, and sounds presented throughout this thesis, together with the code that was used to generate them.

This thesis is organized as follows: Chapter 1 is devoted to providing a comprehensive introduction to mathematical morphology. In this chapter, we present the key operators that will be used throughout the thesis and define the specific mathematical constructs essential for applying MM.

For example, we introduce a new formalism for greyscale MM, known as the residuated triplet, which enables a clear understanding of the roles of each space involved in the process, namely, the input, the structuring element, and the output. This distinction between spaces and precision in the use of operations is crucial in our framework. Unlike in image applications of MM, where the input and the output Introduction are often of the same nature (i.e., an image), in our case this assumption does not hold true and lacks musical significance.

In Chapter 2, we explore the diverse options for time, frequency, and amplitude to construct spaces suitable for applying MM. We examine a range of choices, both discrete and continuous, as well as algebraic and analytical. We then formalize the structures that characterize our main representations: spectrograms and piano rolls.

We place particular emphasis on distinguishing between a space (a set of points) and a group acting on this space (a set of transformations, which in our case are conceived as shifts). This distinction enables us to assign a musical meaning to these actions, specifically the concept of translating a timestamp or a pitch by a certain amount (e.g., shifting A up by 3 semitones results in C).

Moreover, we also address amplitudes in a musically meaningful way. We explore various possible amplitude ranges and associate them with corresponding musical phenomena. This effort led to the concept of a residuated triplet; in traditional image processing, the amplitude range is often limited to grayscale values, but in the case of piano rolls, this may not be the case.

Finally, this chapter introduces a formal definition of a piano roll as well as a derived representation known as the chroma roll. While these concepts are standard in computational musicology, formal mathematical definitions have been lacking. We aim to provide one that is both flexible and rigorous.

After establishing the fundamental theoretical framework in Chapters 1 and 2, we turn to our primary contributions in the subsequent chapters. The remaining three chapters delve into the potential applications of MM for addressing specific Music Information Retrieval (MIR) tasks.

Chapter 3 explores the use of MM for analyzing spectrograms of sounds produced by musical instruments. Such sounds possess specific characteristics (an attack, a sinusoidal component, and a noise element) that make them well-suited for MM analysis. Specifically, the attack usually manifests as a vertical line in the spectrogram, while the sinusoidal component appears as a series of horizontal lines (see Figure 2a). In this context, we lean towards the conventional usage of MM, as it is typically applied in image analysis. Rather than endowing our structuring elements with musical significance, we design them to alter the image in a manner conducive to isolating specific details (the vertical and horizontal lines).

The noise component exhibits distinct characteristics compared to the other elements; it manifests as a density of energy punctuated by holes (see Figure 2b). In this case, traditional MM techniques prove to be particularly effective, enabling us to isolate this noise component by filtering out the lines, and thus the other components.

These methods are integrated into a unified morphological processing pipeline. The processed information is subsequently utilized to synthesize a sound that closely resembles the original input.

Chapters 4 and 5 shift our focus to the use of MM for both generating (Chapter 4) and analyzing (Chapter 5) piano rolls. In these chapters, structuring elements change from being mere parameters to becoming central actors. We endow them with musical significance and employ them for both the generation and analysis of pieces represented as piano rolls.

One of the key contributions emerges in Chapter 4: the formalization of music generation through two musical (yet mathematical) parameters: texture and harmony. While these terms can be contextually ambiguous, we give them precise mathematical definitions within our framework. This chapter draws upon tools from other algebraic disciplines, such as the tensor product, and utilizes the full arsenal of techniques previously exposed, including residuated triplets.

To organize textures and harmonies coherently, we introduce an implementation model based on Python objects and XML documents. This model is intended for integration into Computer Assisted Composition (CAC) software, offering an alternative to traditional score editors for generating music that aligns with classical harmonic procedures.

The final chapter, Chapter 5, is dedicated to applying both MM and graph theory to address complex tasks in Music Information Retrieval (MIR). Specifically, we tackle two major challenges: chord segmentation and harmonic analysis. MM proves to be exceptionally well suited for these tasks, particularly for Roman numeral anal-Introduction ysis. In this context, the operation of erosion takes on significant meaning and serves as the foundation for constructing what we term a tonal graph.

The tonal graph represents another major contribution of this thesis. We believe it addresses the challenges of harmonic analysis in a notably elegant manner, offering high levels of customization while still delivering strong results with minimal configuration. Although this concept is introduced for the first time in this work, ongoing research is being conducted and is expected to be published in subsequent studies.

While the core of our work is firmly anchored in Mathematical Morphology, our scope broadens to include a diverse array of fields within Mathematics and Computer Science, such as Group Theory, Abstract Algebra, Fourier Analysis, and Graph Theory. Central to our methodology is the role of MM tools as key elements in a holistic problem-solving pipeline. This approach is vividly demonstrated in Chapters 4 and 5, where we employ algebraic methods and graph-based strategies to address complex challenges, ranging from the generation of musical compositions to the analysis of harmony and texture.

Chapter 1

Mathematical Morphology

Mathematical Morphology (MM) is a theory and technique used for the analysis and processing of geometrical structures, initially developed for image analysis in the mid-1960s by two researches at the École des Mines in Paris: [START_REF] Matheron | An expert system for harmonic analysis of tonal music[END_REF][START_REF] Serra | Image analysis and mathematical morphology[END_REF] Its mathematical foundation draws upon set theory, lattice theory, topology, and random functions. While MM is predominantly applied to digital images, its versatility allows for its use on various other mathematical structures, including graphs, surface meshes, or solids.

In this chapter, we present the standard framework of deterministic Mathematical Morphology (MM) based on lattice theory. Specifically, we develop MM based on structuring elements with our input being sets (leading to binary MM) and functions (leading to greyscale MM).

As a roadmap, we are guided by the works of [START_REF] Najman | Mathematical morphology: From theory to applications[END_REF][START_REF] Najman | Mathematical morphology: From theory to applications[END_REF][START_REF] Bloch | Mathematical morphology[END_REF], adopting their notations with few exceptions. Additionally, we aim to enhance the mathematical formalism of the framework in which we will operate throughout the thesis. To achieve this, we will use the notion of a group acting on a set1 for the application of MM with structuring elements. Moreover, we will incorporate the concept of a residuated lattice triplet for the codomains of functions, ensuring that the operations performed on them are well defined.

The organization of the chapter is as follows: we begin with the algebraic foundations of MM in Section 1.1, where we present the standard operators in the framework of complete lattices. Next, in Section 1.2, we delve into the common framework of MM with structuring elements, introducing the necessity of a group acting on a set and a residuated lattice triplet. Finally, in Section 1.3, we discuss some essential considerations for implementing the operators. In particular, we will present our Chapter 1. Mathematical Morphology contributions in the Python/PyTorch framework, achieved through the creation of the MM library nnMorpho.

Mathematical Morphology on Lattices

The commonly accepted framework for MM in the deterministic setting is the theory of complete lattices [START_REF] Ronse | Why mathematical morphology needs complete lattices[END_REF]. In this section, we provide a review of the definitions of complete lattices and basic morphological operators. For proofs of propositions and more in-depth development about this topic, we refer to [START_REF] Heijmans | The algebraic basis of mathematical morphology i. dilations and erosions[END_REF] and [START_REF] Ronse | The algebraic basis of mathematical morphology: II. openings and closings[END_REF].

Complete Lattices

The definition of a lattice is based on the concept of partially ordered set or, abbreviated, poset; this algebraic structure is extensively presented in [START_REF] Birkhoff | Lattice theory[END_REF] and its definition is recalled in Appendix A (Definition A.2).

In order to define a lattice, we use the notions of supremum and infimum. These notions are defined formally in [START_REF] Birkhoff | Lattice theory[END_REF] and their definitions are recalled in Appendix A (Definitions A.3 and A.4). We recall that the supremum of a set is the lowest upper bound of the set and the infimum of a set is the greatest lower bound of the set. We use the notations exposed in the preamble. These elements may not exist in general lattices.

Definition 1.1 (Lattice). Let (L, ≤) be a partially ordered set, and ∨ and ∧ the supremum and infimum associated with ≤. 3. If (L, ≤) is both an upper semilattice and a lower semilattice then (L, ≤) is a lattice.

We then use the notation (L, ≤, ∨, ∧) for lattices. Let us present the two main examples of lattices that we are using in this thesis.

Examples 1.2.

1. Let E be a set. Then the set of subsets of E equipped with the set inclusion, union and intersection, (P(E), ⊆, ∪, ∩) is a lattice.

Mathematical Morphology on Lattices

2. Let E be a set and (T, ≤) be a lattice. We define the pointwise order by ∀f, g ∈ T E , f g ⇔ ∀p ∈ E, f (p) ≤ g(p) .

Then, (T E , ) is a lattice. In this case, the supremum ∨ and the infimum ∧ are given by:

∨ : T E × T E → T E (f, g) → f ∨ g : E → T p → f (p) ∨ g(p) ∧ : T E × T E → T E (f, g) → f ∧ g : E → T p → f (p) ∧ g(p)
.

These two examples are the ones that we will work with throughout the thesis; the first one is called the binary case and the second one the functional case.

When working with lattices, one can take the supremum and the infimum of any finite subset by considering each element one by one2 . However, we cannot guarantee that the supremum of an infinite subset of the lattice exists. Lattices that satisfy this property are called complete. They are presented now and will be utilized throughout the thesis. Definition 1.3 (Complete lattice). Let (L, ≤, ∨, ∧) be a lattice. We say that (L, ≤ , ∨, ∧) is a complete lattice if ∀A ⊆ L, ∃ A ∈ L and ∃ A ∈ L.

In the case of complete lattices, supremum and infimum can also be considered as unary operators defined on the power set of the lattice:

: P(L) → L A → A and : P(L) → L A → A .
We substitute then ∨ and ∧ by and in the notation.

A direct consequence of (L, ≤, , ) being a complete lattice is that there exist two particular elements:

• the top element, denoted by ⊤, which is the supremum of L, i.e., ⊤ = L,

• the bottom element, denoted by ⊥, which is the infimum of L, i.e., ⊥ = L.

We then use the notation (L, ≤, , , ⊤, ⊥) for complete lattices.

Chapter 1. Mathematical Morphology Remark 1.4. Let (L, ≤, , , ⊤, ⊥) be a complete lattice. Then,

⊤ = ∅ ⊥ = ∅ .
Let us discuss the Examples 1.2 from the perspective of complete lattices.

Examples 1.5.

1. Let E be a set. Then, the lattice (P(E), ⊆) is complete. The top element is E and the bottom element is ∅.

2. Let E be a set and (T, ≤, , , ⊤, ⊥) be a complete lattice. Then, (T E , , , , ⊤, ⊥) is a complete lattice. The unary operators are then written

: P(T E ) → T E F → F : E → T p → {f (p) ∈ T : f ∈ F } : P(T E ) → T E F → F : E → T p → {f (p) ∈ T : f ∈ F }
and the top element and bottom element are

⊤ : E → T p → ⊤ and ⊥ : E → T p → ⊥ respectively.
Finally, let us recall the notion of a complemented lattice. Even if it is not needed for the main tools of mathematical morphology, it is useful for some particular cases and comes handy in some proofs. Definition 1.6 (Complemented lattice). Let (L, ≤, ∨, ∧) be a lattice. We say that (L, ≤, ∨, ∧) is a complemented lattice if there exists a function

• c : L → L a → a c called complementation, that satisfies: ∀a ∈ L, a ∨ a c = ⊤ a ∧ a c = ⊥ .
In this case, we notate (L, ≤, ∨, ∧, • c ).

Dilation and Erosion

Let us now present the most basic morphological operators: dilation and erosion.

In the following, we omit the tedious notation (L, ≤, , , ⊤, ⊥) for complete lattices, and use simply L or, eventually, (L, ≤). In order to be consistent, every time that L is involved, it comes with its usual order, operations and elements (≤, ∨, ∧, , , ⊤ and ⊥), and if several lattices are presented, we assign a subscript to each of them and propagate it through the order, operations and elements to keep the notations consistent (for instance,

L 1 , ≤ 1 , ∨ 1 , ∧ 1 , 1 , 1 , ⊤ 1 and ⊥ 1 ).
Definition 1.7 (Dilation). Let L 1 and L 2 be two complete lattices. We say that an operation δ : L 1 → L 2 is a dilation if it commutes with the supremum, i.e.,

∀A 1 ⊆ L 1 , δ 1 A 1 = 2 δ(A 1 ) , (1.1) where δ(A 1 ) = {δ(a 1 ) ∈ L 2 : a 1 ∈ A 1 }.
Definition 1.8 (Erosion). Let L 2 and L 1 be two complete lattices. We say that an operation ε : L 2 → L 1 is an erosion if it commutes with the infimum, i.e.,

∀A 2 ⊆ L 2 , ε 2 A 2 = 1 ε(A 2 ) , (1.2) where ε(A 2 ) = {ε(a 2 ) ∈ L 1 : a 2 ∈ A 2 }.
A direct consequence of Definitions 1.7 and 1.8 is that dilation and erosion are increasing operators.

Erosions and dilations usually come in pairs; if chosen properly, they form an adjunction.

Definition 1.9 (Adjunction). Let (P 1 , ≤ 1 ), (P 2 , ≤ 2 ) be two partially ordered sets. Let α : P 1 → P 2 and β : P 2 → P 1 be two operators. We say that (β, α) is an adjunction

if ∀a 1 ∈ P 1 , ∀a 2 ∈ P 2 , α(a 1 ) ≤ 2 a 2 ⇔ a 1 ≤ 1 β(a 2 ) . (1.3)
We say that α is lower adjoint of β, and β is upper adjoint of α.

It is important to remark that, since α and β lay at opposite directions of the order symbol ≤, they do not play similar roles. This is why the adjunction has an order (β, α).

The next theorem expresses the equivalence between adjunctions and pairs of erosion-dilation in the case of complete lattices.

Chapter 1. Mathematical Morphology Theorem 1.10. Let L 1 and L 2 be two complete lattices. Then, 1. Given δ : L 1 → L 2 and ε : L 2 → L 1 such that (ε, δ) is an adjunction, δ is a dilation and ε is an erosion.

Conversely,

(a) given a dilation δ : L 1 → L 2 , there is a unique erosion ε such that (ε, δ) is an adjunction, given by:

ε : L 2 → L 1 a 2 → {a 1 ∈ L 1 : δ(a 1 ) ≤ a 2 } , (1.4) (b) given an erosion ε : L 2 → L 1 , there is a unique dilation δ such that (ε, δ)
is an adjunction, given by:

δ : L 1 → L 2 a 1 → {a 2 ∈ L 2 : a 1 ≤ ε(a 2 )}
.

(1.5)

Opening and Closing

We first recall the general algebraic definitions of opening and closing, as particular morphological filters. Forms of these operators can be built by composition of adjoint erosion and dilation.

Definition 1.11 (Opening). Let L be a complete lattice. Let γ : L → L be an operator such that

1. ∀x, y ∈ L, x ≤ y ⇒ γ(x) ≤ γ(y) , (Increasing) 2. ∀x ∈ L, γ(x) ≤ x , (Anti-extensive) 3. γ 2 := γ • γ = γ . (Idempotent)
Then γ is an opening.

Definition 1.12 (Closing). Let L be a complete lattice. Let ϕ : L → L be an operator such that

1. ∀x, y ∈ L, x ≤ y ⇒ ϕ(x) ≤ ϕ(y) , (Increasing) 2. ∀x ∈ L, x ≤ ϕ(x) , (Extensive) 1.2. Mathematical Morphology with Structuring Elements 3. ϕ 2 := ϕ • ϕ = ϕ . (Idempotent)
Then ϕ is an closing.

Openings and closings are particular cases of morphological filters. Opening, being anti-extensive, eliminates (or reduces) elements from the input, while closing, being extensive, add (or increases) elements. Furthermore, the third condition, idempotence, guarantees that all removals or additions take place in the initial iteration. This differs from conventional linear filters, where applying the same filter multiple lead to different results.

A common method of creating openings and closings is by combining dilations and erosions.

Proposition 1.13. Let L 1 and L 2 be two complete lattices and (ε, δ) be be an adjunction with δ :

L 1 → L 2 and ε : L 2 → L 1 . Then, 1. ϕ := ε • δ : L 1 → L 1 is a closing, 2. γ := δ • ε : L 2 → L 2 is an opening.

Mathematical Morphology with Structuring Elements

Common specific forms of morphological operators are defined based on structuring elements. A structuring element can be viewed as a pattern that we seek to find (in the case of erosion) or replicate (in the case of dilation) throughout the input of the operator.

The most abstract approach to achieve this involves having a binary relation between elements of the domain3 . However, in this work, we are going to introduce a group action, that naturally leads to a binary relation.

Action of a Group on a Set

In this section, we introduce the action of an additive4 group (G, +) on a set E. This provides a framework for utilizing the notion of a structuring element. We use [START_REF] Rotman | An introduction to the theory of groups[END_REF] as reference for group theory.

Chapter 1. Mathematical Morphology Definition 1.14. Let (G, +) be a group with identity element 0 and E be a set. We say that a function

+ : E × G → E is an action if 1. ∀p ∈ E, p + 0 = p, (Identity) 2. ∀p ∈ E, ∀x, y ∈ G, (p + x) + y = p + (x + y). (Compatibility)
It should be noted that we deliberately performed an abuse of notation by using + for both the group operation and the group action. This abuse of notation simplifies the expressions significantly and the confusion is impossible since the context makes it clear to which elements + is being applied.

Intuitively, the group G shifts the elements of E. This concept can be compared with the definition of an affine space, where elements of the affine space (called points) are shifted by the vectors of the associated vector space. Similarly, in our case, we also refer to the elements of the set E as points, thus calling E the space, while the elements of (G, +) are referred to as shifts.

The most basic action is the translation action of a group into itself.

Definition 1. 15 (Translation action). Let (G, +) be a group. Then, the translation action of a group into itself is the function

+ : G × G → G (x, y) → y + x
.

(1.6)

A straightforward approach to create an action when we have a group (G, +) and a set E that are in bijective relation is by combining the translation action with the bijection.

Proposition 1.16. Let (G, +) be a group, E be a set and ι : E → G a bijection between E and G. Then,

+ : E × G → E (p, x) → ι -1 (ι(p) + x) (1.7)
is a group action.

We will employ this technique to establish associations between groups and sets. In the following examples, we present the main groups that will be utilized extensively in this thesis.

Examples 1.17.

Mathematical Morphology with

+ : (G 1 × G 2 ) × (G 1 × G 2 ) → G 1 × G 2 (x 1 , x 2 ), (y 1 , y 2 ) → (x 1 + 1 y 1 , x 2 + 2 y 2 )
.

(1.8)

We use these examples to illustrate translation in Figure 1.1. The figure illustrates a significant observation: in this case, the element of the set, denoted by p, represents a point, while the element of the group, denoted by x, represents a shift. Translating a point by a shift results in another point.

Whereas translation actions are defined for particular elements of a set, we can also apply them to a subset of the set; this is done by the following definition: ∀x ∈ G,

∀A ⊆ E, A + x := {p + x ∈ E : p ∈ A} .
(1.9)

Similarly, ∀B ⊆ G, ∀p ∈ E, p + B := {p + x ∈ E : x ∈ B} .
(1.10)

These equations are illustrated in Figure 1.2. We outline that, even if the addition is commutative, the group action is not. To maintain consistency, we always place the point-like element first and the shift-like element after when performing the group action.

The case p+B will be used extensively in the following: B will be the structuring element (a subset of G) and p will be each of the points of the input. Note also the importance of the 0 in this case: arrows are depicted going from 0 to the coordinates of each of the elements of B.

Furthermore, the fact that G is a group makes possible the reflection of B. We denote by B the reflection of B given by the formula B = {-b ∈ G : b ∈ B}. This operation is illustrated in Figure 1

Binary Mathematical Morphology

Binary morphology is often referred to as morphology on sets because it uses the lattice (P(E), ⊆) with E as the base space. In order to apply operators that rely on structuring elements, the action of a group (G, +) on E serves as an appropriate framework, and we employ it throughout this section.

Definition 1.18 (Binary dilation and erosion). Let A ⊆ E and B ⊆ G.

The binary dilation A ⊕ B of A by B is defined by:

A ⊕ B = {p ∈ E : (p + B) ∩ A = ∅} (1.11) = {a + b ∈ E : a ∈ A, b ∈ B} .
(1.12)

The binary erosion A ⊖ B of A by B is defined by:

A ⊖ B = {p ∈ E : p + B ⊆ A} . (1.13)
B is called structuring element. If we fix a structuring element B ⊆ G, we can define the binary dilation and erosion operators

δ B : P(E) → P(E) A → δ B [A] = A ⊕ B (1.14) ε B : P(E) → P(E) A → ε B [A] = A ⊖ B (1.15)
that form an adjunction.

Chapter 1. Mathematical Morphology

The fact that (P(E), ⊆) is a complemented lattice (where the complement of an element A ⊆ E is A c := {p ∈ E : p ∈ A}), gives an interesting duality property that link both operations.

Proposition 1.19. Let A ⊆ E and B ⊆ G. The following equations hold

(A ⊖ B) c = A c ⊕ Bc (1.16) (A ⊕ B) c = A c ⊖ Bc .
(1.17)

An intuitive illustration of how these operators transform a shape is given in Figure 1.4. The composition of a dilation with an erosion gives either an opening or a closing, depending on the order of composition. During the opening, the little circle outside the shape from Figure 1.4 has been filtered out, while during the closing, the little hole inside the shape has been filled in. This occurs because the circle and the hole are smaller (in terms of inclusion) than the structuring element. Despite these changes, the main shape of the object 1.2. Mathematical Morphology with Structuring Elements remains unchanged. This preservation of the general shape is one of the most interesting features of morphological filters: they filter out some parts of the object or its complement that are smaller than the structuring element, yet the overall shape remains unaffected.

Proposition 1.20. Let A ⊆ E and B ⊆ G. Then, 1. γ B := δ B • ε B is an opening, 2. ϕ B := ε B • δ B is a closing.

Functional Mathematical Morphology

Functional mathematical morphology, or mathematical morphology on functions, is based on the lattice (T E , ) (see Examples 1.5 2) of functions defined on a space E and with codomain T , where (T, ≤) is a complete lattice. Let (G, +) be a group that acts on E.

Flat morphology

Definition 1.21 (Flat dilation and erosion). Let f ∈ T E and B ⊆ G.

The flat dilation f ⊕ B of f by B is defined as:

f ⊕ B : E → T p → b∈ B f (p + b)
.

(1.18)

The flat erosion f ⊖ B of f by B is defined as:

f ⊖ B : E → T p → b∈B f (p + b) . (1.19)
B is called flat structuring element. If we fix a flat structuring element B ⊆ G, we can define the flat dilation and erosion operators

δ B : T E → T E f → δ B [f ] = f ⊕ B (1.20) ε B : T E → T E f → ε B [f ] = f ⊖ B (1.21)
that form an adjunction.

Opening and closing associated to flat erosion and dilation are defined as in Proposition 1.20.

It is interesting to note that flat morphology generalizes binary morphology through the usage of the characteristic function.

Proposition 1.22. We recall that

: P(E) → {0, 1} E A → A
is the canonical bijection between the subsets of E and their characteristic functions. We have that

A⊕B = A ⊕ B and A⊖B = A ⊖ B .
Let us illustrate the effect of these operators on an image, with E = Z 2 , G = (Z 2 , +), and T = {0, 1, ..., 255} (the 8-bit grayscale range).

The impact of dilation on an image can be observed in Figure 1.6 for various structuring elements. For each size5 k ∈ 2N + 1, the structuring element is the set

B = {(x, y) ∈ Z 2 : max{x, y} ≤ k-1
2 }. We observe that dark details are suppressed by the dilation and that the overall image gets brighter.

In Figure 1.7, we can see the effect of erosion on the same image. In this case, the bright details are suppressed and the overall image gets darker.

In Figure 1.8, we can observe the impact of the closing and opening defined by the compositions of erosion and dilation on the same image, with a fixed k = 7. For the opening, brighter details have disappeared, while the overall shape is preserved. On the other hand, the closing eliminates darker details (almost entirely); notably, the black line becomes a gray line. Despite these changes, the overall brightness level remains similar to the original image, with a slight shift towards darker (for opening) or brighter (for closing).

Mathematical Morphology with Structuring Elements

Grayscale morphology

In this section, we consider grayscale morphology, that is, morphology where the structuring element is a function. The standard way of defining grayscale morphology [START_REF] Heijmans | The algebraic basis of mathematical morphology i. dilations and erosions[END_REF][START_REF] Bloch | Mathematical morphology[END_REF][START_REF] Najman | Mathematical morphology: From theory to applications[END_REF] is by using a function b : G → T and defining the dilation and erosion of f : E → T by b as: ∀p ∈ E,

(f ⊕ b)(p) = sup x∈G (f (p -x) + b(x)) (f ⊖ b)(p) = inf x∈G (f (p + x) -b(x)) . (1.22)
This definition has a flaw: we cannot assume a priori that addition and subtraction operations exist in the lattice T . Furthermore, in common cases like T = R or Z, there is not canonical way to define these operations since the case ∞ -∞ is undefined. [START_REF] Bloch | Mathematical morphology[END_REF] propose to handle the case ∞ -∞ in a specific manner for each operator, stating that if f (q) + b(pq) takes the form ∞ -∞, it is set equal to -∞, and if f (q)b(qp) takes the form ∞ -∞, it is set equal to ∞.

In this work, we propose an alternative approach to address this issue. We introduce the concept of a residuated lattice triplet, or simply a residuated triplet, which is built upon the idea of a residuated lattice. This framework offers a solution to the problem of undefined operations.

This approach is in the line of using fuzzy sets and logic in MM [START_REF] Bloch | Fuzzy mathematical morphology[END_REF]. Specifically, we introduce analogs to conjunction and implication [START_REF] Deng | Grey-scale morphology based on fuzzy logic[END_REF] through the utilization of lattice multiplication and residuals. Our approach closely resembles the one employed in [START_REF] Maragos | Lattice image processing: A unification of morphological and fuzzy algebraic systems[END_REF], with the extension to triplets of lattices in place of considering a single lattice for scalars. Our adaptation draws upon results from the theory of fuzzy mathematical morphology [START_REF] Bloch | Duality vs. adjunction for fuzzy mathematical morphology and general form of fuzzy erosions and dilations[END_REF][START_REF] Bloch | Mathematical morphology on bipolar fuzzy sets: General algebraic framework[END_REF] due to the similarity between the two contexts.

The theory of residuated lattices started with the works of [START_REF] Krull | Axiomatische begründung der allgemeinen idealtheorie[END_REF][START_REF] Ward | Residuated lattices[END_REF][START_REF] Ward | Residuated lattices[END_REF] and has been developed during the 20 th century. For a modern approach, we suggest the survey of [START_REF] Jipsen | A survey of residuated lattices. Ordered Algebraic Structures[END_REF] or the monograph of [START_REF] Galatos | Residuated lattices: An algebraic glimpse at substructural logics[END_REF] We recall the definition of a residuated lattice.

Definition 1.23 (Residuated lattice). Let (L, ≤, ∨, ∧) be a lattice. Let • : L×L → L be an operation that is associative and has a neutral element 1 ∈ L, i.e., (L, • , 1) is a monoid. Then, (L, ≤, ∨, ∧, • , 1) is a residuated lattice if there exist two operations

/ : L × L → L and \ : L × L → L such that ∀x, y, z ∈ L, x • y ≤ z ⇔ y ≤ x\z ⇔ x ≤ z/y .
(1.23)

We call \ and / the right and left residuals of • , respectively.

Mathematical Morphology with Structuring Elements

We adapt this definition to the case where the operations • , \ and / are defined for different lattices and we call this a residuated triplet. Definition 1.24 (Residuated triplet). Let (L 1 , ≤ 1 ), (L 2 , ≤ 2 ) and (L 3 , ≤ 3 ) be three lattices. If there exists three operations

• : L 1 × L 2 → L 3 (a 1 , a 2 ) → a 1 • a 2 / : L 3 × L 2 → L 1 (a 3 , a 2 ) → a 3 /a 2 \ : L 1 × L 3 → L 2 (a 1 , a 3 ) → a 1 \a 3 such that ∀a 1 ∈ L 1 , ∀a 2 ∈ L 2 , ∀a 3 ∈ L 3 , a 2 ≤ 2 a 1 \a 3 ⇔ a 1 • a 2 ≤ 3 a 3 ⇔ a 1 ≤ 1 a 3 /a 2 (1.24)
we say that they form a residuated triplet denoted by (L 1 , L 2 , L 3 ). The operations / and \ are called respectively left and right residuals (with respect to • ).

Residuated triplets provide a versatile framework for defining greyscale morphology. We choose to set the theory for triplets as it encompasses the most general case (relevant for some applications to music). However, if we have L 1 = L 3 or L 1 = L 2 = L 3 , we will encounter particular cases that are also valuable. Notably, when L := L 1 = L 2 = L 3 and L has an identity element for the lattice multiplication, we obtain an actual residuated lattice.

We will now outline the conditions we desire for an operation • (referred to as lattice multiplication) to serve our purposes. Definition 1.25 (Lattice multiplication). Let (L 1 , ≤ 1 ), (L 2 , ≤ 2 ) and (L 3 , ≤ 3 ) be three complete lattices. We say that an operation

• : L 1 × L 2 → L 3 (a 1 , a 2 ) → a 1 • a 2 (1.25) is a lattice multiplication if ∀a 1 ∈ L 1 , ∀a 2 ∈ L 2 , ∀A 1 ⊆ L 1 , ∀A 2 ⊆ L 2 , 1. a 1 • ( 2 A 2 ) = 3 (a 1 • A 2 ) (Distributive property in L 2 ) 2. ( 1 A 1 ) • a 2 = 3 (A 1 • a 2 ) (Distributive property in L 1 )
where

A 1 • a 2 = {a • a 2 ∈ L 3 : a ∈ A 1 } and a 1 • A 2 = {a 1 • a ∈ L 3 : a ∈ A 2 }.
Chapter 1. Mathematical Morphology Corollary 1.26. Let (L 1 , ≤ 1 ), (L 2 , ≤ 2 ) and (L 3 , ≤ 3 ) be three complete lattices and

• : L 1 × L 2 → L 3 a lattice multiplication. Then, ∀a 1 , b 1 ∈ L 1 , ∀a 2 , b 2 ∈ L 2 , 1. a 1 ≤ 1 b 1 ⇒ a 1 • a 2 ≤ 3 b 1 • a 2 (Order preserving in L 1 ) 2. a 2 ≤ 2 b 2 ⇒ a 1 • a 2 ≤ 3 a 1 • b 2 (Order preserving in L 2 )
Proof. Let us prove 1 and the other is analogous. We know that a

1 ≤ 1 b 1 which is equivalent to say that b 1 = {a 1 , b 1 }. Then, b 1 • a 2 = {a 1 , b 1 } • a 2 Definition 1.25 2 = {a 1 • a 2 , b 1 • a 2 } ⇒ a 1 • a 2 ≤ 3 b 1 • a 2 .
These conditions ensure that the resulting operations satisfy the properties of residuals, thus forming a residuated triplet.

Definition 1.27 (Left and right residuals). Let (L 1 , ≤ 1 ), (L 2 , ≤ 2 ) and (L 3 , ≤ 3 ) be three complete lattices. Let • : L 1 × L 2 → L 3 be a lattice multiplication. We define the left and right residuals of • by

/ : L 3 × L 2 → L 1 (a 3 , a 2 ) → a 3 /a 2 := 1 a 2 ↓ a 3 \ : L 1 × L 3 → L 2 (a 1 , a 3 ) → a 1 \a 3 := 2 a 1 ↓ a 3 where a 2 ↓ a 3 := {a ∈ L 1 : a • a 2 ≤ 3 a 3 } and a 1 ↓ a 3 := {a ∈ L 2 : a 1 • a ≤ 3 a 3 }.
Proposition 1.28. The operations • , / and \ defined in Definition 1.25 and Definition 1.27 make (L 1 , L 2 , L 3 ) a residuated triplet.

Proof. We shall prove that ∀a

1 ∈ L 1 , ∀a 2 ∈ L 2 , ∀a 3 ∈ L 3 , a 2 ≤ 2 a 1 \a 3 ⇔ a 1 • a 2 ≤ 3 a 3 ⇔ a 1 ≤ 1 a 3 /a 2 . Let us prove that a 2 ≤ 2 a 1 \a 3 ⇔ a 1 • a 2 ≤ 3 a 3 . The prove of a 1 • a 2 ≤ 3 a 3 ⇔ a 1 ≤ 1 a 3 /a 2 is analogous. ⇐ Since a 1 • a 2 ≤ 3 a 3 then a 2 ∈ a 1 ↓ a 3 ⇒ a 2 ≤ 2 2 a 1 ↓ a 3 ⇒ a 2 ≤ 2 a 1 \a 3 . ⇒ 1.2. Mathematical Morphology with Structuring Elements a 2 ≤ 2 a 1 \a 3 Corollary 1.26 2 ⇒ a 1 • a 2 ≤ 3 a 1 • (a 1 \a 3 ) ⇒ a 1 • a 2 = a 1 • 2 a 1 ↓ a 3 Definition 1.25 1 = 3 (a 1 • a 1 ↓ a 3 )
where

a 1 • a 1 ↓ a 3 := {a 1 • a ∈ L 3 : a ∈ a 1 ↓ a 3 }. Since ∀a ∈ a 1 ↓ a 3 , a 1 • a ≤ 3 a 3 then a 3 is an upper bound of a 1 • a 1 ↓ a 3 and thus 3 (a 1 • a 1 ↓ a 3 ) ≤ a 3 . We have finally a 1 • a 2 ≤ 3 3 (a 1 • a 1 ↓ a 3 ) ≤ a 3 .
Now that we have a residuated triplet, we can define the grayscale dilation and erosion.

Definition 1.29 (Grayscale dilation and erosion). Let (T 1 , ≤ 1 ), (T 2 , ≤ 2 ) and (T 3 , ≤ 3 ) be three complete lattices that form a residuated triplet with the operations • , / and \, where • is a lattice multiplication.

Let

g ∈ T E 1 , b ∈ T G 2 and f ∈ T E 3 .
We define the greyscale dilation g ⊕ b of g by b as:

g ⊕ b : E → T 3 p → x∈G g(p + x) • b(-x)
.

(1.26)

We define the greyscale erosion f ⊖ b of f by b as:

f ⊖ b : E → T 1 p → x∈G f (p + x)/b(x)
.

(1.27)

b is called structuring function. If we fix a structuring function b ∈ T G 2 , we can define the flat erosion and dilation operators

ε b : T E 3 → T E 1 f → ε b [f ] = f ⊖ b (1.28) δ b : T E 1 → T E 3 g → δ b [g] = g ⊕ b .
(1.29)

Proposition 1.30. The greyscale erosion and dilation form an adjunction and thus are actual erosions and dilations.

Proof. The proof is a direct consequence of the structure of residuated triplet. See [START_REF] Bloch | Duality vs. adjunction for fuzzy mathematical morphology and general form of fuzzy erosions and dilations[END_REF] for a similar proof in the context of conjunctions and implications. This is the first example in this work of an erosion and dilation with different domains; indeed, the adjunction may be represented by the following diagram:

δ b [g] f ∈ L E 3 L E 1 ∋ g ε b [f ] b ∈ L G 2 ε b δ b
This diagram also illustrates the anti-extensivity and extensivity properties of the composition of erosion and dilation (resulting in opening and closing, respectively), a consequence of the adjunction. By following the diagram, we have

δ b [ε b [f ]] f and g ε b [δ b [g]] .
It is worth specifying what constitutes the lattice multiplication and residual in the standard definition of greyscale morphology exposed in Equation (1.22). The residuated triplet (which is actually a residuated lattice) is (R, R, R) with the extended addition, defined as

+ : R × R → R (x, y) → x + y =      -∞ if (x, y) = (-∞, ∞) -∞ if (x, y) = (∞, -∞)
x + y otherwise (1.30) and the extended subtraction, defined as

-: R × R → R (x, y) → x -y =      ∞ if (x, y) = (∞, ∞) ∞ if (x, y) = (-∞, -∞)
xy otherwise .

(1.31)

The extended addition happens serves as lattice multiplication (with identity element being 0) while the extended subtraction acts as its left residual (the right residual is analogous).

Mathematical Morphology with Structuring Elements

Although the concept of a residuated triplet is not particularly relevant in this context (since a residuated lattice would suffice), in Chapter 2 we will present a residuated triplet with different L 1 , L 2 , L 3 when presenting our model for piano rolls. For presenting the derived operators of MM we will restrict ourselves to the structure of residuated lattice (instead of triplet) to remain closer to the existing literature.

Derived Operators

So far, we have introduced the four fundamental operators of mathematical morphology: dilation and erosion, which are the basic operators, as well as opening and closing, which are morphological filters. However, it is important to note that Mathematical Morphology offers a range of other operators that provide additional capabilities. In the following sections, we present some of these additional operators, that will be employed in this work.

Hit-or-miss transform

The hit-or-miss transform is an operator from mathematical morphology derived from erosion. In its binary version, it uses the complementary of a set imposing specific conditions on it. As an intuition, the hit-or-miss transform with two structuring elements C and D imposes that the translation of C, p + C, should be contained in the set and the translation of D, p + D, should be contained in the complement. The following definition presents this concept formally.

A ⊛ (C, D) = {p ∈ E : p + C ⊆ A and p + D ⊆ A c } (1.32) = (A ⊖ C) ∩ (A c ⊖ D) .
(1.33)

We call C and D the foreground and background structuring elements, respectively.

Extending the hit-or-miss operator to greyscale morphology is not straightforward, primarily because the hit-or-miss is not an increasing operator. To address this challenge, several approaches have been proposed; some notable works in this area include [START_REF] Ronse | A lattice-theoretical morphological view on template extraction in images[END_REF][START_REF] Soille | Advances in the analysis of topographic features on discrete images[END_REF][START_REF] Soille | Morphological image analysis: Principles and applications[END_REF]Barat et al., 2003a[START_REF] Barat | Pattern matching using morphological probing[END_REF], which are summarized and unified in [START_REF] Naegel | Grey-level hit-or-miss transforms-part II: Application to angiographic image processing[END_REF].

Chapter 1. Mathematical Morphology

In this work, we rely on the hit-or-miss transform proposed in [START_REF] Soille | Advances in the analysis of topographic features on discrete images[END_REF], called there the unconstrained hit-or-miss, but in a version closer to the one exposed in [START_REF] Naegel | Grey-level hit-or-miss transforms-part II: Application to angiographic image processing[END_REF]. We call it flat hit-or-miss. Definition 1.32 (Flat hit-or-miss). Let (T, ∨, ∧, +, -, 0) be a residuated lattice 6 . Let f ∈ T E and C, D ⊆ G with C ∩ D = ∅. The flat hit-or-miss of f by (C, D), denoted by f ⊛ (C, D), is defined by: ∀p ∈ E,

(f ⊛ (C, D))(p) = (f ⊖ C)(p) -(f ⊕ Ď)(p) ∨ 0 .
(1.34)

Our definition is slightly different from the one in [START_REF] Naegel | Grey-level hit-or-miss transforms-part II: Application to angiographic image processing[END_REF] by two aspects: first, we only define it for flat structuring elements and second, we handle the subtraction of infinities by the residuation operation; this leads, for instance, to a different value of ∞ -∞: in [START_REF] Naegel | Grey-level hit-or-miss transforms-part II: Application to angiographic image processing[END_REF] it is set to 0 and we set it to ∞ (when dealing with the lattice (R, ≤)). This yields to the same output for the thinning operation.

Thinning

We now introduce the thinning operation, which will prove to be highly useful in Chapter 3. We present together both the binary and greyscale cases, each relying on its respective hit-or-miss transform.

Ψ (C, D) = Ψ -(Ψ ⊛ (C, D)) (1.35)
wheredenotes either set difference or left residuation, depending on the nature of

Ψ. Let (C 1 , D 1 , C 2 , D 2 , ..., C n , D n ) ∈ P(G)
2n be a sequence of structuring elements that we call templates. The application of successive elementary thinnings

((((Ψ (C 1 , D 1 )) )(C 2 , D 2 ) ...) (C n , D n )) (1.36)
is called a thinning.

If we apply this operation iteratively until stability is reached, we obtain what is known as an ultimate thinning, denoted by ∞ .

Mathematical Morphology with Structuring Elements

Top-hat

The top-hat operations fall under the category of residues in mathematical morphology [START_REF] Najman | Mathematical morphology: From theory to applications[END_REF]. These operations involve taking the difference between an input image and its opening (white top-hat) or between the closing and the image (black top-hat) [START_REF] Soille | Morphological image analysis: Principles and applications[END_REF]. As opening is anti-extensive and closing is extensive, the resulting values are positive.

Definition 1.34 (Top-hat). Let (T, ∨, ∧, +, -, 0) be a residuated lattice. Let f ∈ T E and B ⊆ G.

The white top-hat of f by B is given by: ∀p ∈ E,

WTH B [f ](p) = f (p) -γ B [f ](p) if f (p) = γ B [f ](p) 0 if f (p) = γ B [f ](p) .
(1.37)

The black top-hat of f by B is given by: ∀p ∈ E,

BTH B [f ](p) = ϕ B [f ](p) -f (p) if ϕ B [f ](p) = f (p) 0 if f (p) = ϕ B [f ](p) . (1.38)
We present the definition in multiple cases because when using the residuation -, we have ∞ -∞ = ∞ instead of the desired7 ∞ -∞ = 0.

Skeleton

Regarding the skeleton, we use Lantuéjoul formula [START_REF] Lantuéjoul | La squelettisation et son application aux mesures topologiques des mosaiques polycristallines[END_REF].

Definition 1.35 (Skeleton). Let A ⊆ E. Then, the skeleton of A, denoted by

S(A), is the set S(A) = i∈N ε i B 1 [A] \ γ B 1 [ε i B 1 [A]] (1.39)
where B 1 ⊆ G is the elementary structuring element8 .

This skeleton is equal to the set of the centers of maximal balls (according to the same distance) included in A.

Geodesic Transformations

Up to this point, we have introduced operations that involve a single input and one or two structuring elements. Geodesic transformations, however, require two different inputs: a marker and a mask. The marker is either expanded (dilation) or shrunk (erosion), and the mask imposes limitations to this expansion or shrinking, hence the term "geodesic". We now present both geodesic dilation and erosion, as well as their corresponding reconstructions, which lead to and opening and a closing, respectively. We define the operations in the functional case since the others are restrictions of it.

Geodesic dilation

Definition 1.36 (Geodesic dilation). Let f, g ∈ T E . The geodesic dilation of size 1 of the marker f with respect to the mask g, denoted by δ 1 g [f ], is defined by:

δ 1 g [f ] = δ B 1 [f ] ∧ g (1.40)
where B 1 ⊆ G is the unit ball of the grid.

The geodesic dilation of size n of f with respect to g, denoted by δ n g [f ], is defined recursively by:

δ n g [f ] = δ 1 g [δ n-1 g [f ]] (1.41)
where δ 0 g [f ] = f ∧ g. Geodesic dilation exhibits some useful properties.

Proposition 1.37. Geodesic dilation is increasing in both arguments, extensive in the marker argument and anti-extensive in the mask argument, i.e.,

1. ∀f 1 , f 2 , g 1 , g 2 ∈ T E , f 1 f 2 ∧ g 1 g 2 ⇒ δ 1 g 1 [f 1 ] δ 1 g 2 [f 2 ], 2. ∀f, g ∈ T E , f ∧ g δ 1 g [f ], 3. ∀f, g ∈ T E , δ 1 g [f ] g.

Geodesic erosion

Definition 1.38 (Geodesic erosion). Let f, g ∈ T E . The geodesic erosion of size 1 of the marker f with respect to the mask g, denoted by ε 1 g [f ], is defined by:

ε 1 g [f ] = ε B 1 [f ] ∨ g (1.42)
where B 1 ⊆ G is the elementary structuring element.

Mathematical Morphology with Structuring Elements

The geodesic erosion of size n of f with respect to g, denoted by ε n g [f ], is defined recursively by:

ε n g [f ] = ε 1 g [ε n-1 g [f ]] (1.43)
where ε 0 g [f ] = f ∨ g. Geodesic erosion exhibits some useful properties.

Proposition 1.39. Geodesic erosion is increasing in both arguments, anti-extensive in the marker argument and extensive in the mask argument, i.e.,

1. ∀f 1 , f 2 , g 1 , g 2 ∈ T E , f 1 f 2 ∧ g 1 g 2 ⇒ ε 1 g 1 [f 1 ] ε 1 g 2 [f 2 ], 2. ∀f, g ∈ T E , ε 1 g [f ] f ∨ g, 3. ∀f, g ∈ T E , g ≤ ε 1 g [f ].

Morphological reconstruction

In practical situations, geodesic dilation and erosion are typically applied iteratively until stability is achieved. This iterative process enables us to define morphological reconstructions, specifically the reconstruction by dilation and reconstruction by erosion.

Definition 1.40 (Reconstruction by dilation). Let f, g ∈ T E . Then, the reconstruction by dilation of the mask g from the marker f , denoted9 by δ ∞ f [g], is defined by:

δ ∞ f [g] = δ N g [f ] (1.44)
where

N ∈ N is such that δ N g [f ] = δ N +1 g [f ]
. When E is finite, N always exists.

Definition 1.41 (Reconstruction by erosion). Let f, g ∈ T E . Then, the reconstruction by erosion of the mask g from the marker f , denoted by ε ∞ f [g], is defined by:

ε ∞ f [g] = ε N g [f ] (1.45)
where

N ∈ N is such that ε N g [f ] = ε N +1 g [f ]
. When E is finite, N always exists.

Proposition 1.42. The reconstruction by dilation is an opening and the reconstruction by erosion is a closing.

Implementation of Mathematical Morphology Operators

In this section, we will discuss the implementation of the mathematical morphology operators discussed in the previous sections.

We have chosen to use Python 3 [START_REF] Rossum | Python 3 reference manual: Python documentation manual part 2[END_REF] for the implementations and applications in this thesis. There are several libraries in Python that implement MM operators; we found four: SciPy [START_REF] Virtanen | SciPy 1.0: Fundamental algorithms for scientific computing in python[END_REF], scikit-image [START_REF] Walt | Scikit-image: Image processing in python[END_REF], OpenCV [START_REF] Bradski | The OpenCV library[END_REF] and Kornia [START_REF] Riba | Kornia: An open source differentiable computer vision library for PyTorch[END_REF].

However, these libraries do not fully align with our needs and thus we have chosen to implement our own libraries: PyMorpho 10 , serving as a general-purpose MM library, and nnMorpho 11 , which is equipped with a PyTorch engine for GPU acceleration and seamless integration with neural network architectures.

In the following section, we expose the main implementation considerations specific to our work. Before delving into the computational aspects, let us recapitulate the mathematical objects that we intend to implement computationally. We require a space E equipped with an additive group (G, +) that acts on it. Furthermore, we need a lattice (T, ≤) to serve as the range for functional morphology. For implementing greyscale morphology, we must also have a residuated triplet (T 1 , T 2 , T 3 ).

Implementation Considerations

There are several aspects that should be decided when implementing a library for mathematical morphology operators, namely: the data structure, the data types, the operators families (binary, flat, grayscale), the management of the origin, the management of the border, the dimensions and the topology of the underlying space.

Data structure

The elements on which we apply mathematical morphology operators can be mathematically represented as either sets (A ⊆ E) or functions (f ∈ T E ). In a computational context, both sets and functions can be modeled using arrays.

An array is a collection of elements, typically of the same data type, that are indexed by a set of integers. Each element in the array can be accessed using its corresponding index. Arrays allow for efficient storage and retrieval of data, making them well-suited for representing sets and functions in computational settings.

Implementation of Mathematical Morphology Operators

For instance, a set can be represented as a binary array where each element of the base space E corresponds to an index in the array. If an element is present in the set, its corresponding entry in the array is marked as true; otherwise, it is marked as false. In the case of functions, the value of the array at a specific index represents the value of the function.

Data types

The data type represents our lattice (T, ≤); indeed, a data type that serves our purposes need to be equipped with an order and with arithmetic operations, as well as bottom and top elements. The most frequent data types are listed in These data types come with the order operator <= and the arithmetic operators + and -, which will serve as our lattice multiplication and left residuation, respectively. We recall that the lattice multiplication will have the additive form since it is the most common in the greyscale MM literature.

It is important to note that critical cases of addition and subtraction, such as ∞ -∞ in floating-point numbers and overflow in integers, are not handled as they should in some implementations. For instance, in the case of SciPy, these operations may result in nan values in floating-point numbers or in modular overflow in integers, as exemplified by 255 + 2 = 1 in uint8 data type.

Operators families

We have presented three families of operators: binary, flat and grayscale. Usually, libraries distinguish between binary and greyscale morphology, and consider flat Chapter 1. Mathematical Morphology morphology as a particular case of greyscale morphology 12 . In our libraries, we will also distinguish between binary and greyscale, but we do this way because of different reasons.

In principle, by the data types of the input data (the input array and the structuring element) we should be able to infer the family of the operator, namely 13 Input data type Structuring element data type Family Boolean Boolean Binary Numeric Boolean Flat Numeric Numeric Greyscale .

However, for our applications we require another type of morphology for numeric data types, that actually corresponds to the residuated triplet presented in the following example.

Example 1.43. Let (T, ≤) be a lattice and ({0, 1}, ≤) be the boolean lattice. Then, the residuated triplet ({0, 1}, T, T ) is defined with the following lattice multiplication 14 :

• : {0, 1} × T → T (b, x) → b • x = x if b = 1 ⊥ if b = 0 (1.46)
which has the corresponding left residuation

/ : T × T → {0, 1} (x, y) → x/y = 1 if y ≤ x 0 if y ≤ x .
(1.47)

This residuated triplet will prove to be useful for certain musical applications. From a computational perspective, it corresponds to choosing a numeric range for the data type of the input and the structuring element, and producing a Boolean output. This functionality can be encapsulated in two modules, namely binary and greyscale:

Module 12 Some of them only implement flat morphology. 13 In the following table, we use the term numeric to refer to all types except the Boolean type.

14 In this case we use multiplicative notation for the lattice multiplication and residuation since it is more appropriated.

Implementation of Mathematical Morphology Operators

Origin

Since the domain of the structuring element is a group (G, +), there is a notion of origin. The origin is the identity element for the group (in our case, it is usually denoted as 0). When using arrays, they do not come with an associated origin (which is not a problem for a space E). As a result, we need to specify the origin of the array.

In common applications, the origin is typically set as the central element of the structuring element array, which is usually assumed to be of odd size. However, this approach is not sufficient in general and do not suit our musical applications.

In our implementation, we add a parameter for specifying which pixel is the origin, set by default to the center.

Border

Another important element to consider is the border. When the structuring element is not limited to a single pixel, translating the structuring element near the border of the input array can cause an overflow of indices. This is a consequence of working with a finite subset of the space.

To manage this overflow, there are two classical approaches:

1. the Euclidean approach: in this approach, we consider that the input array is extended with bottom elements outside the actual image. This is equivalent to considering that our image is defined in the entire space with compact support.

2. the geodesic approach: in this approach, we do not consider the values outside the input array when taking the infimum or supremum. This approach is called geodesic because of its similarities with geodesic operators.

In practice, this overflow issue only affects the computation of erosion: the dilation operation is not affected since extending with bottom elements does not alter the output of a supremum.

Dimensions

Even though our main focus will be on using two-dimensional inputs, it is important to note that mathematical morphology is defined for every space E with a group (G, +) acting on it, including cases where E = G = R d or Z d . In particular, mathematical morphology is extensively used in processing 3D scan images.

Topology of the underlying space

Every Mathematical Morphology library considers the arrays as being embedded in a Euclidean space like R d or Z d . However, in Chapter 5, in Section 5.3 we will use a space E ≃ Z × Z 12 , which has cylindrical topology. This has a deep impact on the implementation of the operators since the translation in the direction of Z 12 has no border and rotates the structuring element.

Although this behavior can be simulated by stacking copies of the input array, our libraries offer the flexibility to consider other topologies, particularly the cylindrical (or toroidal) topology.

Computational Model

In this section, we present the computational model implemented in our libraries, PyMorpho and nnMorpho. We will only discuss the abstraction needed for applying mathematical morphology with structuring elements, which forms the core of PyMorpho. The low-level algorithms in nnMorpho are mere adaptations of these abstractions, implemented as C++ extensions for PyTorch, including CUDA kernels.

The computational model presented below, with a syntax similar to that of Python, exposes the objects needed for implementation. We employ the terms shift, point, and level to refer to elements of the group, space, and lattice, respectively. The term image pays homage to the origins of MM but should be understood as the object upon which MM operations can be performed.

We expose one by one the classes we use with their attributes and methods 15 and give a brief explanation of them.

Shift:

__neg__(self) -> Shift Shift corresponds to an element of the group x ∈ G. We override the operatorfor being able to refer to its opposite -x, needed in the definition of dilation.

Group: shift_type: Type

[Shift] __iter__(self) -> Iterator[Shift]
Group corresponds to the group itself, (G, +). We make it iterable for being able to go through each one of its elements, that are Shifts. We also include the shift_type as an attribute to link the group with its shift type16 .

Implementation of Mathematical Morphology Operators

Point: __add__(self, shift: Shift) -> Point

Point corresponds to an element of the space p ∈ E. We override the operator + that takes a Shift as second argument for being able to refer to compute the value

p + x. Space: point_type: Type[Point] __iter__(self) -> Iterator[Point]
Space corresponds to the space itself, E. We make it iterable for being able to go through each one of its elements, that are Points. We also include the point_type as an attribute to link the space with its point type 16 .

Level: __add__(self, other: Level) -> Level __sub__(self, other: Level) -> Level __le__(self, other: Level) -> bool
Level corresponds to an element of the lattice t ∈ T . We override the operators + andfor having a lattice multiplication and left residuation 17 . Moreover, we impose the levels to be comparable by the operator <=, even is this is not necessary for the implementation of operators. Lattice corresponds to the lattice itself, (T, ≤). We require it to have both the bot and top elements, that are Levels and to define two methods, supremum and infimum. We also include the level_type as an attribute to link the lattice with its level type 16 . In addition, we override the operators * and / to be able to determine the resulting lattice in lattice multiplication and left residuation.

Chapter This abstraction allows us to implement the two fundamental operators of Mathematical Morphology: dilation and erosion. The general (suboptimal) algorithms for these operators are presented in Algorithms 1 and 2, and based on Equation (1.22):

(f ⊕ b)(p) = sup x∈G (f (p -x) + b(x)) (f ⊖ b)(p) = inf x∈G (f (p + x) -b(x))
It is important to note that this abstraction reflects the fact that we do not necessarily require a full group (G, +) acting on E for defining MM operators. The only essential aspect we need is a notion of neighborhood, represented by the methods add and neg from Point and Shift, which can be transformed into add and sub associated with Point.

These algorithms, along with some examples of their applications, are implemented in the PyMorpho library. However, while these algorithms demonstrate decent performance when applied to small arrays (such as piano rolls), their implementation in pure Python with a high level of abstraction renders them less suitable for practical applications involving large arrays (as is often the case with spectrograms).

Implementation of Mathematical Morphology Operators

Algorithm 1 Algorithm for dilation To address this limitation, we have developed the nnMorpho library, which includes GPU acceleration for some of the most commonly used cases.

In Figure 1.9, we present an overview of nnMorpho's performance and features in comparison to other libraries, presented as a poster in DGMM [START_REF] Romero-García | nnMorpho, a PyTorch library for mathematical morphology operators[END_REF]. nnMorpho surpasses all other libraries when utilized with GPU acceleration. Furthermore, it offers extensive customization options.

Feature SciPy scikit-image OpenCV kornia nnMorpho PyTorch ✗ ✗ ✗ ✓ ✓ Non-flat structuring element ✓ ✓ ✗ ✓ ✓ GPU capability ✗ ✗ ✗ ✓ ✓ Border parameter ✗ ✗ ✗ ✓ ✓ Cylindric topology ✗ ✗ ✗ ✗ ✓ Batch processing ✗ ✗ ✗ ✓ ✓ More than 2D ✓ ✗ ✗ ✗ ✗ Computation of gradients ✗ ✗ ✗ ✓ ✓ Origin parameter ✓ ✓ ✗ ✓ ✓
✓ ✓ ✗ ✓ ✓ GPU capability ✗ ✗ ✗ ✓ ✓ Border parameter ✗ ✗ ✗ ✓ ✓ Cylindric topology ✗ ✗ ✗ ✗ ✓ Batch processing ✗ ✗ ✗ ✓ ✓ More than 2D ✓ ✗ ✗ ✗ ✗ Computation of gradients ✗ ✗ ✗ ✓ ✓ Origin parameter ✓ ✓ ✗ ✓ ✓

Chapter 2

Time-Frequency Representations of Music

In this chapter, our objective is to present various time-frequency representations of music that allow us to apply mathematical morphology to them. We will refer to each representation as a musical space, denoted as M, and it will have the form:

M = A T ×F = {f : T × F → A} (2.1)
where T represents time, F represents frequency, and A represents amplitude. Each element f in the musical space M is a function that maps a pair of time and frequency (t, ξ) to an amplitude value f (t, ξ).

If we endow A with a complete lattice structure and we define a group (G T ×F , +) that acts on T × F , we are able to apply greyscale MM with structuring elements. In order to achieve this, we consider separately time and frequency in a first instance, create groups that act on each of them, and then couple them through the Cartesian product.

In Section 2.1, we explore the various choices available for representing time, frequency and amplitude, and the resulting mathematical structures that arise from these choices. Subsequently, we delve into the two primary representations that we will employ in our applications: spectrograms, that are covered in Section 2.2, and piano rolls, discussed in Section 2.3.

Algebraic Structures for Musical Spaces

The algebraic structures underlying musical spaces play a crucial role in defining how music can be represented and analyzed. In this section, we explore the different choices available for each element in a musical space. By examining various combinations of these choices, we will uncover the diverse representations that can be achieved for musical data in this manner.

For the domain of the functions f ∈ M, which represents the time-frequency plane, we use a space T × F with a group (G T ×F , +) acting on it (see Section 1.2.1). This choice is related to the Generalized Music Interval model proposed by [START_REF] Lewin | Generalized musical intervals and transformations[END_REF] It is worth mentioning that other approaches, such as the use of the differential geometry paradigm proposed by [START_REF] Tymoczko | Generalizing musical intervals[END_REF][START_REF] Tymoczko | In quest of musical vectors[END_REF], would offer more refined possibilities, particularly when considering spaces with borders. However, for the spaces considered in this work, the use of a group defined for the entire space is adequate and sufficiently flexible.

For the codomain of f , we use a complete lattice (A, ≤) that models the amplitude. This endows (M, ) with the complete lattice structure induced by the pointwise order.

Time

In order to achieve the desired algebraic structure for representing time, we introduce a space T , the elements of which are referred to as timestamps, and a group (G T , +), the elements of which are known as time shifts. This combination will enable us to define the necessary operations for time manipulation and analysis within the musical space.

Indeed, time can be measured using different units, and we can represent each of these units using different mathematical structures. Here we consider three ways of representing time:

1. time measured in seconds, that can be modeled using the set of real numbers R, 2. time measured in a computational unit (such as samples or MIDI ticks), that can be modeled using the set of integers Z, 3. time measured in wholes1 , that can be modeled using the set of rational numbers Q.

Measuring time in seconds

For measuring time in seconds, we assume that time is continuous and use (G T , +) = (R, +) as the group for time shifts. The space of timestamps, denoted as T s , represents time in seconds elapsed from a particular reference point, such as the start of a piece of music. This leads to a straightforward bijection2 between T s and (R, +) which, as shown in Proposition 1.16, induces a group action of (R, +) on T s . The timestamps can be represented in the format3 hh:mm:ss.ms, following the ISO standard ISO, 2019a, or using abbreviations like mm:ss or ss.ms. This split into hours, minutes, and seconds makes the action of (R, +) on T s more complex, as it requires successive Euclidean divisions. Nevertheless, we choose this representation as opposed to providing the full number of seconds, which may be overwhelming in a long excerpt.

Measuring time in computational units

When working with digital representations of music, such as .wav or .midi files, we use a discrete representation of time. In this context, we consider the group for time shifts as (G T , +) = (Z, +). For representing timestamps, we use T 1 = Z, where t ∈ T 1 counts the number of units elapsed from the start of the file.

In the context of .wav files, time is measured in samples, and the sampling frequency provides the conversion factor between the number of samples and seconds. The common sampling frequencies are 44.1 kHz or 48 kHz.

For .midi files, time is measured in ticks. To convert between ticks and seconds, we need the information about the ticks per beat and the beats per minute. These parameters allow us to determine the tempo and perform the conversion from ticks to seconds. We will delve into this conversion process in more detail in Section 2.3.2.1.

Measuring time in wholes

Measuring time inside a musical score is indeed a critical task, and different approaches can be taken based on the specific requirements and goals of the analysis.

One possible approach is to use the tempo information provided in the score to transform all note durations into seconds and then measure time using seconds as Chapter 2. Time-Frequency Representations of Music the unit. This method can be valid and straightforward when tempo information is available and consistent throughout the score. However, it does have some limitations. For example, if the tempo indication is not provided or if there are frequent tempo changes throughout the score, this approach may become less practical and relevant.

Alternatively, another approach is to take advantage of the existing abstraction of note values in the score. To do that, we measure durations, i.e., time shifts, using note values. Some common note values are  , , , and , and further refined note values exist. They are defined by the relation

 = 2 = 4 = 8 = 16 .
(2.2)

As pointed out by Equation (2.2), all the note values are measured in terms of the whole note  . In fact, we can re-arrange Equation (2.2) as

= 1 2  , = 1 4  , = 1 8  , = 1 16  . (2.3)
It is natural then to associate note values with the rational numbers Q, and consider then the group (Q, +) to be the group (G T , +). Notice that durations can be different from negative powers of 2; if we want to define the duration that is equal to 3 4  , the symbol might be used. The use of tuplets allows us to consider base powers different from 2 and, in theory, we are able to create a specific note value that has the duration p q  for every p q ∈ Q. Whereas in previous measuring options we decided to fix a starting point and measure time from it, in this case we try to stick to musicological canons; thus, we avoid naming timestamps by counting the elapsed wholes from the starting point.

When musicians need to specify a particular time in the score, they use bars. The first bar is given the value 1, and if an anacrusis is present in the score, we assign it the value 0. To be more precise in the specification of time, musicians use beats to specify timestamps inside a bar. A beat represents the number of beats4 elapsed from the start of the bar, starting at 1. For example, in a 4 4 time signature, where the beat is , there are four beats

(1, 2, 3, and 4), each of them one beat apart from the previous. We set the beat of a time signature to be the note value corresponding to the denominator of the time signature. For instance, the beat of 4 4 is and the beat of 6 8 is . However, it is important to note that this convention deviates slightly from musical standards, where is the usual beat associated to 6 8 . We adopt this simplified convention for practical reasons.

Whereas this notation may be enough for communicating timestamps between musicians, we introduce a refined notation for musicological purposes. In this notation, we add an additional element called the offset, which allows us to fully determine every timestamp in a piece. The offset represents the displacement from the beat and starts at 0 (finally in alignment with computational conventions). By incorporating the offset, we can specify timestamps in a more precise and flexible manner, enabling accurate representation and analysis of musical data.

The time space that we use is T p q  , where p q is the corresponding time signature5 . It can be defined as

T p q  = {(m, b, o) ∈ N × N * × Q : b ∈ {1, 2, ..., p}, 0 ≤ o < 1 q } . (2.4)
This refined representation allows for precise specification of timestamps within a musical piece, remaining close to common musical nomenclature. Let us provide an example to clarify this notation.

Example 2.1. We consider the beginning of the Violin Sonata No.1 in G minor, BWV 1001 from Johann Sebastian Bach (see Figure 2.1). We use the space T  := T 4 4 . The following list 6 shows when the onsets of the consecutive notes7 occur:

1. Gm → (1, 1, 0), 2. F5 → (1, 2, 1 32 ), 3. E  5 → (1, 2, 2 32 ), 4. D5 → (1, 2, 3 32 ), 5. C5 → (1, 2, 4 32 ), 6. B  4 → (1, 2, 5 32 ), 7. A4 → (1, 2, 6 32 ), 8. B  4 → (1, 2, 7 32 ), 9. G4 → (1, 2, 15 64
). It is worth to mention that this system does not lead to a straightforward action between the group (Q, +) and the space T p q  . It can be compared to the hexadecimal approach of minutes and hours used for time. While measuring time in wholes and keeping it simple would have been an option, we chose to adhere to musicological standards.

Frequency

For representing frequency, we also require a space F and a group (G F , +) acting on it. In this context, the elements of the space are pitches or chromas, depending on the associated space, while the elements of the group are frequency shifts. As in the case of time, this choice is closely related to the Generalized Music Interval framework [START_REF] Lewin | Generalized musical intervals and transformations[END_REF].

Frequency can be measured using different units; we explore two options:

1. frequency measured in Hertz, 2. frequency measured in semitones.

Depending on the specific use cases, the groups used for each of these units may vary. In the following, we detail these options.

Measuring frequency in Hertz

In signal processing, the unit commonly used to measure frequency is Hertz, which corresponds to cycles per second. If we assume that the frequency space is continuous, we can define F Hz = R as the space of frequencies measured in Hertz. We naturally assign (R, +) as the group acting on it.

However, in some cases, the frequency varies logarithmically, such as when using the Constant-Q transform (CQT) (see Section 2.2.1.2). In such scenarios, the space of frequencies is still measured in Hertz but restricted to positive frequencies, leading us to define F log Hz = R + * . In this case, the group acting on it is (R + * , •).

Measuring frequency in semitones

In Western classical music, the common practice is to measure frequency in semitones.

In this case, the set of frequencies F will consist of the pitches used in Western Classical Music, denoted by N . These pitches are a combination of the octave on 2.1. Algebraic Structures for Musical Spaces which they are played and one of the twelve chromas8 :

N 12 = {C, C  , D, E  , E, F, F  , G, A  , A, B  , B} .
(2.5)

We assume the enharmonic equivalences, such as C  = D  and so on. The pitches are then defined as:

N = N 12 × Z . (2.6)
While theoretically, we could write (N, n), where N ∈ N 12 and n ∈ Z, we use the notation [pitch][octave] instead, according to the notation suggested by the Acoustical Society of America [START_REF] Young | Terminology for logarithmic frequency units[END_REF]. For instance, the central C of the piano is notated as C4, and the pitch with a frequency of 440 Hz is called A4, following the ISO standard [START_REF] Iso | Acoustics -standard tuning frequency (standard musical pitch)[END_REF] In this case, the group G F is set to (Z, +). It acts on pitches by shifting the pitch upwards (for positive shifts) or downwards (for negative shifts). For instance, C4 + 1 = C  4, C4 + 12 = C5 and C4 -5 = G3. Since there is a bijection9 between N and Z, we can also define the subtraction between pitches; the result is the shift needed to translate one pitch into the other. For instance, C4 -C3 = 12 and C4 -E4 = -4.

It is important to note that we can extend semitones continuously, allowing (R, +) to be the group for frequency shifts. We call the space of semitones F st = R, where st stands for semitone. This extension would grant access to finer intervals, such as quarter tones and other divisions of the octave, and also allows us to explore different temperaments beyond the equal temperament.

Lastly, let us discuss another space for representing frequencies: the chromas. As mentioned before, we denote this space as N 12 . It is particularly interesting from a musical perspective because many considerations, especially harmonic ones, are made up to an octave. In this case, the group G F for the chromas would be Z 12 . We can also extend this representation continuously to the space of continuous chromas using (12T, +) as the group, where T represents the one dimensional torus T = R ⧸ Z .

Lattice Structure for the Amplitude Range

For the musical space to be a complete lattice with pointwise order, we need the amplitude range A to be a complete lattice. In the following, we have a look at the different options for A that we will use throughout this work.

Chapter 2. Time-Frequency Representations of Music

We split them into two classes: continuous lattices (whose cardinal is strictly superior to ℵ 0 ) and discrete ones (with cardinal less or equal to ℵ 0 ). All the lattices that we present are complete lattices.

Continuous lattices

We use the following continuous lattices:

([0, 1], ≤) (R, ≤) (R + , ≤) (R -, ≤)
These continuous lattices are used for spectrograms, with their specific choice depending on the transform and the unit (e.g., no unit or dB).

It is noteworthy that all of these lattices are isomorphic. In fact, we have the following increasing bijections:

tan : [0, 1] → R x → tan (x -1 2 ) • π (2.7) 20 log 10 : [0, 1] → R - x → 20 log 10 (x)
(2.8)

20 log 10 : R + → R x → 20 log 10 (x)
.

(2.9)

These bijections preserve the order, ensuring that these lattices have the same structure.

We equip these lattices with a lattice multiplication and thus a structure of residuated lattice by using the canonical residuation presented in Definition 1.27.

For [0, 1], we employ the classical multiplication •, which is an internal operation with an absorbing bottom element. Its corresponding residuation is the classical division / where the case x/0 is set to 1, a consequence of the definition of residuation.

Indeed, ∀x ∈ [0, 1], x/0 = {a ∈ [0, 1] : a • 0 ≤ x} = 1.
Similarly, for R + , we use multiplication with the absorbing bottom element, meaning 0 • ∞ = 0. The residuation here behaves similarly to [0, 1], with x/0 = ∞.

In the case of R, we use the extended addition defined in Equation (1.30).

Finally, for R -, we use the addition, which exhibits similar behavior to R, but with the residuation (the subtraction) resulting in:

-∞ -(-∞) = {a ∈ R -: a + (-∞) ≤ -∞} = R -= 0 .

Binary and ternary lattices

The binary and ternary lattices are A 2 = {0, 1} and A 3 = {⊥, •, ×}, respectively. The former, that has the usual order, is called the Boolean lattice. The later, whose order is ⊥ < • < ×, is not given in numeric form for distinguishing it from the Boolean one (as they will appear together often). It is called the rhythmic range because it helps us to model rhythms. Its elements are called silence (⊥), sustain (•) and onset (×). As discussed in Section 4.1.1, we will use this range to define the notion of rhythm (see Definition 4.1).

The Boolean lattice can be combined with any other lattice to form a residuated triplet by defining the lattice multiplication and left residuation10 by:

• : A 2 × A → A (a, b) → b if a = 1 ⊥ if a = 0 / : A × A → A 2 (a, b) → 1 if a ≥ b 0 if a ≥ b .
In particular, the combination of the Boolean lattice with the rhythmic range will be extensively used in Chapters 4 and 5.

Dynamics lattices

In music, the intensity with which a note is played is often expressed using dynamics.

These dynamics are represented in scores by symbols such as p, mf, f, and so on.

We define the lattice of score dynamics as follows:

D pf = {⊥ < ... < ppp < pp < p < mp < ∅ < mf < f < ff < fff < ... < ⊤} (2.10)
where the ∅ dynamic means that there is no dynamic specified.

When working with a MIDI file, we do not have the symbolic representation of intensity (dynamics) as in traditional music scores. Instead, we use a numeric representation known as MIDI dynamics.

The MIDI dynamics lattice is defined as follows:

D 128 = ({0, 1, ..., 127}, ≤) . (2.11)
Each level in this lattice indicates a specific intensity level, with 0 representing silence.

Amplitudes for piano rolls

In this section, we present the most refined range we have developed for representing MIDI files and scores as piano rolls in a musical space. This is, in fact, the first example of the application of a residuated triplet in which none of the triplet's elements are repeated.

For the sake of generality, we use D for either D pf or D 128 , depending on whether we want to model scores or MIDI files. We then proceed to describe two different ways of coupling D with A 3 :

1. the pianistic dynamics, 2. the sustained dynamics.

The pianistic dynamics represent the dynamics that can be performed on a piano, where once a note is hit, there is no further control over its intensity. The resulting lattice is given by:

A P D = D ∪ {•} (2.12)
where

⊥ D < • < d, ∀d ∈ D \ {⊥ D }.
We claim that • is smaller than every other dynamic to capture the concept of piano scores. In piano notation, whenever a note appears, it should be played at the indicated dynamic level. Consider the following example:

ff pp c & w Ó İn
this example, the first note is hit with ff intensity, and the second note is hit with pp intensity while the previous one is still sustained. In this case, the last two beats would sound at pp, since the last dynamic takes priority. Although this may seem unusual, it is allowed in musical staff notation.

The sustained dynamics represent the dynamics that can be performed on those music instruments that can control the intensity of a note after hitting it (like strings or winds). It is defined as

A D = (A 3 × D) ⧸ ∼ (2.13)
where

(a, b) ∼ (a ′ , b ′ ) ⇔      (a, b) = (a ′ , b ′ ) a = a ′ = ⊥ A 3 b = b ′ = ⊥ D (2.14)
We associate to A D the pointwise order. We can also represent it by

A D = {⊥} ∪ {•, ×} × D . (2.15)
This range allows in particular the execution of crescendos and diminuendos; the following excerpt ff pp might now be expressed by the sequence of amplitudes

(×, pp), (•, p), (•, mp), (•, mf), (•, f), (×, ff) .
Moreover, we are able to give a sense to the dynamic fp through (×, f), (•, p).

Whereas the pianistic dynamics are totally ordered and can be visualized as

⊥ D < • < ... < pp < p < mp < ∅ < mf < f < ff < ... < ⊤ D
the sustained dynamics are a partial order that is not total, and may be visualized as

⊥ < < ... ... < < (×, p) (•, p) < < < (×, mp) (•, mp) < < < (×, ∅) (•, ∅) < < < (×, mf) (•, mf) < < < (×, f) (•, f) < < < ... ... < < (×, ⊤ D ) (•, ⊤ D ) < .
We can endow the lattice of sustained dynamics with a residuated triplet structure. Let us define the lattice multiplication

• : A 3 × D → A D (a, b) → [(a, b)] (2.16)
where [(a, b)] is the equivalence class of (a, b) under the equivalence relation ∼.

It is trivial to see that it is a lattice multiplication, since ∀A ⊆ A 3 , ∀b ∈ D,

A • b = A, b = {(a, b) ∈ A D : a ∈ A} = (A • b) .
Its residuation is defined by

/ : A D × D → A 3 ((a, b), c) → a if b ≤ c ⊥ if b ≤ c \ : A 3 × A D → D (a, (b, c)) → c if a ≤ b ⊥ if a ≤ b . (2.17)

Representing Music with Spectrograms

In this section, we explore how to represent music using spectrograms, a widely used and valuable time-frequency representation. Spectrograms serve as a means to transform music from a signal format (e.g., encoded in .wav format) into an timefrequency representation, i.e., a function f within a musical space M.

We will present two types of spectrograms and a generalization of both:

1. a spectrogram generated from the Short-time Fourier transform (STFT), 2. a spectrogram generated from the Constant-Q transform (CQT), 3. a combined approach, the Time-frequency-scale transform (TFST), which encompasses both STFT and CQT.

All of these spectrograms are Fourier-based transformations, and we will use the notations introduced in the Preamble and described in Appendix B.

The basic idea behind spectrograms is to transform a function f : T → R, which represents a wave of musical sound, into a function S f : T × F → A. The choices of T , F, and A are dependent on the specific spectrogram and the nature of T .

In the following sections, we first define the continuous transformations from which spectrograms can be extracted in Section 2.2.1. Next, we adapt these continuous transformations to the discrete framework in Section 2.2.2, which is used for actual computations. Finally, in Section 2.2.3, we show how to compute a spectrogram using these transformations.

Continuous Definitions

Short-time Fourier transform

The Short-time Fourier transform (STFT) is a widely used operator in signal analysis and processing, particularly in audio signals. We use the definitions from [START_REF] Gröchenig | Foundations of time-frequency analysis[END_REF] and the notations presented in the Preamble and the Appendix B.

Definition 2.2 (Short-time Fourier transform). Let f ∈ L ∞ (R; C) and let g ∈ L 1 (R; C).
The Short-time Fourier transform of f with respect to the window g is defined by

STFT g [f ] : R × R → C (τ, ω) → R f (t)g(t -τ )e -2πıtω dt .
(2.18) Some properties of the STFT are stated in the next proposition.

Proposition 2.3. Let g ∈ L 1 (R; C). Then, the operator

STFT g : L ∞ (R; C) → L ∞ (R × R; C) f → STFT g [f ]
(2. [START_REF]an abbreviation for Q ∪ {-∞, ∞}[END_REF] is linear and

||STFT g [f ]|| ∞ ≤ ||f || ∞ ||g|| 1 (2.20)
which makes it a continuous operator from

L ∞ (R; C) to L ∞ (R × R; C). In particular, if ||g|| 1 = 1, ||STFT g [f ]|| ∞ ≤ ||f || ∞ .
(2.21)

If we limit ourselves to the functions in L 2 (R; C), we can express the STFT using the translation and modulation operators, defined as

T τ f (t) = f (t -τ ) and M ω f (t) = f (t) • e 2πıtω , respectively (detailed in Appendix B). Proposition 2.4. Let f, g ∈ L 2 (R; C). Then, ∀(τ, ω) ∈ R × R, STFT g [f ](τ, ω) = f, M ω T τ g .
(2.22)

An important property, called the fundamental identity of time-frequency analysis in [START_REF] Gröchenig | Foundations of time-frequency analysis[END_REF], that we will use in Chapter 3, is the following.

Proposition 2.5 (Fundamental identity of time-frequency analysis

). Let f, g ∈ L 2 (R; C). Then, ∀(τ, ω) ∈ R × R, STFT g [f ](τ, ω) = e -2πıωt STFT F [g] [F [f ]](ω, -τ ) .
(2.23)

Proof. Since this property is not proved in [START_REF] Gröchenig | Foundations of time-frequency analysis[END_REF] we include its proof here:

STFT g [f ](τ, ω) Proposition 2.4 = f, M ω T τ g = F [f ], F [M ω T τ g] = F [f ], T ω M -τ F [g] = f, e 2πıτ ω M -τ T ω g = e -2πıτ ω f, M -τ T ω g = e -2πıτ ω STFT F [g] [F [f ]](ω, -τ )
To finish with a brief introduction to the STFT, let us recall the inversion formula.

Proposition 2.6 (Inversion formula for the STFT

). Let g, γ ∈ L 2 (R; C) such that g, γ = 0. Then, ∀f ∈ L 2 (R; C) f = 1 γ, g R R STFT g [f ](τ, ω)M ω T τ γ dω dτ . (2.24)
This leads us to the definition of the inverse STFT. , is given by:

∀t ∈ R, iSTFT[S](t) = 1 γ, g R R S(τ, ω)M ω T τ γ(t) dω dτ .
(2.25)

Constant-Q transform

The Constant-Q transform (CQT) is an operator that is well-suited for music representation due to its logarithmic frequency resolution, differing from STFT, which has a linear frequency resolution. This characteristic is particularly important for music since the musical tuning is based on the concept of octave intervals, which is a logarithmic feature.

The CQT was initially proposed by [START_REF] Youngberg | Constant-q signal analysis and synthesis[END_REF], and later an efficient method for its computation was introduced [START_REF] Brown | Calculation of a constant q spectral transform[END_REF]. As mentioned in [START_REF] Schörkhuber | Constant-q transform toolbox for music processing[END_REF], the CQT can be viewed as a particular case of a wavelet transform. 54 2.2. Representing Music with Spectrograms Definition 2.8 (Continuous wavelet transform). Let ψ ∈ L 2 (R; C) such that R ψ(t) dt = 0. Then, the continuous wavelet transform (CWT) is defined by

W : L 2 (R; C) → L ∞ (R × R + * ; C) f → W [f ] : R × R + * → C (τ, σ) → 1 √ σ R f (t)ψ( t-τ σ ) dt . (2.26)
We can express the continuous wavelet transform in terms of the scalar product. To do that, we recall that the 2-unitary dilation operator is

D 2 σ f (t) = 1 √ σ f t σ (detailed in Appendix B). Proposition 2.9. Let f, ψ ∈ L 2 (R; C). Then, ∀(τ, σ) ∈ R × R + * , W ψ [f ](τ, σ) = f, T τ D 2 σ ψ (2.27) and ||W ψ [f ]|| ∞ ≤ ||f || 2 • ||ψ|| 2 .
(2.28)

Proof. ∀(τ, σ) ∈ R × R + * , f, T τ D 2 σ ψ = R f (t) • T τ D 2 σ ψ(t) dt = R f (t) • 1 √ σ ψ t -τ σ dt = 1 √ σ R f (t) • ψ t -τ σ dt = W ψ [f ](τ, σ) . Then, ∀(τ, σ) ∈ R × R + * , |W ψ [f ](τ, σ)| = | f, T τ D 2 σ ψ | ≤ ||f || 2 • ||T τ D 2 σ ψ|| 2 = ||f || 2 • ||ψ|| 2 which implies ||W ψ [f ](τ, σ)|| ∞ ≤ ||f || 2 • ||ψ|| 2 .

Chapter 2. Time-Frequency Representations of Music

The Constant-Q transform can now be expressed as a particular case of a wavelet transform, where ψ(t) = g(t)e 2πıt , with g being a window function. However, we change the 2-unitary dilation to the 1-unitary dilation (presented in Appendix B), in order to obtain an inequality involving the infinity norm, as shown in Equation (2.20).

Definition 2.10 (Constant-Q transform). Let f ∈ L ∞ (R; C). Let g ∈ L 1 (R; C). We define the Constant-Q transform of f with the window g as CQT g [f ] : R × R + * → C (τ, σ) → 1 σ R f (t) • g( t-τ σ )e -2πı t σ dt (2.29)
While the CQT is a time-scale transform, we can interpret it as a time-frequency transform with ξ = 1 σ . We can express the Constant-Q transform as a scalar product.

Proposition 2.11. Let f ∈ L ∞ (R; C). Let g ∈ L 1 (R; C). Then, ∀(τ, σ) ∈ R × R + * , CQT g [f ](τ, σ) = f, M 1 σ T τ D 1 σ g (2.30) and ||CQT g [f ](τ, σ)|| ∞ ≤ ||f || ∞ • ||g|| 1 . (2.31) Proof. ∀(τ, σ) ∈ R × R + * , f, M 1 σ T τ D 1 σ g = R f (t) • M 1 σ T τ D 1 σ g(t) dt = R f (t) • e 2πıt 1 σ 1 σ g t -τ σ dt = 1 σ R f (t) • g t -τ σ e -2πı t σ dt = CQT g [f ](τ, σ) . Then, ∀(τ, σ) ∈ R × R + * , |CQT g [f ](τ, σ)| = | f, M 1 σ T τ D 1 σ g | ≤ ||f || ∞ • ||M 1 σ T τ D 1 σ g|| 1 = ||f || ∞ • ||g|| 1 which implies ||CQT g [f ](τ, σ)|| ∞ ≤ ||f || ∞ • ||g|| 1 .

Time-frequency-scale transform

By using the scalar product formulation of the STFT and the CQT, we can observe their similarity. This similarity can be summarized as a three-dimensional transformation that we call Time-frequency-scale transform (TFST). The TFST combines the time-frequency approach of the STFT with the time-scale approach of the CQT, resulting in a three-dimensional representation.

While the TFST is not a time-frequency representation, it fits perfectly within the postulates of this work, where we need a space (in this case three-dimensional) and a group acting on it. The group would be (R, +)

× (R, +) × (R + * , •).
This transformation is implicitly present in deep-learning applications under different names such as multi-scale spectral loss [START_REF] Engel | DDSP: Differentiable digital signal processing[END_REF] or multi-resolution spectral distance [START_REF] Wang | Neural source-filter waveform models for statistical parametric speech synthesis[END_REF]. In these cases, an intermediate TFST is computed and then contracted into a single scalar value, which is used as a loss function for gradient descent.

Previous works have explored similar directions [START_REF] Levine | Multiresolution sinusoidal modeling for wideband audio with modifications[END_REF][START_REF] Bonada | Automatic technique in frequency domain for near-lossless timescale modification of audio[END_REF][START_REF] Bonada | Audio time-scale modification in the context of professional postproduction[END_REF][START_REF] Dorran | Audio time-scale modification[END_REF][START_REF] Juillerat | Enhancing the quality of audio transformations using the multi-scale short-time fourier transform[END_REF][START_REF] Mateo | Bridging the gap between the short-time fourier transform (STFT), wavelets, the constant-q transform and multi-resolution STFT[END_REF], proposing two-dimensional representations with varying time-frequency resolutions instead of using the third dimension.

The TFST can be viewed as a way to overcome the time-frequency uncertainty principle at the cost of introducing an extra dimension in the representation. Definition 2.12 (Time-frequency-scale transform). Let g ∈ L 1 (R; C). We define the Time-frequency-scale transform with window g as

TFST g : L ∞ (R; C) → L ∞ (R × R × R + * ; C) f → TFST g [f ] : R → C (τ, ω, σ) → f, T τ M ω D 1 σ g . (2.32)
We can establish an inequality for the TFST, similar to those of the STFT and the CQT. This inequality implies that the TFST is a continuous operator between

L ∞ (R; C) and L ∞ (R × R × R + * ; C). Proposition 2.13. Let f ∈ L ∞ (R; C). Let g ∈ L 1 (R; C). Then, ||TFST g [f ]|| ∞ ≤ ||f || ∞ • ||g|| 1 .
(2.33)

Proof. ∀(τ, ω, σ) ∈ R × R × R + * , |TFST g [f ](τ, ω, σ)| = | f, T τ M ω D 1 σ g | ≤ ||f || ∞ • ||T τ M ω D 1 σ g|| 1 = ||f || ∞ • ||g|| 1 57 Chapter 2. Time-Frequency Representations of Music then ||TFST g [f ]|| ∞ ≤ ||f || ∞ • ||g|| 1
As mentioned, we can express the STFT and the CQT as particular cases of the TFST.

Proposition 2.14. Let f ∈ L ∞ (R; C). Let g ∈ L 1 (R; C). Then, ∀(τ, ω) ∈ R × R, STFT g [f ](τ, ω) = e -2πıωτ TFST g [f ](τ, ω, 1) (2.34) and ∀(τ, σ) ∈ R × R + * , CQT g [f ](τ, σ) = e -2πı τ σ TFST g [f ](τ, 1 σ , σ) . (2.35) Proof. ∀(τ, ω) ∈ R × R, STFT g [f ](τ, ω) = f, M ω T τ g = f, e 2πıτ ω T τ M ω g = e -2πıτ ω f, T τ M ω D 1 1 g = e -2πıωτ TFST g [f ](τ, ω, 1) ∀(τ, σ) ∈ R × R + * , CQT g [f ](τ, σ) = f, M 1 σ T τ D 1 σ g = f, e 2πıτ 1 σ T τ M 1 σ D 1 σ g = e -2πıτ 1 σ f, T τ M 1 σ D 1 σ g = e -2πı τ σ TFST g [f ](τ, 1 σ , σ)
In addition, we can also see express the TFST as a convolution.

Proposition 2.15. Let f ∈ L ∞ (R; C). Let g ∈ L 1 (R; R). Then, ∀(τ, ω, σ) ∈ R × R × R + * , TFST g [f ](τ, ω, σ) = (f * M ω D 1 σ g * )(τ ) (2.36) 58 2.2. Representing Music with Spectrograms Proof. ∀(τ, ω, σ) ∈ R × R × R + * , (f * M ω D 1 σ g * )(τ ) = f, T τ (M ω D 1 σ g * ) * = f, T τ M ω D 1 σ (g * ) * = f, T τ M ω D 1 σ g = TFST g [f ](τ, ω, σ) . Corollary 2.16. Let f ∈ L ∞ (R; C). Let g ∈ L 1 (R; R). Then, ∀(τ, ω) ∈ R × R, STFT g [f ](τ, ω) = e -2πıτ ω (f * M ω g * )(τ ) (2.37) and, ∀(τ, σ) ∈ R × R + * , CQT g [f ](τ, σ) = e -2πı τ σ (f * M 1 σ D 1 σ g * )(τ ) . (2.38)
Representing the STFT, CQT, and TFST as convolutions proves to be advantageous for computational purposes. While the traditional approach of using Fast Fourier Transform (FFT) [START_REF] Cooley | An algorithm for the machine calculation of complex fourier series[END_REF] is usually preferred due to its logarithmic complexity, modern GPU computations have shown superior performance by using highly parallelizable operations like convolutions [START_REF] Cheuk | nnAudio: An onthe-fly GPU audio to spectrogram conversion toolbox using 1d convolutional neural networks[END_REF].

Discrete Definitions

To perform actual computations, we need to transition from the continuous representation to the discrete one. We consider an input time series f = (f[n]) N -1 n=0 of size N ∈ N * with a sampling frequency ξ s ∈ R, expressed in Hz.

Discrete STFT

The discrete STFT is then defined as follows.

Definition 2.17

(Discrete STFT). Let N, J ∈ N * . Let f = (f[n]) N -1 n=0 ∈ C N , let g = (g[j]) J-1 j=0 ∈ C J . Let j 0 ∈ {0, 1, .
.., J -1} be the index corresponding 11 to the 11 This parameter, often overlooked in the literature by assuming it to be 0 or M /2, is actually essential for achieving good compatibility between the continuous and discrete STFT. This aspect is related to what has been discussed in Chapter 1 regarding the distinction between a set of points E and a group of shifts (G, +); in this case, the indices of f and STFT[f] correspond to points, while the indices of g correspond to time shifts. element of g that acts as 0 of the group (R, +). The discrete STFT of f with window g and with K ∈ N * frequency bins is defined by

∀m ∈ {0, 1, ..., N -1}, ∀k ∈ {0, 1, ..., K -1}, STFT g [f][m, k] = m-j 0 +J-1 n=m-j 0 f[n]g[n -m + j 0 ]e -2πın k K (2.39) j=n-m+j 0 = J-1 j=0 f[m + j -j 0 ]g[j]e -2πı(m+j-j 0 ) k K , (2.40)
where we assume that f[n] = 0, ∀n ∈ {0, 1, ..., N -1}.

The discretization points of the STFT are t m = m ξs , and ξ k = kξ s with ξ s ∈ R being the sampling frequency.

We can also introduce another parameter H ∈ N * , known as the hop size, and subsample the formula by considering STFT g [f][mH, k].

This definition may differ slightly from other common definitions, especially regarding the parameter j 0 . In many cases, there is no specific attention given to the computational problem of aligning the window with the signal. However, since windows are often concentrated in the center bin, we aim to align this bin (indexed by j 0 ) with the f[m] value. Additionally, we want the oscillatory factor to have the value 1 at this point. This consideration also resolves the problem of how to pad the signal, i.e., by adding j 0 zeroes at the beginning and J -1j 0 at the end.

Discrete CQT

The discretization of the CQT that we use is based on [START_REF] Schörkhuber | Constant-q transform toolbox for music processing[END_REF]. However, there are some modifications in our approach. The support of the window function is defined as t ∈ [-1 2 , 1 2 ] in our case, whereas Schörkhuber and Klapuri, 2010 use t ∈ [0, 1]. Additionally, we center the complex exponential at t = 0, whereas they center it at t = 1 2 . Moreover, we allow g to be a complex-valued function, even though for practical cases we will often use a real-valued window function.

Definition 2.18 (Discrete CQT). Let f = (f[n]) N -1 n=0 ∈ C N . Let g ∈ C ∞ (R; C) with supp(g) = [-1 2 , 1 2 ]. Let K ∈ N * .
The CQT of f with the window12 g is defined by

∀m ∈ {0, 1, ..., N -1}, ∀k ∈ {0, 1, ..., K -1}, CQT g [f][m, k] = 1 N k m+⌊ N k 2 ⌋ n=m-⌊ N k 2 ⌋ f[n]g n -m N k e -2πın ξ k ξs (2.41) = 1 N k 2⌊ N k 2 ⌋ j=0 f j + m -⌊ N k 2 ⌋ g j -⌊ N k 2 ⌋ N k e -2πı(m+j-⌊ N k 2 ⌋) ξ k ξs (2.42)
where t m = m ξs are the discretization points of the time, ξ k = ξ 0 2 k B the discretization points of the frequency, and

N k = qξs ξ k (2 1 B -1)
, with the following parameters:

• ξ s ∈ (0, ∞): the sampling frequency,

• ξ 0 ∈ (0, ξ s ): the lowest frequency,

• B ∈ N * : the number of bins per octave,

• q ∈ (0, 1): the scaling factor (inverse of the oversampling), typically equal 1.

With the given parameters, the resulting quality factors Q k for each band 13 are expressed as:

Q k := ξ k ∆ξ k = N k ξ k ∆ωξ s ≈ q ∆ω(2 1 B -1) := Q
where ∆ξ k denotes the -3 dB bandwidth of the frequency response of the timefrequency atom:

(a k [j]) N k j=0 := 1 N k g j N k - 1 2 e 2πıj ξ k ξs N k j=0 (2.43)
Additionally, ∆ω is the -3 dB bandwidth of the main lobe of the spectrum of the window function g, which is approximately equal to 1.50 frequency bins for the Hann window, for example.

It may be interesting to outline the relation between the dilation factor σ from the continuous CQT and all these parameters. From the frequency component, we derive that

1 σ k = ξ k ξ s (2.44)
where (σ k ) K-1 k=0 is the sampling of the variable σ from the continuous dilation space R + * . From that, we obtain the relation between N k and σ k , that is

N k = q 2 1 B -1 σ k = λσ k (2.45) with λ = q 2 1 B -1 .
This relation implies a relation between the window function from the continuous case (denoted by g c ) and the window function from the discrete case (denoted by g d ); by identification from Equations (2.29) and (2.41), we have that ∀k ∈ {0, 1, ..., K}, ∀t ∈ R,

1 σ k g c t σ k = 1 N k g d t N k = 1 λσ k g d t λσ k and hence g c t σ k = 1 λ g d t λσ k t ′ := t σ k ⇔ g c (t ′ ) = 1 λ g d t ′ λ ⇔ g c = D 1 λ g d .

Discrete TFST

The TFST requires the discretization of three variables: time, frequency, and scale. We present its discrete version based on the convolution formula (Equation (2.36)).

Definition 2.19 (Discrete TFST). Let f = (f[n]) N -1 n=0 ∈ C N . Let g ∈ C ∞ (R; C) with supp(g) = [-1 2 , 1 2 ]. Let K, L ∈ N * .
The TFST of f with the window g is defined by 62 2.2. Representing Music with Spectrograms

∀m ∈ {0, 1, ..., N -1}, ∀k ∈ {0, 1, ..., K -1}, ∀l ∈ {0, 1, ..., L -1}, TFST g [f][m, k, l] = 1 N l m+⌊ N l 2 ⌋ n=m-⌊ N l 2 ⌋ f[n]g n -m N l e -2πı(n-m) ξ k ξs (2.46) = 1 N l 2⌊ N l 2 ⌋ j=0 f j + m -⌊ N l 2 ⌋ g j -⌊ N l 2 ⌋ N l e -2πı(j-⌊ N k 2 ⌋) ξ k ξs (2.47)
where ξ s ∈ R + * is the sampling frequency, and (ξ k ) K-1 k=0 and (N l ) L-1 l=0 represent the discretization of the frequency and scale variables, respectively. The discretization points of the time are given by t m = m ξs . In this transform, we have the flexibility to choose the level of discretization for the frequency and scale variables according to our needs. For instance, if we set a fixed value N l = N for all l ∈ {0, 1, ..., L -1} and vary the frequency linearly as

ξ k = k K ξ s for k ∈ {0, 1, ..., K -1}
, we recover the STFT14 . On the other hand, if we establish an inverse relationship between the scale and frequency, we obtain the CQT 14 . The time variable is always discretized linearly, as it is directly linked to the variable of f, but we can subsample it using a hop size H ∈ N * , as mentioned earlier.

For computing the STFT and the CQT, we use the Python/PyTorch library nnAudio [START_REF] Cheuk | nnAudio: An onthe-fly GPU audio to spectrogram conversion toolbox using 1d convolutional neural networks[END_REF]. This library is highly efficient, especially for the CQT, as it leverages GPU acceleration and performs computations using convolutions. As there is no standardized theory for the TFST, we have developed a custom implementation based on the principles of nnAudio.

Spectrograms

Up to this point, we have been working with transformations that produce complexvalued functions. This complex representation is useful in signal processing, as it makes the operators linear, but is inappropriate to our needs where we need the amplitude A to be a complete lattice 15 .

To address this limitation, we adopt a common approach used in this kind of analysis: dropping the phase information and keeping only the modulus. This results in a representation called a spectrogram, which is the square modulus of the original transformation. Using the square instead of the modulus itself is a standard practice, often referred to as the power spectrogram due to its relations with power in electronics and acoustics.

For each transformation, we define a corresponding spectrogram by taking the square modulus of its complex value:

SPEC STFT g = |STFT g | 2 , SPEC CQT g = |CQT g | 2 , SPEC TFST g = |TFST g | 2 .
After taking the square modulus of the complex values, our transforms are no longer linear, and the codomain has changed from C to R + . As a result, we can now consider the usual order in R + and set our amplitude as A = (R + , ≤).

It is a common practice to represent the amplitudes of a spectrogram in a logarithmic scale, particularly in decibels (dB). For an output value z ∈ C, the equation is given by: |z| 2 = 10 log 10 |z| 2 dB = 20 log 10 |z| dB .

(2.48)

As mentioned in Equation (2.9), the transformation of a positive value into decibels is an isomorphism between ordered sets.

Furthermore, it transforms products into sums. This is an interesting feature for our considerations in Mathematical Morphology, as the lattice multiplication we consider for greyscale morphology is either • or +, depending on whether we are working with a linear or logarithmic scale in the amplitude range.

Finally, to further restrict our amplitude range, we use Equations (2.20), (2.31) and (2.33). We consider an input function f with ||f || ∞ = 1, which is equivalent to ∀t ∈ R, -1 ≤ f (t) ≤ 1, a common condition for audio signals. Additionally, we choose the window function g with ||g|| 1 = 1, which is a normalization condition that can be applied to every window. With these conditions, we have the following properties.

Proposition 2.20. Let f ∈ L ∞ (R; C) with ||f || ∞ = 1. Let g ∈ L 1 (R; C) with ||g|| 1 = 1. Then, ||SPEC STFT g [f ]|| ∞ ≤ 1, ||SPEC CQT g [f ]|| ∞ ≤ 1 and ||SPEC TFST g [f ]|| ∞ ≤ 1 . Proof.
Since the STFT and the CQT are particular cases of the TFST with a complex exponential factor with modulus 1, it is enough to prove the result for the TFST.

∀(τ, ω, σ) ∈ R × R × R + * , SPEC TFST g [f ](τ, ω, σ) = |TFST g [f ](τ, ω, σ)| 2 Equation (2.33) ≤ ||f || ∞ • ||g|| 1 2 = 1 which implies that ||SPEC TFST g [f ]|| ∞ ≤ 1.

Representing Music with Piano Rolls

The fact that the spectrograms are bounded by 1 implies that when they are expressed in decibels, the amplitude range is R -= [-∞, 0]. This will be the amplitude range we use for spectrograms in the following.

In Figure 2.2, we present two spectrograms, one from STFT and another from CQT. Both spectrograms are displayed in logarithmic scale for both frequency and amplitude.

Additionally, Figure 2

Representing Music with Piano Rolls

While spectrograms are well-suited for audio signals, they are not directly applicable to symbolic representations of music such as MIDI files or scores. Indeed, since symbolic representations cannot be modeled as continuous functions f : T → R, the standard transformation formulas used for spectrograms are not applicable in this context. Nevertheless, MIDI files and scores are already available in a format that straightforwardly yields a time-frequency representation. This format, commonly known as a piano roll, can be effectively modeled as a function within a musical space, denoted P : T × F → A, as we will explore further in the following sections.

Piano Roll

Let us begin with a brief overview of piano rolls. Originally, piano rolls were a mechanical means of recording music before audio recording methods existed. They were used to preserve performances of great musicians from the early 20 th century, and many archives, such as the Stanford University Piano Roll Archive 16 , still house these historical records.

In modern music contexts, the concept of a piano roll has evolved to encompass any piano roll-like representation of music. Specifically, it refers to a two-dimensional representation of musical notes, where one axis represents time and the other axis represents the notes of a piano. In this thesis, we adopt this extended notion of a 16 https://exhibits.stanford.edu/supra 66 2.3. Representing Music with Piano Rolls piano roll and provide a formal definition.

It is important to note that there is no universally accepted definition of what a piano roll should be. The term is used informally, assuming that we all refer to this particular representation. [START_REF] Temperley | The cognition of basic musical structures[END_REF] exposes several piano rolls and points out the need of marking the onset of each note (which we will do by using the rhythmic lattice, in our case).

Thus, we present a formal definition of a piano roll.

Definition 2.21 (Piano roll). Let T be a set representing time. Let F be a countable set representing pitches 17 . Let (A, ≤) be a complete lattice. Then, a piano roll P of N ∈ N notes is an element of A T ×F such that

P = N n=1 ν n (2.49)
where ∀n ∈ {1, 2, ..., N }, ν n ∈ A T ×F and supp(ν n ) = [s n , e n [×{ξ n } for some s n , e n ∈ T and ξ n ∈ F . Each function ν n ∈ A T ×F is called a note, with s n being its start, e n being its end and ξ n being its pitch.

In this definition, a piano roll P is represented as a supremum of individual notes, each defined by a function ν n . The specific way we define a note varies depending on the amplitude range A that we use.

In Chapters 4 and 5, we will use another related concept that is the activations piano roll, which we abbreviate as activations.

Definition 2.22 (Activations piano roll). Let T be a set representing time. Let F be a countable set representing pitches. Let (A, ≤) be a either A 2 or one of the presented dynamics D. Then, an activations piano roll A of N ∈ N notes is an element of A T ×F such that

A = N n=1 α n (2.50)
where ∀n ∈ {1, 2, ..., N }, α n ∈ A T ×F and supp α n = {t n } × {ξ n } for some t n ∈ T and ξ n ∈ F . Each function α n ∈ A T ×F is called an activation, with t n being its timestamp and ξ n being its pitch.

It can be a bit obscure what this concept means. We present two intuitions: in the MM framework, the activations are either the input of a dilation or the output of an erosion; they mean when a structuring element should be replicated (in the case of dilation) or may be present (erosion). In the framework of music representations, each activation means that we activate a motive at a particular time-frequency point (with, eventually, a dynamic information).

Before diving into the different piano roll representations, we introduce a derived notion that will help us handle more complex musical data.

Definition 2.23 (Piano roll stack). Let T , F and A three sets that may represent a piano roll. Let I be a countable set of indexes. We call a piano roll stack a sequence S = (P i ) i∈I of piano rolls belonging to A T ×F .

We have

S : T × F × I → A (t, ξ, i) → S(t, ξ, i) = P i (t, ξ)
.

(2.51)

This mathematical object allows us to consider several piano rolls that share time, frequency, and amplitude. It is particularly useful for handling MIDI files with several tracks and scores with multiple instruments. While we presented for piano rolls, the same principle applies to activations piano rolls.

In the following sections, we explain how we define piano rolls and stacks for each input format: in Section 2.3.2, we define it for MIDI files, and in Section 2.3.3, we define it for scores. Finally, in Section 2.3.4, we introduce a derived version of a piano roll: the chroma roll.

Representing MIDI Files as Piano Rolls

MIDI is a widely used format for sharing musical data, and is particularly well adapted for being represented as piano roll with minimal information loss. In the following sections, we specify the various choices we can make for T , F, and A when dealing with MIDI files. These choices allow us to customize the piano roll representation to suit different applications and requirements.

Time

Inside a MIDI file, time is expressed using a unit called tick. Specifically, a MIDI file consists of a series of messages, and each message is separated by a time interval of ∆ ∈ N ticks from the previous one.

To convert ticks into seconds for playing a MIDI file, two parameters are involved:

1. The ticks_per_beat parameter: an integer value that represents the number of ticks per beat for the entire MIDI file. The term "beat" in this context refers to the quarter note value.

2. The microseconds_per_beat parameter: an integer value that can be changed by a set_tempo message within the file. It represents the number of microseconds per beat.

Using these two parameters, ticks can be converted into seconds to play the MIDI file. The microseconds_per_beat parameter allows for tempo changes within the file, providing flexibility in the playback speed.

Let us present the conversion formulas from ticks to seconds. We call ∆t tk ∈ N a time interval expressed in ticks and ∆t s , ∆t µs ∈ R a time interval expressed in seconds and microseconds, respectively. Let ∆t b ∈ N be a time interval expressed in beats (quarter notes).

If we call tpm ∈ N * the value of ticks_per_beat and mpb ∈ N * the value of microseconds_per_beat, we have that

∆t b • tpb = ∆t tk ∆t b • mpb = ∆t µs ∆t µs = 10 6 ∆t s .
(2.52)

From these formulas, we can deduce:

∆t s = ∆t µs • 10 -6 = ∆t b • mpb • 10 -6 = ∆t tk • mpb tpb • 10 -6 = spt • ∆t tk .
(2.53)

where seconds_per_tick = mpb tpb 10 -6 ∈ Q is the conversion parameter. This combination of parameters results in two methods for measuring time within a MIDI file: using the tick unit or measuring it in seconds after conversion. These two approaches may not always yield the same results due to the possibility to change the tempo.

To illustrate this phenomenon, Figure 2.4 shows a musical score, which has been converted into a MIDI file using a music editor software. The MIDI file is then represented as two different piano rolls: one with time measured in ticks (Figure 2.4b) and the other with time measured in seconds (Figure 2.4c). We observe that the Largo and Adagio (with tempos set to = 48 and = 42, respectively) occupy much more time in the seconds version than in the ticks version, in contrast to the Allegro (with a tempo of = 242), which occupies much less time.

As a result, the choice of measuring time in ticks or seconds can significantly impact the representation of the MIDI file in piano roll format. Depending on the application's requirements, either method may be preferred.

We associate the corresponding spaces and groups for each representation:

Chapter 2. Time-Frequency Representations of Music

Frequency

The space we choose for frequency is F = N (the space of pitches) since it is trivially related with the MIDI numbers; we associate 60 to C4, 69 with A4, etc. This space is associated with the group (Z, +), where the shift 1 means shift the note one semitone up. For instance, A4 + 1 = B  4.

In actual MIDI files, the pitches are comprised between 0 and 127, i.e., from C-1 to G9.

Notice that, under these conditions, A  = B  . This is a consequence of encoding frequencies by MIDI numbers inside a MIDI file.

Representing Music with Piano Rolls

Amplitude

In a MIDI file, the amplitude of notes is determined by the velocity parameter associated with each note. Additionally, the use of note_on and note_off messages specifies when a note starts and stops playing. Some MIDI files may omit the note_off message and instead use a note_on message with a velocity of 0 to represent the end of a note. For our representation, we can easily handle both cases by treating a note_on message with velocity 0 as equivalent to a note_off message.

Regarding the amplitude representation A for a MIDI file, there are three options:

• Using only the rhythmic range A 3 (as discussed in Section 2.1.3.2), which omits the velocity information.

• Using a lattice that considers dynamics, as explained in Section 2.

A P D 128 = D 128 ∪ {•} a ≤ P 128 b ⇔      a = ⊥,
A D 128 = {⊥} ∪ {•, ×} × D 128 a ≤ 128 b ⇔ a = ⊥, or a 1 ≤ 3 b 1 and a 2 ≤ 128 b 2 .
In Figure 2.5, we present the piano roll of a MIDI file generated from the score in Figure 2.5a with two different amplitude representations: A = A 3 (which we will use most often due to its simplicity) represented in Figure 2.5b, and A = A D 128 (a more refined representation, left for future research), represented in Figure 2.5c.

Since there are four instruments, we might represent this excerpt as a piano roll stack with the set of indexes I = {Bass, Vla., Vln. 2, Vln. 1}, one per instrument. While the instruments are different and exhibit different timbers, they are all bowed string instruments and thus have a similar sound. This is why, we might want to represent all of them in the same piano roll. In order to do that, we use the contraction 18 in the set of indices of S, i.e., i∈I S :

T × F → A (t, ξ) → i∈I P i (t, ξ)
.

(2.54)

Representing Scores as Piano Rolls

Representing scores as piano rolls involves converting the musical data of a score into functions P : T × F → A in a musical space. However, it should be noted that scores can contain a vast amount of information, and not all of it can be fully rendered under this representation. The information that we choose to retain in the piano roll representation includes:

• the number of instruments, which may be merged for instruments of the same family (e.g., merging all the strings into a single representation),

• the pitch information, with potential loss of enharmonic notes,

• the rhythm, up to a certain realization, as some elements such as trills or grace notes loose their generality after being rendered,

• the dynamics information, if we choose an amplitude space such as A P D or A D . Some of the information that we typically drop in the piano roll representation includes: the key, the bars, the time signature, the articulation, the tempo, the timber, the divisi, the moods, etc.

As an example, Figure 2.6 shows an actual score and a score with only the features we keep.

Time

The time inside a score is measured in wholes as exposed in Section 2.1.1.3. We will then use T p q  as space of timestamps, with p q being a time signature 19 .

¢ °¢ °¢

Vla. To create a computational model, we need to discretize time. One approach is to arbitrarily select a minimum note value, for example, , to serve as the smallest time unit. However, this approach may not work well when dealing with triplets or other complex rhythmic patterns.

Bass

To address this issue, we can determine the tatum of the score, which is a common notion in music theory that can be thought of as the greatest common divisor of all note values present in the score. The tatum serves as the minimal note value for our computational model, allowing us to handle various rhythmic patterns accurately.

In the representation of scores as piano rolls, we may encounter grace notes or trills, which require determining their rhythmic realization. This means establishing how these ornamentations are performed within the time structure of the music.

Frequency

In scores, the frequency is typically quantized, except in some particular cases like the glissando where there may be a continuous pitch change. Despite having more information in scores compared to MIDI files (since enharmonic pitches are distinguished), we still quantize the frequency and thus have F = N .

Amplitude

In the case of scores, we do not have a velocity range like in MIDI files. However, we have a range of dynamics denoted by D pf , as presented in Section 2.1.3.3. Similarly to MIDI files, we can choose to keep only the rhythmic component and use A 3 to represent the amplitude lattice. This simplification allows us to focus on the essential rhythmic aspects of the score while disregarding more detailed dynamic variations.

Chroma Roll

In this section, we explore the transformation of the space of pitches N into the space of chromas N 12 to create another piano roll representation that we call chroma roll. The chroma roll is particularly useful for analysis, as it takes advantage of the concept of equivalence up to the octave commonly used in music theory.

To achieve the chroma roll representation, we consider the description of pitches given in Equation (2.6), where N = N 12 × Z. We then project this space onto the chromas component using the first projection, denoted as π 1 . This projection is defined as follows:

π 1 : N 12 × Z → N 12 (N, n) → N .
(2.55)

Next, we extend this projection to the entire musical space A T ×N using the supremum operator. The extended projection, denoted as π 12 , is given by:

π 12 : A T ×N → A T ×N 12 P → P : T × N 12 → A t, ξ → f t, π -1 1 ξ (2.56) 75
Chapter 2. Time-Frequency Representations of Music where f t, π -1

1 ξ = {f (t, η) ∈ A : η ∈ π -1 1 ξ }.
Representing a chroma roll poses some challenges, as we aim to preserve the cylindrical topology of T × N 12 . To address this, we represent the chroma roll in two dimensions while keeping in mind that the frequency dimension wraps around.

As an example, Figure 2

Conclusion

In conclusion, we have presented various time-frequency representations of music, all of which can be organized as a musical space M = A T ×F , with the possibility of 76 2.4. Conclusion By endowing A with a complete lattice structure (A, ≤), we establish that (M, ) is also a complete lattice, enabling the application of Mathematical Morphology on these representations. Moreover, as the domain of functions in M is a space T × F with a group that acts on it, we can employ mathematical morphology techniques that use structuring elements.

Table 2.1 provides a summary of the representations we have presented, along with the different choices available for T , F, and A. In the forthcoming chapters, we will employ some of these spaces to undertake various musicological tasks.

Representation

Space Group unit Amplitude

Spectrogram STFT T s × F Hz (R × R, +) s × Hz R + or [0, 1] R or R - (in dB) CQT T s × F log Hz (R, +) × (R + * , •) s × Hz R + or [0, 1] R or R - (in dB) T s × F st (R × R, +) s × st R + or [0, 1] R or R - (in dB) Piano roll MIDI file T 1 × N (Z × Z, +) tick × st A 3 , A 128 or A P 128 Score T p q  × N (Q × Z, +)  × st A 3 , A P D pf or A D pf Chroma roll MIDI file T 1 × N 12 (Z × Z 12 , +) tick × st A 3 , A 128 or A P 128 Score T p q  × N 12 (Q × Z 12 , +)  × st A 3 , A P D pf or A D pf
Table 2.1: Representations of music depending on the choices for time and frequency.

Chapter 3

Analyzing Spectrograms with Mathematical Morphology

In the preceding chapters, we established a framework for applying mathematical morphology to various time-frequency representations of music. Particularly, we demonstrated that a spectrogram can be viewed as a representation of the form A T ×F , enabling the application of morphological operators based on structuring elements.

While MM has been applied to analyze spectrograms [START_REF] Steinberg | Segmentation of a speech spectrogram using mathematical morphology[END_REF][START_REF] Cadore | Morphological processing of spectrograms for speech enhancement[END_REF][START_REF] Xu | A mathematical morphological processing of spectrograms for the tone of chinese vowels recognition[END_REF][START_REF] Zhang | The detection of crackles based on mathematical morphology in spectrogram analysis[END_REF] of speech, it is not up to (Romero-García, Agón, et al., 2022) that it was first applied to analyze spectrograms of music.

The primary objective of this chapter is to synthesize an audio signal y(t) that closely resembles an input signal x(t) from a musical instrument. To achieve this, we need a method to analyze the input signal x(t) and a method to synthesize the output signal y(t).

The approach we take is to transform x(t) into a spectrogram S(τ, ω) and then apply MM operators to extract features that can be used to synthesize the output signal y(t). The features we need to extract are determined by the synthesis model we use.

In this thesis, we employ the spectral modeling synthesis (SMS) (X. [START_REF] Serra | Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decomposition[END_REF] for generating sounds of musical instruments. SMS involves analyzing the spectrum of a signal to extract its spectral features and subsequently synthesizing a new signal that closely resembles the original one. We adopt an extension of the SMS model, known as the STN model (Sines plus Transients plus Noise) (T. S. [START_REF] Verma | Extending spectral modeling synthesis with transient modeling synthesis[END_REF], which incorporates an additional component for transients.

The original SMS model, which consists of sines plus noise, is well-suited for many musical instrument signals, as it captures the significant sinusoidal component (e.g., from bowed strings or winds) and the accompanying noise component (e.g., the sound of the bow or the blow). However, some instruments possess a transient component that significantly contributes to their sound (e.g., plucked or struck strings and idiophones like wood blocks, marimbas, or vibraphones). As a result, the STN model proves to be more accurate in such cases.

The task we want to perform can be summarized as follows:

1. Transform the input signal x(t) into a spectrogram S(τ, ω).

2. Apply MM operators to estimate the parameters of the STN model.

3. Synthesize the output signal y(t) with the STN model.

The overall pipeline is depicted in Figure 3.1.

In the following, we introduce the STN model in Section 3.1, and we elaborate on the MM pipeline for parameter extraction in Section 3.2. Finally, in Section 3.3, we showcase and analyze the results and limitations of our method when applied to various musical instruments.

Sines, Transients and Noise Model

The process of generating synthetic sounds requires a suitable synthesis model. The earliest model for sound synthesis was additive synthesis, which can be traced back to the work of [START_REF] Fourier | Harmonicpercussive-residual sound separation using the structure tensor on spectrograms[END_REF][START_REF] Helmholtz | Die Lehre von den Tonempfindungen als physiologische Grundlage für die Theorie der Musik[END_REF] However, a significant improvement in sound quality was achieved with the introduction of Sines + Noise synthesis (X. [START_REF] Serra | Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decomposition[END_REF], particularly for music instrument sounds.

The latest evolution in sound synthesis came with the incorporation of a transient component, resulting in the Sines + Transients + Noise (STN) synthesis. This breakthrough emerged from a series of papers published towards the end of the century (T. S. [START_REF] Verma | Transient modeling synthesis: A flexible analysis/synthesis tool for transient signals[END_REF][START_REF] Verma | An analysis/synthesis tool for transient signals that allows a flexible sines+transients+noise model for audio[END_REF][START_REF] Verma | Extending spectral modeling synthesis with transient modeling synthesis[END_REF]. The STN model has since garnered significant research interest and has seen further developments in recent times [START_REF] Driedger | Extending harmonic-percussive separation of audio signals[END_REF][START_REF] Fourier | Harmonicpercussive-residual sound separation using the structure tensor on spectrograms[END_REF][START_REF] Fierro | Enhanced fuzzy decomposition of sound into sines, transients, and noise[END_REF].

The STN model is based on decomposing a signal y(t) into three components: the sines s(t), the transients h(t), and the noise w(t). The equation representing this decomposition is given by:

y(t) = s(t) + h(t) + w(t), (3.1)
where t is a variable representing time measured in seconds. The domain of t will depend on whether we are in the continuous case (t ∈ R) or in the discrete case Chapter 3. Analyzing Spectrograms with Mathematical Morphology (t = T s n ∈ T s Z), where T s is the sampling period1 .

Sinusoidal Oscillators

To generate the sines, we use a sinusoidal oscillator, a well-known technique in the literature (see [START_REF] Smith | Spectral audio signal processing[END_REF]). The formula for generating the sines is as follows:

s(t) = I i=1 s i (t) = I i=1 a i (t) sin 2πΦ i (t) (3.2)
where I ∈ N * is the number of sinusoidal components, a i (t) is the time-varying amplitude of the i th component s i , and Φ i (t) = t 0 ξ i (u) du + Φ 0 , with ξ i (t) and Φ 0 are the instantaneous frequency at the time t and the starting phase of the i th component, respectively.

The parameters to be estimated are then a i (t) and ξ i (t) (we drop the starting phase information Φ 0 and set it to 0), from which we can deduce Φ i (t) = t 0 ξ i (u) du.

Filtered Noise

To generate the stochastic part w(t), we create a white noise signal and filter it using a linear time-varying filter (LTV filter).

First, we generate the white noise by using a random process distributed as a normal distribution N (0, σ) with a standard deviation σ ∈ R + . We select a value of σ that results in an overall power density of 0 dB. The process of finding the appropriate σ is described in Section 3.2.2.2.

Next, we apply the LTV filter to the white noise. To do this, we follow the approach known as the STFT filter [START_REF] Boashash | Time-frequency synthesis and filtering[END_REF], which involves the following three steps:

1. Calculate the short-time Fourier transform (STFT) S(τ, ω) of the input signal x(t).

2. Multiply S(τ, ω) by a weight function Θ(τ, ω).

Sines, Transients and Noise Model

3. Synthesize the output signal y(t) by performing an inverse STFT of S(τ, ω) •

Θ(τ, ω).

The filter itself is represented by the parameter Θ(τ, ω), that we call mask, which is the parameter we aim to estimate using mathematical morphology as discussed in Section 3.2.

Transient Generation

To generate the transient component, we follow the approach presented in (T. S. [START_REF] Verma | Extending spectral modeling synthesis with transient modeling synthesis[END_REF]. However, instead of using the inverse cosine transform as in the original paper, we use the usual Fourier transform and adapt the formulas accordingly.

The transient is considered the dual of a sinusoid, as sinusoids appear as horizontal lines in spectrograms, while transients appear as vertical lines. To achieve the rotation in the time-frequency plane, we use the Fourier transform. Specifically, we make use of the property stated in Proposition 2.5:

STFT g [x](τ, ω) = e -2πıω•t STFT F [g] [F [x]](ω, -τ ) . (3.3)
To generate the transient, suppose we want a transient with amplitude a(ξ) and time t(ξ) (where frequency ξ is the variable). First, we generate the signal s(ξ) = a(ξ) e 2πıΦ(ξ) + e 2πıΦ (-ξ) (3.4)

where Φ(ξ) = ξ 0 t(ν) dν. Then, we obtain the transient component h(t) by applying the Fourier transform to s(ξ):

h(t) = F [ξ → s(ξ)](t) .
(3.5)

The result of this process is illustrated in Figure 3.2. When transients exhibit significant variations in amplitudes across frequencies, they are prone to experiencing temporal leakage. This temporal leakage poses a problem as it introduces sinusoids before and after the transient, which interferes with the main feature of the transient -its time concentration. Such artifacts are undesirable.

Although this issue was not explicitly addressed in (T. S. [START_REF] Verma | Transient modeling synthesis: A flexible analysis/synthesis tool for transient signals[END_REF][START_REF] Verma | An analysis/synthesis tool for transient signals that allows a flexible sines+transients+noise model for audio[END_REF][START_REF] Verma | Extending spectral modeling synthesis with transient modeling synthesis[END_REF], we propose a solution to mitigate this problem. We apply a window to the transient that is equal to 1 between min ξ∈R t(ξ) and max ξ∈R t(ξ), and zero outside of this interval2 .

Mathematical Morphology Analysis

In this section, we will provide a detailed explanation of the process of estimating the parameters for the given signal x(t) using MM. The parameters we need to estimate are as follows:

1. For the sinusoidal oscillators, we need to estimate the number of components I ∈ N, the amplitudes a s i (t) and the frequencies ξ i (t), ∀i ∈ I. 2. For the filtered noise, we need to estimate the mask Θ(τ, ω).

For the transient generator, we need to estimate the number of transient com-

ponents P ∈ N, the amplitudes a h p (ξ) and the times t p (ξ), ∀p ∈ P .

Discrete Version of the Problem

Let us now expose the discrete version of the problem. We consider an input signal x(t) ∈ [-1, 1] R with the time measured in seconds and finite support [0, T ], with

T ∈ R + .
To transform it into a discrete array, we sample it at a sampling frequency ξ s ∈ R + measured in Hertz (with corresponding sampling period T s = 1 ξs measured in seconds), and we quantize the amplitudes in floating point values of 32 bits.

We get then a number of samples N = ⌈T • ξ s ⌉ ∈ N * , corresponding to the timestamps t n = nT s ∈ R measured in seconds. We define the discrete input signal as

{x[n]} N -1 n=0 ∈ [-1, 1] N , with x[n] = x(t n ), ∀n ∈ {0, 1, ..., N -1}.
To transform the discrete signal {x[n]} N -1 n=0 into a time-frequency representation, we use the discrete STFT exposed in Section 2.2.2.1; we leave for future research the use of the CQT and the TFST.

We choose a STFT with a window g of size J ∈ N * and center sample j 0 ∈ {0, 1, ..., J -1}. We choose a hop size of H ∈ N * and a number of frequency bins K ∈ N * . By setting M = ⌈ N H ⌉ ∈ N * and adapting Equation (2.39) to our case, we obtain:

Z[m, k] = J-1 j=0 x[m + j -j 0 ]g[j]e -2πı(m+j-j 0 ) k K . The STFT array {Z[m, k]} M -1, K-1 m=0, k=0 ∈ C M ×K corresponds to the time-frequency points {(τ [m], ω[k]} M -1,K-1
m=0,k=0 that are given by: • τ [m] = t mH ∈ R with time precision T p = HT s ∈ R.

•

ω[k] = k ξs K ∈ R + with frequency precision ξ p = ξs K ∈ R + .
τ and T p are measured in seconds and ω and ξ p are measured in Hertz.

The window function g is chosen to be real, positive and symmetric, having then

g * = g. Its length is J ∈ N * which gives a time observation T o = JT s ∈ R seconds.
We have then an array {g[j]} J-1 j=0 ∈ (R + ) J . We set j 0 = ⌊ J 2 ⌋ as the center sample. We normalize it such that ||g|| 1 = 1 in order to satisfy the conditions of Equation (2.21).

The spectrogram

{S[m, k]} M -1, K-1 m=0, k=0 ∈ [-∞, 0] M ×K
is calculated from the STFT array using the formula:

S[m, k] = 10 log 10 |Z[m, k]| 2 . (3.6)
The values used for the parameters are given in

Processing Pipeline

The main contribution of this chapter is the use of MM for extracting the following parameters:

1. For the harmonic oscillator: k=0 of each transient. To estimate these parameters, we input the spectrogram S into the morphological pipeline described in Figure 3.3. In the next sections, we will provide a detailed explanation of each step.

For illustrative purposes, we use an input signal of a woodblock, as it exhibits all three features required for our analysis: the sinusoidal component, the noise component, and a transient. The spectrogram of the woodblock sound is shown in Figure 3.4. This spectrogram will serve as the input to the morphological pipeline.

Pre-processing

To prepare the input spectrogram for further processing, we apply two consecutive morphological operations: reconstruction by erosion and erosion.

Reconstruction by erosion

The first step of our processing is to "fill the holes" in the spectrogram. The noisy part of a spectrogram often contains holes and hills, as shown in Figure 3.5a. To ensure that the subsequent operations are not biased by the presence of holes, we use the reconstruction by erosion technique, as explained in Section 1.2.5.3.

The marker function for the reconstruction by erosion is the zero function, which is the top element of the space of functions [-∞, 0] R×R (or, in its discrete version, the space of arrays [-∞, 0] M ×K ).

Erosion

Once we obtain the result of the reconstruction by erosion, we proceed to apply a greyscale erosion. The structuring element b we use for this operation is the window function in dB, i.e., if g is the window, and it is given by: b = 20 log 10 (g) .

(3.7)

This step is crucial as it helps to reduce the temporal leakage of the spectrogram and ensures that the masks for the noise and sinusoids are accurately aligned with 88

Mathematical Morphology Analysis

Processing for the noise component

To obtain the mask Θ for filtering the noise, we simply need to apply an opening operation to the erosion obtained in the previous step.

Opening

The structuring element used for the opening is a square with the sizes t w and ξ w , which satisfy the following conditions:

20 log 10 (w(0)) -20 log 10 (w(t)) > 60 dB, ∀t ∈ R : |t| > t w 2 (3.8) 20 log 10 ( ŵ(0)) -20 log 10 ( ŵ(ξ)) > 60 dB, ∀ξ ∈ R : |ξ| > ξ w 2 , (3.9)
This means that the width of the square ensures a 60 dB drop both in time and frequency. With the parameters provided in Table 3.1, we get t w = 44 ms and ξ w = 193 Hz for the time and frequency dimensions of the rectangular structuring element, respectively. The result of applying the opening operation to the erosion result is shown in Figure 3.7. The resulting image now resembles the overall shape of the noise component.

The importance of the reconstruction by erosion step becomes visible in this process. Its absence would lead the opening operation to propagate the values of the holes in the spectrogram rather than achieving the desired average value of 0 dB. This effect is illustrated in Figure 3.8, where we observe that applying the opening before the reconstruction by erosion (Figure 3.8b) results in a not constant image. Conversely, applying the opening after the reconstruction yields a uniform image with an average value of 0 dB.

In order to achieve an average value of 0 dB after the reconstruction by erosion, we tested several values of σ and determined that setting σ = 30 achieves this desired outcome3 , as illustrated in Figure 3.8.

The application of an erosion to the reconstruction by erosion is a key step, as it ensures that the noise component aligns accurately with the transient. Without this erosion step, the noise component could start before the transient. In Figure 3.9, we display the mask and the filtered noise side by side, highlighting the effects of temporal leakage that can occur if the erosion is not applied.

We have now obtained the mask Θ, which allows us to effectively filter the white noise. The result is presented in Figure 3.10, where we can observe both the original spectrogram and the spectrogram of the filtered noise. Notably, the noisy part is accurately recovered, showcasing the success of the filtering process in faithfully reconstructing the noise component.

Processing for the sinusoidal component

We now explain how we use morphological operators to estimate the parameters for the harmonic oscillator, which are I, a s i and ξ i . We use as input for the processing the erosion since it has no holes and no temporal leakage.

Vertical thinning

The first operator we apply is a vertical thinning; the thinning operator is explained in Definition 1.33. For obtaining a vertical thinning, i.e., for contracting the image in the vertical direction to obtain horizontal components, we need to remove the north, south, north-east, south-west, north-west and south-east points4 . To remove these points, we select the following pair of structuring elements: These patterns should be interpreted in the following manner: ones correspond to the elements of set C, zeroes correspond to the elements of set D, andmeans that the point is not considered in the structuring element. Moreover, we assume that the origin is located at the center pixel. More formally, if we denote the matrix associated with the pattern as (a ij ) : i, j = 1, 2, 3, the corresponding sets C and D are given by:

(C, D) N =   0 0 0 -1 - -1 -   (C, D) N E =   -0 0 1 1 0 -1 -   (C, D) N W =   0 0 - 0 1 1 -1 -   (C, D) S =   -1 - -1 - 0 0 0   (C, D) SW =   -1 - 0 1 1 0 0 -   (C, D) SE =   -1 - 1 1 0 -0 0   S,
C = {(i -2, j -2) ∈ Z 2 : a ij = 1}
(3.10) (3.11) This vertical thinning process transforms the ridges of the input into lines of one-pixel thickness. However, as seen in Figure 3.11, these lines cannot be directly used to obtain our parameters. We still need to remove the background information to obtain precise lines. This is achieved by using the top-hat operation.

D = {(i -2, j -2) ∈ Z 2 : a ij = 0}

Vertical top-hat

To isolate the lines and remove the background, we apply a top-hat operation to the thinned image. Top-hat is explained in Section 1.2.4.3. Since our objective is to retrieve the horizontal lines, we use a vertical top-hat by using a structuring element with a size of 1 × 3 pixels (1 pixel in time and 3 pixels in frequency). The output of this process is shown in Figure 3.12.

Threshold vertical top-hat

While horizontal lines appear neat in the top-hat image, various artifacts arise due to the nature of spectrograms. To mitigate some of these artifacts5 we apply a threshold. Moreover, this threshold operation is done on the reconstruction by erosion but with the values of the top-hat i.e., if we denote the output of the threshold by S > , the reconstruction by erosion as S 0 and the result of the top-hat as S Id-γ , we have

S > (τ, ω) = S 0 (τ, ω) if S Id-γ (τ, ω) > τ v -∞ if S Id-γ (τ, ω) ≤ τ v (3.12)
with τ v being the threshold for the vertical top-hat. The value we chose is

τ v = 5 dB.
The result is shown in Figure 3.13.

Mathematical Morphology Analysis

Filter small horizontal lines

This step is intended to eliminate lines that are too small to be considered genuine sinusoids and are more likely to be artifacts. While these lines might represent actual signals, they tend to contribute less to the desired output and can introduce unwanted sounds that are perceived as artifacts.

To address this issue, we employ a two-step processing approach. First, we shrink the lines that are below a certain length threshold, causing them to disappear if they are too small. Then, we use a reconstruction by dilation to recover the parts that were previously shrunk.

For shrinking the lines, we remove the west and east points by applying a thinning that uses the following patterns:

C W =   0 -- 0 1 - 0 --   C E =   --0 -1 0 --0   . (3.13)
Following the shrinking operation, we perform a reconstruction by dilation. We use the shrunken image as the marker and the output of the threshold as the mask. The minimum length that we allow for a line to be retained serves as a parameter for this process. In this case, a minimum length of 100 ms was chosen. The result of this operation is shown in Figure 3.14.

Retrieving the parameters for sinusoids

The last image of the process serves as input to for parameter recovery. This recovery is not a processing in itself, but rather a "transducer": it transforms an image into a list of parameters. The process works as follows:

1. We recover the I connected components, representing individual lines, using the SciPy [START_REF] Virtanen | SciPy 1.0: Fundamental algorithms for scientific computing in python[END_REF] library's functionality for this purpose.

For each component indexed by i, we create an array

{(t i m , ξ i m , S O (t i m , ξ i m )} M i m=1
where S O is the output of our morphological pipeline for sinusoids.

3. We sort the array with respect to the time.

This approach yields i arrays, each corresponding to a sinusoidal component. However, two potential issues arise if we synthesize directly from these arrays:

1. Multiple branches for each line may exist, leading to multiple ξ values for the same t. Chapter 3. Analyzing Spectrograms with Mathematical Morphology 2. Working with images composed of pixels implies frequency quantization, resulting in significant steps that could generate artifacts, particularly in lower frequencies.

Mathematical Morphology Analysis

To address both issues simultaneously, we apply a filter to the array of frequencies. This smooths the frequencies and eliminates artifacts. Specifically, we use a Butterworth filter [START_REF] Butterworth | On the theory of filter amplifiers[END_REF] of order 3, with a critical frequency set to 0.05 times of the Nyquist frequency6 . To avoid border problems, we employ Gustafsson's method [START_REF] Gustafsson | Determining the initial states in forward-backward filtering[END_REF]. The result is shown in Figure 3.15.

Processing for transient component

The approach used for retrieving transient parameters is a dual of the one employed for sinusoids, with the exception that the reconstruction by erosion itself (not its eroded version) is used. The corresponding steps are as follows:

• Horizontal thinning: instead of a vertical thinning we use an horizontal thinning, with the templates corresponding to the east, west, north-west, southeast, north-east, south-west points. The result is shown in Figure 3.16b.

• Horizontal top-hat: instead of a vertical top-hat, we employ an horizontal top-hat operation, with a structuring element of size 3 × 1 pixels (3 in time and 1 in frequency). The result is shown in Figure 3.16c.

• Threshold: the threshold operation is performed on the output of the horizontal top-hat, with the same threshold value τ v = 5 dB. The result is shown in Figure 3.16d.

• Filter small vertical lines: we use the same approach as before, but with a vertical thinning (using templates C N and C S ) and with minimal length we allow being 100 Hz. However, this step did not affect the image in this case as there are two long lines.

• Retrieving the parameters for transient: the process for retrieving the lines and applying the filter (in this case to the times array) is the same as used for sinusoids. The result is shown in Figure 3.17

Application to Music Instruments

In this section, we evaluate the performance of the proposed method by applying it to different musical instruments. We use the same set of parameters and test the method on sounds produced by various instruments: marimba (with a pronounced transient component), violin (with a prominent sinusoidal component), gong (primarily consisting of a noise component), and piano (featuring a balanced combination of three components).

All of the sounds used in this chapter are sourced from the University of Iowa Musical Instrument Samples7 and Studio-On-Line Database [START_REF] Ballet | Studio online 3.0: An internet "killer application" for remote access to IRCAM sounds and processing tools[END_REF] sound libraries.

Marimba

The marimba sound features a significant transient part and also a notable sinusoidal component. The processing results are displayed in Figure 3.19. The transient part has been recovered with high fidelity. However, there are some challenges in the recovery of the sinusoidal component.

One major issue is the failure to capture an important sine wave with a frequency of around 2600 Hz. This is a critical concern as, despite its brevity, the sine wave is clearly intense and prominent, as shown in the spectrogram. The problem occurred due to the threshold operation, where the sine was split into two parts, both of which were subsequently suppressed because they were too small. Another issue is the presence of interference between two sine waves around 1300 Hz causing both of them to go undetected. Additionally, interference effects are the cause of the break of the sine wave with a frequency of around 500 Hz.

Overall, the retrieval of the sinusoidal component from the marimba sound is considered to be of mediocre quality.

Violin

The results from the violin sound exhibit improvements over those from the marimba, as demonstrated in Figure 3.20. Multiple sinusoidal components were successfully detected, and the noise component generated by the bowing of the string was accurately recovered. The input and output spectrograms closely resemble each other.

Application to Music Instruments

However, a notable limitation of the process is related to the attack of the sound. In the case of the violin, particularly when the attack is strong, it produces a crackling sound rather than a traditional transient. This unique nature of the attack prevents its simulation through a transient component, resulting in an unsuccessful recovery using the noise component.

The method was also applied to a violin sound featuring vibrato, and the results are depicted in Figure 3.21. The majority of the lines were reasonably recovered, even if they are not straight. Some lines, due to their brevity and lack of connection to others, were not recovered. Additionally, interference, this time with the noise, led to the disconnection of certain lines.

Gong

Among the tested instruments, the gong showcases the best results. The noise component is accurately recovered, demonstrating high fidelity in reproducing the original noise characteristics. While some minor sinusoidal components are present, their influence on the output is largely masked by the dominant noise component. However, the fundamental bass sine, the only sine that is clearly perceptible, is successfully recovered. The transient elements, although not extensively pronounced in the gong sound, are still moderately captured by the method.

Piano

The piano results are probably the more disappointing: while the sinusoidal and noise components are accurately recovered, the transient component retrieval is notably inadequate. This outcome contradicts initial expectations, given that the piano sound is renowned for its significant transient component during its attack.

Despite this unexpected result, several insights might shed light on the situation. Close examination of the piano spectrogram reveals that the transient is not present as a distinct, isolated vertical line, as is typical in some other cases. Instead, there exist small vertical lines at the onset of each sine wave. These shorter vertical lines correspond to high variations of the amplitude of the sine, experiencing spectral leakage.

This observation prompts the question of whether the transient component alone is sufficient or if it should be use as a sub-parameter of the sinusoidal component, or perhaps even use a hybrid approach tailored to each specific case.

Discussion and Conclusion

Throughout this chapter, we have explored how MM can be applied to the analysis of spectrograms of music instrument sounds. These sounds often exhibit distinct ge- ometric patterns, such as lines and holes, making MM well-suited for their detection. However, the performance of the proposed method falls short of our expectations. While it does reasonably well in detecting the desired geometric patterns, it lacks the robustness required for such signal processing applications. This limitation is evident from the impact of the thresholding process and the presence of interferences. The method excels in recovering the noise component, which involves fewer steps. Nonetheless, it is important to acknowledge that the core of noise component recovery lies in the reconstruction by erosion, a computationally expensive operation that demands several seconds of GPU computation for only a few seconds of sound.

Many of the challenges in our method are not exclusively rooted in the domain of mathematical morphology. The shortcomings also extend to the STN synthesis model. This model, while conceptually appealing, does not perform as well as expected. In particular, the transient generation has notable issues. For instance, the synthesized signals are symmetric, an unrealistic feature in musical sounds. Additionally, the lack of coherent synchronization between transients and sines results in audible discrepancies.

It is important to note that the distinction between the signal and the noise can often be ambiguous, with each potentially obscuring the other and yielding undesirable outcomes. When the signal overshadows the noise, the situation is less problematic since the opening operation effectively transforms it into noise, preserving the coherence of the noise component. However, when the noise masks the signal, it results in fragmented lines with fading effects to manage, as well as the issue of small lines that might be overlooked. In conclusion, we believe that significant room for improvement exists both in the MM pipeline itself to enhance its robustness and in the synthesis method to achieve smoother component integration.

Additionally, we acknowledge the significant influence that parameter tuning can have on the outcomes. This emphasizes the importance of incorporating human expertise into the process. While this might pose challenges for full automation of the pipeline, it presents an opportunity for meaningful collaboration between humans and machines. Specifically, we envision the development of a desktop application that empowers users to interact with the system. Users could select specific regions of a spectrogram and use MM techniques to effectively separate noise from the desired signal. This interactive approach would not only harness the strengths of mathematical morphology but also leverage human intuition and domain knowledge to enhance the overall accuracy and quality of the results.

Chapter 4

Mathematical Morphology Applied to Generate Piano Rolls

In the previous chapter, we have shown how to use MM to analyze spectrograms. In this chapter and the following, we apply morphological operators to another kind of time-frequency representation of music: piano rolls. Piano rolls have been exposed in Section 2.3 as a useful representation of MIDI files and scores. In this chapter, we focus on generating music in this format through the use of MM.

In the following, we use the notations exposed in Chapter 2. In particular, we consider the complete lattice (A T ×F , ), where T × F is a time-frequency space with (G T ×F , +) a group acting on it, and (A, ≤) is one of the amplitude ranges exposed in Section 2.1.3, endowing A T ×F with a structure of complete lattice given by the pointwise order. With this structure, we can apply MM based on structuring elements as exposed in Section 1.2.

The choices of T , F and A may vary depending on the specific cases, as exposed in Chapter 2. The selection of T depends on whether the input is derived from a MIDI file or a musical score. F will be either N or N 12 . While the amplitude range might also differ, for the sake of simplicity, we use A 3 = {⊥, •, ×} in the examples provided. Additionally, we employ the Boolean lattice A 2 = {0, 1}, which results in a residuated triplet through the lattice multiplication operation

• : A 3 × A 2 → A 3 exposed 1 in Section 2.1.3.2.
The chapter is structured as follows: in Section 4.1, we begin by establishing the definitions of texture and harmony, which we subsequently employ to construct a concept that we call harmonic texture. Then, in Section 4.2, we expose the process Chapter 4. Mathematical Morphology Applied to Generate Piano Rolls of generating piano rolls using MM and harmonic textures.

Texture and Harmony

The main contribution of this chapter is the establishment of a framework for the generation of music based on two concepts borrowed from music theory: texture and harmony. While these terms might exhibit varied and sometimes vague definitions across different contexts, our objective here is to provide precise mathematical definitions tailored to our specific objectives.

Texture

The term texture holds various interpretations across different domains. In the realm of music, the concept of texture lacks a universally agreed-upon definition [START_REF] Moreira | Textural design: A compositional theory for the organization of musical texture[END_REF]. In [START_REF] Couturier | A dataset of symbolic texture annotations in mozart piano sonatas[END_REF], two distinct yet interconnected [START_REF] Herold | Timbre et analyse musicale : Les possibilités d'intégration du timbre dans l'analyse formelle des oeuvres pour piano du dix-neuvième siècle[END_REF] interpretations of the term texture have been identified.

The first interpretation is the orchestral texture, which concerns the timbral interactions between musical instruments. This aspect has been extensively studied in orchestration from the 19 th century [START_REF] Berlioz | Grand traité d'instrumentation et d'orchestration modernes[END_REF][START_REF] Piston | Orchestration[END_REF][START_REF] Nordgren | A measure of textural patterns and strengths[END_REF][START_REF] Guigue | The structural function of musical texture: Towards a computer-assisted analysis of orchestration[END_REF].

The other interpretation, known as symbolic texture, has received less attention, possibly due to its elusive nature. However, it is this latter interpretation that forms the basis of our discussion, and we aim to provide a formal definition that captures its essence. In recent years, there has been a growing interest in investigating symbolic texture [START_REF] Giraud | Towards modeling texture in symbolic data[END_REF][START_REF] Parada-Cabaleiro | Automatic recognition of texture in renaissance music[END_REF][START_REF] Soum-Fontez | Symbolic textural features and melody/accompaniment detection in string quartets[END_REF]Couturier et al., 2022a[START_REF] Couturier | A dataset of symbolic texture annotations in mozart piano sonatas[END_REF].

To formulate our definition of texture, we draw upon the concept of rhythm. We provide a custom definition of rhythm for our framework, but we will see later that it is fairly compatible with the definition of rhythm based on trees [START_REF] Agon | Representation and rendering of rhythm structures[END_REF] that is used in Computer Assisted Composition [START_REF] Jacquemard | A structural theory of rhythm notation based on tree representations and term rewriting[END_REF][START_REF] Jacquemard | Generating equivalent rhythmic notations based on rhythm tree languages[END_REF][START_REF] Ycart | A supervised approach for rhythm transcription based on tree series enumeration[END_REF].

Definition 4.1 (Rhythm). We say that R ∈ A Q 3 is a rhythm if ∃N ∈ N such that R = N n=1 h n (4.1)
where the

h n ∈ A Q 3 are such that ∃s n ∈ Q, ∃d n ∈ Q + : h n : Q → A 3 t → h n (t) =      × if t = s n • if t ∈]s n , s n + d n [ ⊥ if t / ∈ [s n , s n + d n [ and h n ∧ h n ′ = ⊥ if n = n ′ .
Each h n is called a hit and has a start s n and a duration2 d n . We notate

h n ≡ (s n , d n ) and R ≡ {(s n , d n )} N n=1
for simplicity. This definition links the notion of rhythm with the notion of time span that is presented in [START_REF] Lewin | Generalized musical intervals and transformations[END_REF]; each hit is equivalent to a time span, and a rhythm is given by a set of hits.

In order to include also the possibility of a rhythm to be defined over a time space T , we include the definition of a placed rhythm.

Definition 4.2 (Placed rhythm). Let T be a time space and ι :

T → Q a function 3 . We call R 0 ∈ A T 3 a placed rhythm if ∃t 0 ∈ T , ∃R ∈ A Q 3 such that R 0 = t 0 + R : T → A 3 t → R 0 (t) = R(t -t 0 ) := R(ι(t) + (-ι(t 0 ))) . (4.2)
These definitions might appear intricate for rhythms, but they represent a notion of rhythm that can handle previous mathematical formalizations like the one exposed in [START_REF] Toussaint | The geometry of musical rhythm[END_REF] and the one proposed in [START_REF] Agon | Representation and rendering of rhythm structures[END_REF] (besides the hierarchical component).

To be able to simplify the representation of rhythms, we are showing that a rhythm can be represented by a vector.

Let R be a rhythm. We are creating a vector R ∈ A M 3 for some M ∈ N that represents this rhythm. We know that R can be written as

R = N n=1 h n

Texture and Harmony

with the GCD being a = 1 8 . We can rewrite then

s 1 = -2 s 2 = -1 s 3 = 0 s 4 = 4 e 1 = -1 e 2 = 0 e 3 = 2 e 4 = 6
and have

m 0 = -2, m 1 = 6 and M = 6 -(-2) = 8.
The resulting vector is given by In the following, we will plot such a vector as 0 where we left the square empty for ⊥ and specify the place of the 0 and the tatum. Notice that we used the GCD (the tatum) to quantize the rhythm, but that we might have use a divisor of the tatum. In the case of the Example 4.3, have we set the value of a to = 1 16 and we would have get the vector 0 Notably, we are capable of converting any rhythm tree into a rhythm. In [START_REF] Jacquemard | A structural theory of rhythm notation based on tree representations and term rewriting[END_REF], it is demonstrated that each rhythm tree possesses an associated duration sequence. From this sequence, we can deduce the sequence of hits that correspond to a rhythm in accordance with our definition6 . This transformation does involve a loss of the hierarchical structure inherent in rhythm trees, yet it is important to acknowledge that this hierarchical nature cannot be represented within the piano roll model. Now, let us introduce our definition of texture.

R = (×, ×, ×, •, ⊥, ⊥, ×, •) .
Definition 4.4 (Texture). A texture T is a countable tuple of rhythms, i.e., T =

(R i ) i∈I ∈ R I ⊂ A T ×I

3

, where I is a countable set. A finite texture of size |I| is a texture such that |I| < ∞. This definition might not immediately appear closely related to the symbolic texture, but to give it meaning, we cannot separate it from the concept of harmony. This concept will be exposed in detail in the following section, but before let us show that a finite texture can be represented by a matrix.

Indeed, if T = (R i ) i∈I is a finite texture, we can represent each of its rhythms by a vector by choosing the tatum of each of them. We choose the tatum of all of them to make them quantized on the same grid, and then we stack them in a matrix in an order related to I. We show that in the following example.

Example 4.5. Let us consider the rhythms issued from the excerpt from the first movement of Beethoven's Piano Sonata No.14,Op.27 No.2. We may associate a rhythm for each pitch in the following way:

R 1 = 0, 1 12
for G  3 R 2 = 1 12 , 1 12
for C  4 R 3 = 2 12 , 1 12 for E4 R 4 = 0, 3 16 , 3 16 , 1 16
for G  4 4.1. Texture and Harmony or in vector form

R 1 = 0 R 2 = 0 R 3 = 0 R 4 = 0
In order to stack them into a texture, we need to put all of them in the same rhythm quantization. Since the tatum of R 1 , R 2 and R 3 is 1 12 and the one of R 4 is 1 16 , and the GCD between is 1 48 , we can rewritten the rhythms as

R 1 = 0 R 2 = 0 R 3 = 0 R 4 = 0
and stack them into

T = 0 R 1 R 2 R 3 R 4
.

Notice that we reversed the traditional order for stacking vectors into matrices in order to be more coherent with musical notation (where lower indices mean lower pitches and are place in the bottom).

Harmony

The term harmony can be even more intricate to define than texture. "Harmony" can refer to a field of study, a feature of a musical piece, or even an adjective.

In this context, harmony will be precisely defined in a mathematical manner to suit our purpose. In order to achieve this, akin to the process with texture, we first need to establish what we mean by a chord. Definition 4. 6 (Chord). A chord C is a finite subset of F or G F , i.e., an element of P(F) or P(G F ).

This definition is intentionally broad. The absence of specification for the space F and the group G F was intentional to keep the chord concept as general as possible. However, we will encounter particular chord types in specific scenarios. The following definition outlines four chord types that will be pertinent to our discussions. Definition 4.7 (Types of chords). We call • a positioned chord an element of P(N ),

• a chroma chord an element of P(N 12 ),

• a positioned pattern an element of P(Z),

• a pattern an element of P(Z 12 ).

These definitions comprise the four more general conceptions of chords; we list some examples in the following.

Examples 4.8.

1. We usually consider the chord F major to be FM = {F, A, C} ∈ P (N 12 ), that is a chroma chord.

2. The positioned chord CM 1,3,5,8 4 = {C4, E4, G4, C5} ∈ P(N ) is a root position of the C major chord.

3. A cadential six-four chord is the positioned pattern 6 4 = {-5, 0, 4} ∈ P(Z) where the 0 represents the tonic. 4. We say that the quality of C major is to be a major triad; the pattern of a major triad starting from its root is M = {0, 4, 7} ∈ P (Z 12 ).

The examples provided illustrate that what are commonly referred to as chords are in fact one of the different types of chords we proposed. The subsequent examples highlight instances where our definition identifies entities as chords, although this may not align with conventional usage.

Examples 4.9.

1. The silence ∅ is any kind of chord, i.e., ∅ ∈ P(N ), ∅ ∈ P(N 12 ), ∅ ∈ P(Z), ∅ ∈ P(Z 12 ) .

Texture and Harmony

2. A note n ∈ N forms a (positioned) chord {n} ∈ P(N ).

3. The major scale is the pattern {0, 2, 4, 5, 7, 9, 11} ∈ P (Z 12 ), where 0 represents the tonic.

4. The third mode of limited transposition of Messiaen is the pattern {0, 2, 3, 4, 6, 7, 8, 10, 11} ∈ P(Z 12 ).

5. The chromatic scale is Z 12 ∈ P(Z 12 ).

6. The V degree is the pattern {2, 7, 11} ∈ P(Z 12 ).

A first inversion of an

A minor chord is the positioned chord {C4, E4, A4}.

The augmented sixth chords may be described by the patterns following patterns:

Italian augmented sixth: {-4, 0, 6} ∈ P(Z)

French augmented sixth: {-4, 0, 2, 6} ∈ P(Z) German augmented sixth: {-4, 0, 3, 6} ∈ P (Z) where the 0 represents the tonic.

As in the case of rhythms, every finite non empty chord can be represented by a vector belonging to A M 2 for some M ∈ N * . This is done as follows: let C be a chord; then, |C| = N ∈ N * . Let C = {c 1 , c 2 , ..., c N }. We shall discuss two different cases:

F = Z or N and F = Z 12 or N 12 .
In the first case, let us consider the order in F given by the isomorphism with

Z. Then, we can define c 0 = min C ∈ F and c 1 = max C ∈ F, with gives 7 us M = (c 1 -c 0 ) + 1 ∈ N * . We define C ∈ A M 2 by ∀m = 0, 1, ..., M, C m = 1 if c 0 + m ∈ C 0 if c 0 + m ∈ C . (4.5)
The case when F = Z 12 or N 12 is tackled by using systematically a vector of size 12. We will also use always the index 0 corresponding either to the shift 0 either to the chroma C.

Chords will be plotted similarly as rhythms, but in this case, as vertical vectors, which will be exploited in the following section. While this representation of chords is adapted to our model, it takes a lot of place, so we alternate between this and the traditional presentation as sets.

Having defined chords, let us now proceed to define harmony. It is important to note that the definition of harmony we present is tailored to our specific objectives and may not align with a universal understanding of the term.

Definition 4.11 (Harmony).

A harmony H is an countable tuple of chords, i.e., H = (C i ) i∈I ∈ (P(F)) I ≃ P(F × I), with I being a countable set. A finite harmony of size |I| is a harmony where |I| < ∞.

We see that the notion of harmony is analogous to the notion of texture with the substitution of time by frequency and rhythm by chord. We can also establish the different kinds of harmonies depending on the different choices for the frequency space or group. Definition 4.12 (Types of harmonies). Let I be a countable set. We call • a positioned harmony an element of (P(N )) I ,

• a chroma harmony an element of (P(N 12 )) I ,

• a positioned harmonic pattern an element of (P(Z)) I ,

• a harmonic pattern an element of (P(Z 12 )) I .

Before showing how to combine textures and harmonies to create music, let us give some examples of harmonies.

Examples 4.13.

1. A cadential six-four in major mode is the positioned harmonic pattern ({-5, 4, 7, 12}, {-5, 2, 7, 11}, {-12, 4, 7, 12})

∈ (P(Z)) 3
where 0 is the tonic.

Its corresponding positioned harmony in F is

({C4, A4, C5, F5}, {C4, G4, C5, E5}, {F3, A4, C5, F5}) ∈ (P(N )) 3 . 3. The V 7 -I harmonic pattern is ({7, 11, 2, 5}, {0, 4, 7}) ∈ (P(Z 12 )) 2 .

Its corresponding chroma harmony in F is

({C, E, G, B  }, {F, A, C}) ∈ (P(N 12 )) 2 .
These examples are illustrated in Figure 4.2 These examples highlight the relationship between chords and patterns. While chords correspond to points in the space F, patterns are shifts within the group G F . In the upcoming discussion, we will simplify notation by using F interchangeably for either F or G F . This approach aims to enhance the ease of generalization when dealing with chords.

As in the case of textures, harmonies can be represented by a matrix.

4.1. Texture and Harmony function, that is

: P(F) → A F 2 C → C . (4.6)
We omit the characteristic function and say that a chord C belongs to A F 2 . In this case, a harmony is an element of A F 2

I ≃ A F ×I 2 .
Secondly, we use the residuated triplet induced by the lattice multiplication exposed in Section 2.1.3.2.

• : A 3 × A 2 → A 3 (a, b) → a • b = a if b = 1 ⊥ if b = 0 . (4.7)
Now we are able to define a rhythmed chord.

Definition 4.15 (Rhythmed chord). Let R ∈ A T 3 be a rhythm and C ∈ A F 2 be a chord. Then, the rhythmed chord R ⊗ C is defined as

R ⊗ C : T × F → A 3 (t, ξ) → R(t) • C(ξ) . (4.8)
Let us illustrate a rhythmed chord in the following example. Then, if we consider the placed rhythm

R 0 = t 0 + R ∈ A T   3 given by 8 t 0 = (1, 1, 0) ∈ T   and R ≡ {(0, 1 8 ), ( 1 8 , 1 8 ), ( 2 8 , 1 8 ), ( 3 8 , 1 8
)}, and the positioned chord Am = {A3, C4, E4} ∈ A N 2 , we have that R 0 ⊗ Am represents the two first beats of the left hand. Figure 4.4 illustrates this fact.

We can now define our fundamental mathematical object. Definition 4.17 (Harmonic texture). Let I be a countable set of indices. Let T = {R i } i∈I ∈ A T ×I 3 be a texture and let H = {C i } i∈I ∈ A F ×I 2 be a harmony. We define the harmonic texture generated by T and H as

T ⊗ H = i∈I R i ⊗ C i ∈ A T ×F 3 .
(4.9)

(a) First measure T and H are (strictly speaking) functions with I as domain, which means that their tensor product would have I × I as domain.

(1, 1, 0) (1, 3, 0) (2, 1, 0) Time (m, b, o) ⊥ ⋅ × (b) R 0 (1, 1, 0) (1, 3, 0) (2, 1, 0) Timestamps A3 C4 E4 Notes ⊥ ⋅ × (c) R 0 ⊗ Am A3 C4 E4 Notes 0 1 (d) Am
If we present texture and harmony with matrices, i.e., T ∈ A N ×I 3 and H ∈ A M ×I 2 , with N, M, I ∈ N * , we find that the harmonic texture is a matrix multiplication, replacing the sum by the supremum.

Indeed, if we have a texture T = (R i ) i∈I and a harmony H = (C i ) i∈I , both indexed by I, the harmonic texture T ⊗ H is written as that shall be compared with the matrix multiplication between two matrices A = (a ti ) and B = (b iξ ) that is given by

(T ⊗ H)(t, ξ) = i∈I R i ⊗ C i (t, ξ) = i∈I R i (t) • C i (ξ)
(A • B) tξ = (a ti ) • (b iξ ) = i∈I a ti • b iξ
with the same requirement of index compatibility.

Moreover, if we use the tensor notation for texture and harmony, i.e., T i n , H m j , the harmonic texture HT m n would be the (supremum) contraction on the index i, that is, using the Einstein summation convention, Mozart's Piano Sonata No.16 in C major,. Indeed, if we choose the following harmonies:

HT m n = T i n H m i := i T i n H m i (4.

It is used for instance in the left hand of

H 0 =   {E4} {G4} {C4}   H 1 =   {E4} {G4} {C4}   H 2 =   {F4} {G4} {D4}   H 3 =   {E4} {G4} {C4}   H 4 =   {F4} {A4} {C4}   H 5 =   {E4} {G4} {C4}   H 6 =   {D4} {G4} {B4}   H 7 =   {E4} {G4} {C4}   124
(t 1 + R) ⊗ C 0 ∨ (t 2 + R) ⊗ C 1 ∨ (t 3 + R) ⊗ C 0 ∨ (t 4 + R) ⊗ C 1 where ∀n ∈ {1, 2, 3, 4}, t n = (n, 1, 0) ∈ T   .
An even more revealing example is the start of the Lacrimosa of Mozart's Requiem in D minor, K.626.

Example 4.22. We consider together the parts of the Violins and Viola of the first two measures of the Lacrimosa of Mozart's Requiem in D minor,K.626 (Figure 4.8).

Let T ∈ A Q 3 3 be the texture

H 0 = ({D4, F4}, {C  5}, {D5}) H 1 = ({F4, A4}, {A5}, {B  5}) H 2 = ({E4, G4}, {D5}, {C  5}) H 3 = ({G4, C  4}, {C6}, {B  5}) H 4 = ({F4, D5}, {A5}, {D6}) H 5 = ({G3, E4}, {B  5}, {G5}) H 6 = ({A3, D4}, {E5}, {F5}) H 7 = ({A3, G4}, {A5}, {C  5}) .
Then, the excerpt may be represented by the supremum  ×I

3

.

A potential limitation of the notion of texture is that several different textures in the musical sense may appear as the same texture in our definition. For instance, the textures and are both represented by the texture R = (R i ) 3 i=0 with R i ≡ {( i 8 , 1 8 )} but they are fairly different musically. This flaw is solved by defining the concept of chord texture. With these textures, we can then model the left hand through chord textures with the following chords:

1. for Nocturne 1, mm. 1-2

C 1 = (B  2, F3, B  3, D  4, F4) C 2 = (B  2, F3, A3, E  4, F4) C 3 = (B  2, F3, B  3, D  4, F4) C 4 = (B  2, F3, B  3, D  4, F4)
2. for Nocturne 2, m. 1

C 1 = (E  2, G3, B  3, E  4, G4) C 2 = (E  3, A  3, C  4, D4, A  4) C 3 = (E  2, G3, B  3, E  4, G4) C 4 = (D2, G3, B  3, E  4, G4) .
Another interesting point of chord textures is that we may model melodies and motives with them 9 . Let us see an example. Actually, if we had chosen as texture the sub-texture {R i } 2 i=0 of texture T , we may even justify the second half of measure 2 and the first half of measure 3.

Generating Piano Rolls with Harmonic Textures and Mathematical Morphology

We have now a potentially powerful tool for generating piano rolls and explaining the way scores are created through the use of time-frequency entities (rhythms/chords and textures/harmonies). However, to generate complex tonal10 pieces, we need to be able to combine these harmonic textures into bigger structures. We do that by using morphological dilation and a tree structure. In Section 4.2.1, we present how to combine several harmonic textures by using the morphological greyscale dilation. Then, in Section 4.2.2, we organize several dilations into a tree, which enables us to create complex pieces through the use of hierarchical structures. Finally, in Section 4.2.3, we expose how to implement this computationally, in particular by the creation of an XML schema that defines a ScoreXML file that can be compiled into a PianoRoll data structure.

Combining Harmonic Textures with Dilation

In previous examples, we have seen how to produce music excerpts by using harmonic textures. However, there was always a moment in which we needed to combine different harmonic textures to render an excerpt. In these examples, the combination was made by using the supremum operator on shifted versions of the texture, with expressions like t + T , which emulated concatenation.

In this section, we will generalize both concatenation and superposition by using morphological dilation. Let us illustrate that with an example. We consider measures 0 to 8 of Beethoven's Piano Sonata No. [START_REF]an abbreviation for N ∪ {∞}[END_REF]. We are showing how to build them by using harmonic textures and morphological dilation. We now form the following chord textures:

Dmin 2 = T 2 ⊗ Dmin A7 2 = T 2 ⊗ A7 Dm 1 = T 1 ⊗ Dm Dmin ′ 2 = T ′ 2 ⊗ Dmin A7 ′ 2 = T ′ 2 ⊗ A7 AM 1 = T 1 ⊗ AM .

Generating Piano Rolls with Harmonic Textures and Mathematical Morphology

We want to combine them into a single piano roll P . For doing that, we use the morphological dilation

P = A ⊕ B := j∈J A j ⊕ B j
where A j ∈ A T This procedure is a way of generating infinite possibilities of piano rolls, using three elements:

B 1 = Dmin 2 A 1 = { (1, 1, 0), 0 , (2, 1, 0), 0 , (3, 1, 0), 0 } B 2 = Dmin ′ 2 A 2 = { (4, 1, 0), 0 } B 3 = A7 2 A 3 = { (
1. a collection of activations A = (A j ) j∈J ∈ A T ×Z×J 2 , generated by (a) a collection of textures T = (T j ) j∈J = (R j i ) j∈J i∈I j ∈ A

Q×I j ×J 3 , (b) a collection of harmonies H = (H j ) j∈J = (C j i ) j∈J i∈I j ∈ A F ×I j ×J 2 .
The generation of a piano roll P ∈ A T ×F 3 is summarized by the formula12

P = A ⊕ B = A ⊕ T ⊗ H (4.12) = j∈J A j ⊕ T j ⊗ H j = j∈J A j ⊕   i∈I j R j i ⊗ C j i   = j∈J i∈I j A j ⊕ R j i ⊗ C j i . (4.13)
Some interesting nuances merit to be outlined; the textures have Q × I j × J as domain, and the harmonies F × I j × J, which make the harmonic textures having as domain Q × F × J. This is compatible with the fact that the activations have T × Z × J as domain, since there is a sum action defined between T and Q, and between F and Z. However, another possibility, like using T for textures and Q for activations, can work. In fact, every combination that allows to have a sum between members of the different spaces (which is required by the definition of the dilation) is acceptable.

This leads us to explain the meaning of the 0 that is the frequency used in every point of A j ; this frequency means "no shift". However, if we consider the measures 150 to 158 of the same piece (Figure 4.14), we see that we might have represented this excerpt by using the exact same harmonic textures but changing the activations, making a translation in time of 150 • 3 8 and using -4 instead of 0 in the frequencies. These considerations lead us to the definition of compatibility. Definition 4.26 (Compatibility). Let T A and T B be either a time space either a time group. Let F A and F B be either a frequency space either a frequency group. We say that T A (resp. F A ) is compatible with T B (resp. F B ) with output T C (resp. F C ) if there is a sum operator + :

T A × T B → T C (resp. + : F A × F B → F C ).
activations 5,1,0),D5 ,(6,1,0),D5 ,(7,1,0 , and we would produce the same piano roll. We finish this section mentioning that this formalism can handle also the use of dynamics: if we use functions A j ∈ D T ×F and the residuated triplet exposed in Section 2.1.3.4, we would have

A 1 = { (1, 1, 0), D5 , (2, 1, 0), D5 , (3, 1, 0), D5 } A 2 = { (4, 1, 0), D5 } A 3 = { (
A ∈ D T ×F×J , T ∈ A Q×I j ×J 3 , H ∈ A Z×I j ×J 2
, and

P = A ⊕ T ⊗ H ∈ A T ×F D . (4.14)

Structuring a Piano Roll as a Tree

We have now a way of creating piano rolls that relies on applying the morphological dilation on activations and harmonic textures that share a set of indexes (that we called J). However, this is not very practical for creating bigger structures like complete pieces of music. A mere set of indexes J dos not tell that much about the structure of a piece and how it is organized. This is why we develop in this section a more refined strategy, that is organizing the piano roll as a tree.

Hierarchical structures are omnipresent in music analysis and composition; it is the core of the generative grammar [START_REF] Chomsky | Aspects of the theory of syntax[END_REF] that impulsed the Generative Theory of Tonal Music [START_REF] Lerdahl | A generative theory of tonal music[END_REF]. In particular, hierarchy is used to explain structure in music [START_REF] Lerdahl | An overview of hierarchical structure in music[END_REF], and its representation in form of a tree is a very used one [START_REF] Marsden | Towards computable procedures for deriving tree structures in music : Context dependency in GTTM and schenkerian theory[END_REF][START_REF] Orio | A measure of melodic similarity based on a graph representation of the music structure[END_REF][START_REF] Koelsch | Processing of hierarchical syntactic structure in music[END_REF][START_REF] Carnovalini | A new corpus for computational music research and a novel method for musical structure analysis[END_REF].

In this section, we are going to organize the structure of a musical piece in a tree formed by activations and piano rolls. This organization will permit us to understand better the fragments and their relative role. We call this tree a score tree.

where Exp.14 consists on mm. 1-28, Dev. 14 on mm. 29-41 and Rec. 14 on mm. 42-73. The split can go even further, but writing it in mathematical form is cumbersome. We choose rather a diagram shown in Figure 4. 15. While the leaves in the tree are not precisely harmonic textures, they exhibit a certain textural similarity that allows the possibility of generating them with few textures and harmonies.

For instance, let us show how to render A 2 as a score tree. We can indeed decompose it as in Figure 4.16 with with Maj b a being the chord formed by taking the elements a th to b th from the list Maj = (0, 2, 4, [START_REF]the set of positive rational numbers[END_REF][START_REF]the set of real numbers[END_REF][START_REF]the set of positive real numbers[END_REF][START_REF]ı: the imaginary unit, i.e., the number such that ı 2 = -1[END_REF][START_REF]the set of complex numbers[END_REF]14,16,[START_REF]an abbreviation for N ∪ {∞}[END_REF][START_REF]an abbreviation for Q ∪ {-∞, ∞}[END_REF]21) . Exp.

T 1 = 0 T 2 = 0 and C 1 = Maj 13 6 , H 1 = ({-4}, {5, 12}), C 2 = Maj

Th. A

A 1

A 2 A 3
Br.

P 1 Th. B B 1 B 2 B 3 B 4 Part 2 Dev. D 1 D 2 D 3 ReExp. Th. A ′ A ′ 1 A ′ 2 A ′′ 2 A ′ 3 Br. ′ P ′ 1 Th. B ′ B ′ 1 B ′ 2 B ′ 3 B ′ 4 ⊕ (1, 1, 0), C4
⊕{(0, 0), ( 28

A 2 RH T 1 ⊗ C 1 T 1 ⊗ C 2 T 1 ⊗ C 3 T 1 ⊗ C 4 LH T 2 ⊗ H 1 T 2 ⊗ H 2 T 2 ⊗ H 3 T 2 ⊗ H 4 ⊕{(0, 0)} ⊕{(0, 0)} ⊕{(1, 0)} ⊕{(2, 0)} ⊕{(3, 0)} ⊕{(0, -12)} ⊕{(0, 0)} ⊕{(1, 0)} ⊕{(2, 0)} ⊕{(3, 0)}

Computational Implementation

This framework allows us to generate piano rolls by using harmonic textures organized in score trees. In this section, we show how to implement this model computationally in an object-oriented language. In particular, we use two languages that will interact: XML and Python.

The idea is to describe the score tree as an XML file (that we call ScoreXML) with a XML Schema Definition (XSD) associated that validates it. Once the score tree is written, it can be compiled by a Python script to produce a piano roll (using morphological dilation). This piano roll, which is a Python object, can then be plotted 15 , transformed into a MIDI file or, in future extensions, written as a MusicXML file.

The full code is available in the repository that accompanies this thesis16 , so we limit ourselves to expose a reduced description of the objects (that serve both for the XML and for the Python objects). We recall that in Python the methods __neg__, __add__, __mul__ override the operators -, + and *, respectively. We also precise that frac stands for fractional numbers and int for integers.

We expose first the most basic elements of our model: time and frequency.

Time: { value: frac } Chapter 5

Mathematical Morphology Applied to Analyze Piano Rolls

The analysis of piano rolls using mathematical morphology was the first application of MM to music [START_REF] Karvonen | Using mathematical morphology for geometric music information retrieval[END_REF][START_REF] Karvonen | Using mathematical morphology for geometric music retrieval[END_REF][START_REF] Karvonen | Error-tolerant content-based music-retrieval with mathematical morphology[END_REF][START_REF] Lascabettes | Mathematical morphology applied to music (Master's Thesis[END_REF][START_REF] Lascabettes | Analyse de représentations spatiales de la musique par des opérateurs simples de morphologie mathématique[END_REF][START_REF] Lascabettes | Computational analysis of musical structures based on morphological filters[END_REF]. The primary focus of these works has been on motif detection. In this thesis, we extend the analysis by incorporating considerations related to texture and harmony. An attempt to analyze harmony using MM has already been made in a previous study (Romero-García, Bloch, et al., 2022).

In the previous chapter, we proposed a description of a piano roll P ∈ A T ×F as a combination of activations of harmonic textures by the formula1

P = A ⊕ T ⊗ H (5.1)
where

A ∈ A T A ×F A ×J A , T ∈ A T T ×I j ×J T , H ∈ A F H ×I j ×J H
are such that (A A , A T , A H ) form a residuated triplet, (T A , T T ) and (F A , F H ) are compatible (see Definition 4.26), and J and I j , ∀j ∈ J, are sets of indices.

This framework has enabled us to establish a method for constructing music using elementary components known as harmonic textures. These textures are positioned within the piano roll by means of morphological dilation between them and the activations. This approach highlights two fundamental parameters of music: texture and harmony.

In this chapter, our objective is to perform the reverse operation: can we extract the underlying parameters from a given piano roll? In mathematical terms, this can be reformulated as follows: given a piano roll denoted as P , can we determine the values of A , T , and H in such a way that Equation (5.1) is satisfied? Indeed, this task is highly challenging. It is not even well-posed due to the existence of a trivial solution where a single texture, harmony, and activation could be assigned to each note. The more pertinent question is: among the countless possible choices, which values of A , T , and H would constitute the optimal solution?

Providing a definitive answer to this question is not within our expectations, as we believe that a unique solution might not exist. Instead, in the upcoming sections, we delve into the difficulties, theoretical considerations, and practical challenges we encounter in this endeavor.

The chapter is structured into three main sections. In Section 5.1, we employ morphological erosion to analyze piano rolls, specifically choosing harmonic textures as structuring elements. In Section 5.2, we continue to employ erosion, this time with textures as structuring elements. This analytical approach to piano rolls presents several challenges that we address by using graphs. The payoff of this approach is significant, leading to the compression of piano rolls into vertical chords. With these chord sequences as input, we delve into Section 5.3, where we once again employ erosion, this time using harmonies as structuring elements. The interpretation of the erosion proves to be highly productive, enabling us to tackle the Roman numeral analysis problem using this technique and once again making use of graphs.

Analyzing Piano Rolls with Harmonic Textures

We begin by examining what insights the theory of mathematical morphology can offer regarding a piano roll generated with Equation (5.1). To enhance clarity, we will accompany each step with illustrative examples. Let P ∈ A T ×F be a piano roll. In this section, we do not delve into the particular attributes of A, T , and F. Instead, we assume that these aspects have been appropriately determined. We recall that (A T ×F , ) is a complete lattice. Furthermore, we assume that (A, A, A) is a residuated triplet and that T × F is compatible with itself.

In this section, our objective is to find A ∈ A T ×F ×J and B ∈ A T ×F ×J such that

P = A ⊕ B.
Similar research is currently be done by [START_REF] Lascabettes | Mathematical models for the discovery of musical patterns, structures and for performances analysis[END_REF] To approach this task, we can begin by employing the adjoint operator of dilation ⊕, which is the erosion ⊖. For instance, if we let B 1 ∈ A T ×F , applying the erosion 5.1. Analyzing Piano Rolls with Harmonic Textures to P results in:

P ⊖ B 1 := A 1 ∈ A T ×F . (5.2)
Since ⊕ and ⊖ are adjoint operators forming a dilation/erosion pair, we know that • := ⊕ • ⊖ is a closing and • := ⊖ • ⊕ is an opening2 . Consequently, owing to the anti-extensivity property of the opening, we can assert that

A 1 ⊕ B 1 = (P ⊖ B 1 ) ⊕ B 1 = P • B 1 P .
(5.3)

We can proceed by selecting another B 2 ∈ A T ×F and repeating the process to obtain

A 2 ⊕ B 2 = (P ⊖ B 2 ) ⊕ B 2 = P • B 2 P .
(5.4) with A 2 := P ⊖ B 2 . At this point, we can stack B 1 and B 2 into B; to achieve this, we define B = (B 1 , B 2 ) ∈ A T ×F J ≃ A T ×F×J , with J = {1, 2}. Similarly, we can stack A 1 and

A 2 into A = (A 1 , A 2 ) ∈ A T ×F×J .
However, attempting to directly define an erosion P ⊖ B faces compatibility issues due to the differing domains T × F and T × F × J. To address this issue, we introduce the trivial group (0, +) with 0 = {0}, which acts on any set X through the action + : X × 0 → X, (x, 0) → x. In this context, we apply this action to the set J. Additionally, we use the canonical bijection ι : A T ×F → A T ×F ×0 P → P := ι(P ) : T × F × 0 → A (t, ξ, 0) → P (t, ξ) .

(5.5)

Since the domains T × F × 0 and T × F × J are now compatible, the erosion P ⊖ B ∈ A T ×F ×J becomes well defined. Given the isomorphism, we can simplify the statement by noting that the domain of the erosion is simply A T ×F . Recalling that B = (B j ) j∈J , the explicit definition of this erosion is given by .6) Before delving into the applications of these erosions (which will be used extensively throughout this chapter), let us first define the adjoint dilation of this erosion.

ε B : A T ×F → A T ×F ×J P → (P ⊖ B j ) j∈J . ( 5

Analyzing Piano Rolls with Harmonic Textures

Now we have erosion and dilation defined, and thus opening and closing. By letting P ∈ A T ×F being a piano roll, B ∈ A T ×F ×J a stack of structuring elements, and A ∈ A T ×F ×J a stack of activations, we can write

P ⊖ B ∈ A T ×F ×J A ⊕ B ∈ A T ×F P • B ∈ A T ×F A • B ∈ A T ×F ×J .
(5.9)

Let us return to the process of adding more B j to B with the property ∀j ∈ J, P • B j P which can be restated using Equation (5.9) as P • B P . At some point, it might3 happen that P • B = P , allowing us to potentially conclude the search.

Let us illustrate this process with an example. For the examples, we make the following choices for time, frequency and amplitude: our space is T   × N with its corresponding group (Q × Z, +), and the residuated triplet is given by the lattice multiplication

• : A 3 × A 2 → A 3 . This means that P ∈ A T   ×N 3 , B ∈ A Q×Z×J 3 , leading to A ∈ A T   ×N ×J 2 .
Now, let us consider the piano roll P in Figure 5.1, which represents the first two measures of the third movement of Beethoven's Piano Sonata No.14,Op.27 No.2. We want to evaluate the different choices for B = (B j ) j∈J such that P = P • B. To measure the quality of these choices, we introduce the concept of redundancy of a decomposition. For this purpose, we first define a measure for a piano roll with amplitude ranges A 2 or A 3 . Definition 5.2 (Measure of a piano roll). Let P ∈ A T ×F be a piano roll. For all a ∈ A, let µ a : Σ T ×F → R + be a measure, where Σ T ×F is a σ-algebra over T × F . Then, we define the measure of P relative to {µ a } a∈A as

|P | = a∈A µ a ({(t, ξ) ∈ T × F : P (t, ξ) = a}) ∈ R + .
(5.10)

It is evident by the definition that the measure of a piano roll is an increasing function from (A T ×F , ) to (R + , ≤), i.e.,

P 0 P 1 ⇒ |P 0 | ≤ |P 1 |.
The choice of {µ a } a∈A depends on the choices of T , F and A. When dealing with activations piano rolls, i.e., A ∈ A T ×F 2 , we use

|A| = µ δ (supp(A))
(5.11) where µ δ is the discrete measure defined by µ δ (X) = |X|, ∀X ⊆ T × F .

However, when using a piano roll with amplitude represented as the rhythmic range A 3 , we use a different measure. Let P ∈ A T   ×N

3

. Since the time-frequencies (t, ξ) ∈ T   × N where P (t, ξ) is the attack (×) form a discrete set, we use a discrete measure for them. However, for the values where P (t, ξ) is the sustain (•) we use a different measure: the product measure between the Lebesgue measure induced by the isomorphism between Q and T   , and µ δ , the discrete measure presented before. We call it µ. This results in the following measure definition:

|P | = c × • µ δ ({(t, ξ) ∈ T × F : P (t, ξ) = ×}) + c • • µ(supp(P ))
( 5.12) where c × , c • ∈ R + are the coefficients that control the relative weight given to the measure of attacks and the measure of sustains, respectively. Now let us compute the measure of the piano roll shown in Figure 5.1. As there are 25 and 27 , the measure of the attacks is 52c × , and the measure of the sustains is (25 × 1 8 + 27 × 1 16 )c • , leading to a total measure of P :

|P | = c × • 52 + c • • 25 8 + 27 16 = 52c × + 77 16 c • .
With this framework in place, we can introduce the concept of redundancy.

Definition 5.3 (Redundancy). Let P ∈ A T ×F and B ∈ A T ×F ×J . We define the redundancy of B over P denoted by ρ(B, P ) as 5.13) where | • | is defined as in Equation (5.12).

ρ(B, P ) = j∈J |P ⊖ B j | • |B j | -|P | |P | ∈ R , (
The redundancy may be negative. Indeed, if we have P • B P it may happen that j∈J |P ⊖ B j | • |B j | < |P |. When P • B = P , the redundancy is positive and we give it in percentage. Moreover, the redundancy is not limited to 100 %.

While the redundancy relies on the choice of (c × , c • ) ∈ (R + )

2

, to gain an overview, we can use two reference sets of coefficients: (c × , c • ) = (1, 0) and (c × , c • ) = (0, 1). We denote the resulting redundancies as ρ × and ρ • , respectively. Now that we have established a measure to assess the efficiency of different choices for B, we can proceed to evaluate a couple of scenarios. Specifically, we present two examples: one involving a trivial choice and another involving a more effective one.

The trivial choice is to select B = (B j ) 2 j=1 , where we choose the note values that are already present in the score. That is, We can now compute the corresponding redundancies: We have that A

ρ × = 2 j=1 |A j | • µ × (B j ) -µ × (P ) µ × (P ) ρ • = 2 j=1 |A j | • µ • (B j ) -µ • (P ) µ • (P ) = 27 • 1 + 52 • 1 -52 52 = 27 • 1 16 + 52 • 1 16 -77 16
= (A j ) j∈J = P ⊖ B is A = (1, 1) (1, 3) (2, 1) (2, 3) (3, 1) Time (m, b) C 2 B2 A3 G4 F5 Pitch A 1 A 2 A 3 A 4 A 5
The redundancies are

ρ × = 5 j=1 |A j | • µ × (B j ) -µ × (P ) µ × (P ) = 9 • 2 + 1 • 11 + 3 • 4 + 2 • 4 + 2 • 4 -52 52 = 5 52 ≈ 9.6 % ρ • = 5 j=1 |A j | • µ • (B j ) -µ • (P ) µ • (P ) = 9 • 1 4 + 1 • 11 8 + 3 • 1 4 + 2 • 1 4 + 2 • 1 4 -77 16 77 16 = 9 77 ≈ 11.7 % .
These redundancies are fairly small. To reduce them to the minimum, we are using an alternative approach that involves modifying A to contain fewer activations. This strategy will be explored in the following section.

Analyzing Piano Rolls with Textures

In the previous section, we showed how to use erosion and a measure to analyze a piano roll P using a stack of harmonic textures B. This involved searching for A and B such that P = A ⊕ B. However, our original intention was to take this a step further: to find A , T and H such that P = A ⊕ T ⊗ H .

In this section, we attempt to find these elements, with a specific focus on the texture component. Let us consider an example involving measures from later in the same musical piece, as depicted in Figure 5.2. This excerpt can be built by using a single texture applied to several harmonies.

Analyzing Piano Rolls with Textures

If we choose the texture we have that 5.14) where J = {1, 2, ..., 8}, I = {1, 2, 3} and

P = A ⊕ T ⊗ H ∈ A T   ×N 3 (
A = (A j ) j∈J = (53, 1, 0) + j -1 2 , G  3 8 j=1 ∈ A T   ×N ×J 2 T = (T ) 8 j=1 ∈ A Q×I×J 3 H = (H j ) 8 j=1 ∈ A N ×I×J 2 .
The redundancies are

ρ × = 8 j=1 |A j | • µ × (T ⊗ H j ) -µ × (P ) µ × (P ) = 17 + 7 • 13 -108 108 = 0 % ρ • = 8 j=1 |A j | • µ • (T ⊗ H j ) -µ • (P ) µ • (P ) = 17+7•13 8 -108 8 108 8 = 0 % .
The challenge lies in determining how to get T , H , and A from the given piano roll P . We believe that this task is extremely complex, so we will proceed by assuming we have the texture T and demonstrate how to obtain H and A .

Analyzing Piano Rolls with Textures

Let us now transform our problem into a problem on sets: we call 5.20) and we want to find the A min ∈ A P T that are minimal in the sense of ⊆. Let us give a brief overview of how such a task can be achieved. Let X = {x 1 , x 2 , ..., x n } be a set with |X| = n ∈ N * . We call X i 1 ,i 2 ,...,i k = X \ {x i 1 , x i 2 , ..., x i k }. The lattice (P(X), ⊆) of subsets of X with |X| = 3 is shown in Figure 5.3 with the arrows meaning inclusion. If we set X = A P T and remove the elements that do not belong to A P T , we can identify a minimal element as one that has no outgoing edges. However, to exhaustively check all the edges and explore the entire graph6 , we would need to perform A P T ! checks. This quickly becomes infeasible; for instance, in the excerpt of Figure 5.2 of 4 measures, we have A P T = 247, 247! ≈ 10 484 , which is astronomically large.

A P T = {A ⊆ T × F × I : P = A ⊕ T ⊗ 0} (

X

X 3 X 2 X 1 X 2,3 X 1,3 X 1,2 X 1,2,3
Nonetheless, there is a theoretical insight that can aid us. Since we are interested in exploring the elements of A P T , which are those A satisfying A ⊕ T = P , the following property holds: if A 1 ⊆ A 2 , then A 1 ⊕ T ⊆ A 2 ⊕ T . Therefore, if we find a A such that A ⊕ T ⊆ P , there is no need to explore any subset A ′ ⊆ A .

This insight helps to reduce the number of elements to check. Nevertheless, even with this optimization, the worst-case complexity remains O(n!) with n = A P T . However, we know that there are 2 |A P T | subsets of A P T and then we can check if they belong to A P T one by one. This approach leads to the graph depicted in Figure 5.4 and a time complexity of O(2 n ). In order to exploit the property mentioned before, we avoid the exploration of deeper nodes once it is determined that they do not belong to A P T . This pruning strategy significantly reduces the number of paths explored. While this approach is better in terms of time complexity, it has a drawback: we no longer know if a node is minimal with this graph since there are missing edges in terms of inclusion; for instance, if we attain X 1,2 as a minimal node and later we see that X 2 works but not X 2,3 , we may be tempted to say that X 2,3 is minimal (since there is no outgoing edges) but this is not true since X 2 is already an option.

X

X 1 X 2 X 3 X 1,2 X 1,3 X 2,3 X 1,2,3
To determine if a node without outgoing edges in this graph is truly minimal, it is necessary to compare it against all other minimal subsets found earlier. While this is possible, we still have a time complexity of O(2 n ), which is intractable: in our example with A P T = 247, the number of subsets to consider is 2 247 ≈ 10 74 , which is far beyond the capabilities of current computing resources.

Linear Approach

In this section, we present an alternative approach to tackling this problem. This approach leverages the inherent topology of our space and converts the problem into a shortest path problem within a directed acyclic graph (DAG), ultimately transforming it into a linear problem.

As previously discussed, the challenge involves exploring all possible combinations of elements in A P T to identify the minimal subsets. While this is initially an exponential problem, we can capitalize on the structure of the set A P T to reformulate it into a linear problem in the size of the graph.

The central insight guiding our strategy is as follows: since each rhythm R i has a support contained inside an interval [s i , e i ] ⊆ Q, any element in the piano roll denoted as (t, ξ, P (t, ξ)) ∈ T × F × A can be attributed to an activation within [te i , ts i ] × {ξ} × {i} ⊆ T × F × I. In essence, covering an element (t, ξ, P (t, ξ)) necessitates selecting an element from a restricted set A (t,ξ) ⊂ A P T capable of producing the value P (t, ξ).

We use this principle to construct a graph wherein each element of the piano roll has the activations capable of covering it. Subsequently, we traverse paths within this graph, ensuring the inclusion of at least one activation for each element of the piano roll.

Let us illustrate the process of creating the graph using a fragment from our example, as depicted in Figure 5.5. We only consider the measure 54 of Beethoven's Piano Sonata No. 14, Op. 27 No.2. To construct the graph G = (V, E) that represents the potential activations covering each element of P , we begin by determining the vertices V.

Each vertex will be an element of T × T × F × I denoted by (t P , t A , ξ, i) where

• t P ∈ T is the time coordinate of the element of P that is covered,

• t A ∈ T is the time coordinate of the activation that covers it,

• ξ ∈ F is the frequency coordinate of both the element of P and the activation7 ,

• i ∈ I is the index of the activation.

The resulting vertices of the graph are depicted in Figure 5.6. The coordinates (t P , ξ) are used to create the axes that map the elements of the piano roll. Each element belonging to a cell is of the form (t A , i), indicating the timestamp of the activation and the index. Please note that in the diagram, pitch notations are used based on the default system, rather than the ones present in the actual score (for instance, we use E  instead of D  and G instead of F  ).

Having defined the vertices of the graph, our next task is to determine the edges. As a reminder, our goal is to find a path that passes through at least one element of each cell of the grid defined by (t P , ξ), skipping those where there are no elements. This ensures that each element of the score is covered by at least one activation. To model this, we need to select edges that connect elements from each cell of the grid. Furthermore, we will arrange these edges in a topological order, creating a directed acyclic graph that allows us to find a linear solution to the shortest path problem.

Various edge choices are possible, but we opt for the following approach:

1. We arrange the non-empty different cells in the lexicographical order going first through the frequency axis and then through the time axis.

2. For two consecutive cells, we connect every node in the first cell with a node in the second cell.

The result of this process leads to a directed acyclic graph. Moreover, the graph is a chain of bipartite graphs. This concept will be exposed in Section 5.2.4. The resulting graph is depicted in Figure 5.7. Since the number of edges is big and makes the graph difficult to understand, we zoom to the vertices that are on the cells (3, 0), C  3 and (3, 1 8 ), B  3 in Figure 5.7b. Now, each path in this graph that goes from a node at the bottom left to a node at the top right represents a sequence of nodes that corresponds to the activations selected from A to cover P .

The challenge at this point is to devise a method for determining the optimal path. The objective is to minimize the number of distinct activations used, and this as a toy example, which will provide insights into solving the larger problem of finding minimal activations.

Imagine assigning a letter to each node in the graph presented in the previous example. The graph is illustrated as follows:

S A B C D A A E
. Now, the goal is to find a path from node S (start) to node E (end) that uses the minimum number of distinct letters. In other words, we aim to minimize the number of different letters encountered along the path (excluding the start and end nodes). We can assign a weight of 1 to an edge if the letters on the connected nodes are different, and a weight of 0 if they are the same. This gives us the weighted graph: This issue can be addressed using the concept of the derived graph. By differentiating the graph once, we obtain the following modified graph (with artificial start and end nodes denoted as SS and EE):

1 1 0 0 1 1 1 1 0 0 0 0 1 1 SA AC AD SB BC BD CA DA AA AE SS EE . v 1 1 v 2 1 v 2 2 v 2 3 v 3 1 v 3 2 v 4 1 v 4 2 v 4 3 v 4 4 v 5 1 v 5 2 v 6 1 .
We denote the number of vertices of a graph G by |G| V and the number of edges by |G| E . The k th derived graph of chains of complete bipartite graphs have the following properties.

Proposition 5.8. Let K n 1 →n 2 →...→n L be a chain of complete bipartite graphs. Then,

∀k ∈ {1, 2, ..., L -1}, 1. ∂ k K n 1 →n 2 →...→n L is a chain of bipartite graphs of size L -k, 2. ∂ k K n 1 →n 2 →...→n L V = L l=k+1 k m=0 n l-m , 3. ∂ k K n 1 →n 2 →...→n L E = L l=k+2 k+1 m=0 n l-m .
Proof. Let us prove the first statement by induction over k.

∂ 0 K n 1 →n 2 →...→n L = K is indeed a chain of bipartite graphs of length L - 0 = L. Now, if ∂ k K n 1 →n 2 →...→n L is a chain of bipartite graphs of size L -k, then we have V k k+1 , V k k+2 , ..., V k
L disjoint non-empty sets such that the vertices in V l are connected only to those on V l+1 . Thus, the edges can be split into E k+1→k+2 , E k+2→k+3 , ..., E L-1→L , where E l→l+1 is the set of edges that connect elements of V l with elements of V l+1 . They are then disjoint and an element of

E l→l+1 is connected to one in E l ′ →l ′ +1 in ∂ k+1 K n 1 →n 2 →...→n L only if l + 1 = l ′ .
It remains to be seen that the E l→l+1 are non empty. Their cardinal is equal to the cardinal of V k l times n l+1 , so let us compute the cardinal of V k l . We know that the elements of V k l are paths of length k + 1. Since the graph is complete, we know that there are k m=0 n l-m paths, which implies that

|V k l | = k m=0 n l-m and |E l→l+1 | = k m=0 n l-m n l+1 = k+1 m=0 n l+1-m = 0. 166

Analyzing Piano Rolls with Textures

Finally, let us prove the formulas 2 and 3:

∂ k K n 1 →n 2 →...→n L V = L l=k+1 |V k l | = L l=k+1 k m=0 n l-m ∂ k K n 1 →n 2 →...→n L E = L-1 l=k+1 |V k l |n l+1 = L-1 l=k+1 k m=0 n l-m n l+1 = L-1 l=k+1 k+1 m=0 n l+1-m = L l=k+2 k+1 m=0 n l-m .

Modeling the Problem as a Shortest Path Problem

We have currently all the tools that we need to solve our initial problem (to find a minimal set of activations). Indeed, the graph presented in Figure 5.7 is a chain of complete bipartite graphs of length 26: each cell is a set of vertices V l and they are ordered in the lexicographical order starting by the frequency. We have then a graph isomorphic to K 1→2→2→2→2→3→2→2→3→2→3→3→2→2→3→3→3→3→4→3→3→4→2→3→3→2 .

We want now to find a path that starts from one of the first nodes to one of the end nodes. To model that as a shortest path problem, we add an artificial node at the beginning that is connected to all the first nodes and one that is connected to all of the end nodes (thus having a complete bipartite graph of length 28 with n 1 = n 28 = 1).

We want now to assign weights to this graph. We saw in Section 5.2.3 that we can know information from k ∈ N * vertices before if we use the k -1 th derived graph. In our case, we want to know if an activation (t A , ξ, i) is present already in the path, thus assigning 0 to the weight of the edge. To do that, we need to compute the maximum number of cells that can separate two activations. We call it

K T = max{ e i -s i ∆ P : i ∈ I} (5.22)
where T = (R i ) i∈I and supp(R i ) ⊆ [s i , e i ] ⊆ Q and ∆ P is the time resolution of the piano roll to perform the computations8 .

In our case, if we choose as ∆ P the tatum, i.e., , we have

K T = max{ 1 8 -0 1 8 , 3 8 -1 8 1 8 , 4 8 -3 8 1 8 } = max{1, 2, 1} = 2
Instead of the lexicographical order that starts by frequency we can use the one that starts by time and solve the problem with a derived graph of first order. The problem is that this approach, where we only take into account the number of different activations, is that it gives a huge quantity of shortest paths. We would like to refine this approach to find a better path.

The Sparsity of Time Activations

In order to improve the quality of our results we introduce the notion of sparsity of time activations. The idea behind this is that we want to have the textures activated in the minimum possible times. Moreover, we require a property for the chosen activations: when an activation is chosen and has timestamp t A and index i, we need to choose activations with the same timestamp and with the remaining indices in I \ {i}. This is related to the concept that a texture should be complete once it is activated.

This allows us to eliminate several activations that cannot be chosen. Indeed, if we consider the contraction

λ(t) = i∈I ξ∈F A (t, ξ, i) = i∈I ξ∈F A i (t, ξ)
we have the timestamps where this property can hold.

Then, we multiply 9 A by λ and we get the filtered activations that might have this property.

However, even after filtering A with λ, we need to ensure that the property aforementioned holds. To do that, we need to go through every element in frequency. This is why we presented the graph in the lexicographical order of frequency first. We have then that there are at most

K F = max{| supp( P | t )| : t ∈ T }
(5.23) 9 The multiplication is performed between an element of A T ×F ×I 2 and an element of A T 2 , which means that it should be understood as A

• λ : T × F × I → A 2 , (t, ξ, i) → A (t, ξ, i) • λ(t).
We finish by giving some bounds of our problem: while the problem is now linear in N (the number of elements of P ), it has a polynomial component due to the derivation. Indeed, if we call our graph G and we set n max = max{n l : l ∈ {1, 2, ..., L}, we have the bounds

|G| V ≤ (N + 1)n max , |G| E ≤ (N + 1)n 2 max , |∂ K G| V ≤ (N + 1 -K)n K+1 max , |∂ K G| E ≤ (N -K)n K+2 max .

Conclusions

While the utilization of the derived graph proves to be a powerful theoretical tool for transforming the problem into a linear one, it comes at the price of a polynomial bound in the size of texture and number of simultaneous elements in the score. This limitation presents a significant drawback that often renders practical computations unfeasible.

We consider the challenge of finding the minimal activations as a highly intricate one, closely tied to the inherent complexity of music and its extensive combinatorial nature.

Nonetheless, we believe that this approach offers a profound method for music analysis, as it delves into the essence of each note. Future research might attempt to address this problem using machine learning techniques. While the abstract nature of the problem presents significant difficulties, there could exist various heuristics (such as identifying repeated patterns) that facilitate pruning possibilities in rhythmically complex scenarios.

In particular, an approach centered around genetic algorithms, with evolving agents attempting to cover the score, could enhance efficiency and explanatory power. This methodology could mirror the creative process in musical composition, where motifs are often modified and used as foundational material. Exploring such an approach will be a key direction for our future research in this field, although it extends beyond the scope of this thesis.

Analyzing Piano Rolls with Harmonies

In the preceding sections, we analyzed piano rolls using both harmonic textures and textures. In this section, we extend our analysis of piano rolls incorporating a frequency object: harmonies. The principal innovation of this section is the development of what we refer to as the tonal graph, a construct that enables us to reframe the harmonic analysis challenge as a shortest path problem.

Analyzing Piano Rolls with Harmonies

To initiate our exploration, let us delve deeper into the harmonic analysis problem itself.

Harmonic Analysis

The harmonic analysis problem involves providing a harmonic interpretation of a musical score, or a fragment of it, by using harmonic features such as chords, scales, or Roman numerals. Our focus lies in the task of Roman numeral analysis, which entails assigning a label expressed as a Roman numeral, along with potential additional indications, to a each fragment of the score. These indications, coupled with the knowledge of the tonic, allow us to deduce details about the chord, its inversion, and any sevenths or added notes [START_REF] Tymoczko | The romantext format: A flexible and standard method for representing roman numeral analyses[END_REF].

There exist diverse approaches to conduct Roman numeral analysis. In this work, we confine our focus to identifying the tonic, the Roman numeral label, and any added notes. We abstain from specifying the inversion; however, this aspect can be addressed using a straightforward dictionary-based lookup approach. Figure 5.8 provides an illustration of a Roman numeral analysis of the previously provided excerpt. Roman numeral analysis was initially introduced by Weber, 1832, building upon the groundwork laid by Rameau, 1722. While other techniques, such as Riemann's functional analysis [START_REF] Riemann | Vereinfachte harmonielehre; oder, die lehre von den tonalen funktionen der akkorde[END_REF], have garnered the interest of music analysts [START_REF] Agmon | Functional harmony revisited: A prototype-theoretic approach[END_REF][START_REF] Illescas | Harmonic, melodic, and functional automatic analysis[END_REF][START_REF] De Haas | Automatic functional harmonic analysis[END_REF], they maintain a close relationship with Roman numeral analysis. Nevertheless, as highlighted by [START_REF] Tymoczko | The romantext format: A flexible and standard method for representing roman numeral analyses[END_REF], the translation between Roman numerals and tonal functions is not symmetrical, with the former generally encapsulating more information than the latter. Hence, we opt for Roman numeral analysis due to its richness in explanatory capabilities.

Automatic Roman numeral analysis begins with the contributions of [START_REF] Winograd | Linguistics and the computer analysis of tonal harmony[END_REF]Maxwell, 1992, who proposed rule-based algorithms. Subsequently, Tem-perley introduced another rule-based algorithm [START_REF] Temperley | An algorithm for harmonic analysis[END_REF]. Further progress in this direction encompasses works such as [START_REF] Pardo | Algorithms for chordal analysis[END_REF][START_REF] Temperley | A bayesian approach to key-finding[END_REF][START_REF] Temperley | The cognition of basic musical structures[END_REF][START_REF] Illescas | Harmonic, melodic, and functional automatic analysis[END_REF][START_REF] Temperley | A unified probabilistic model for polyphonic music analysis[END_REF][START_REF] De Haas | Automatic functional harmonic analysis[END_REF].

Additionally, a significant advancement within the harmonic analysis domain is the construction of datasets. Although a substantial dataset emerged in 2002 [START_REF] Goto | RWC music database: Popular, classical and jazz music databases[END_REF], it was not until 2015 that we witnessed a proliferation of Roman numeral analysis datasets for classical music [START_REF] Devaney | Theme and variation encodings with roman numerals (TAVERN): A new data set for symbolic music analysis[END_REF][START_REF] López | Automatic harmonic analysis of classical string quartets from symbolic score[END_REF][START_REF] Chen | Functional harmony recognition of symbolic music data with multi-task recurrent neural networks[END_REF][START_REF] Neuwirth | The annotated beethoven corpus (ABC): A dataset of harmonic analyses of all beethoven string quartets[END_REF][START_REF] Tymoczko | The romantext format: A flexible and standard method for representing roman numeral analyses[END_REF], with [START_REF] Micchi | Not all roads lead to rome: Pitch representation and model architecture for automatic harmonic analysis[END_REF] consolidating many of these datasets into a meta-corpus. This influx of datasets has paved the way for the development of machine learning techniques; some of these datasets include neural network architectures for automatic analysis [START_REF] Chen | Functional harmony recognition of symbolic music data with multi-task recurrent neural networks[END_REF][START_REF] Micchi | Not all roads lead to rome: Pitch representation and model architecture for automatic harmonic analysis[END_REF][START_REF] López | AugmentedNet: A roman numeral analysis network with synthetic training examples and additional tonal tasks[END_REF].

The Tonal Graph

In this work, we introduce an alternative approach distinct from rule-based and machine learning methods: a novel model for Roman numeral analysis, framed as a shortest path problem within a chain of bipartite graphs.

To construct this graph, we leverage the outcomes of the preceding section, comprising a collection of activations denoted as A ∈ A T ×N ×I 2 . However, we contract these activations across the dimension I to yield A = i∈I A = i∈I A i ∈ A T ×N 2 . This aggregated representation is depicted in Figure 5.9. Furthermore, the harmonic attributes we aim to identify are independent of the pitch octave. Thus, we use activations up to the octave, considering them as a chroma roll. This leads to the representation depicted in Figure 5.10.

In order to build the tonal graph, we use as input the activations and a harmony. The harmony represents the different Roman numerals we allow in our analysis. For instance, a common harmony that we can use is the one presented in Table 5 The final aspect we need to address before detailing how we construct the tonal graph is the concept of erosion applied to a piano roll by a harmony. Our input consists of A ∈ A T ×N 12 2 along with a harmony represented by an element H ∈ A Z 12 ×I 2 .

To achieve this, we apply a similar approach to the one discussed in Section 5.2 concerning the erosion of a texture. In this case, we treat A as an element of A T ×N 12 ×0 2 , and H as an element of A 0×Z 12 ×I 2 . The erosion operation is then defined as follows: .25) where H = (C i ) i∈I . We now expose the creation of the tonal graph. be a chroma roll. Let H = A Z 12 ×I 2 be a chroma harmony. We consider the erosion of A by H, A ⊖ H ∈ A T ×N 12 ×I 2 .

A ⊖ H : T × N 12 × I → A 2 (t, ξ, i) → (A ⊖ C i )(t, ξ) (5

Analyzing Piano Rolls with Harmonies

We can then build the tonal graph that may be represented as .

We now aim to reformulate the task of identifying the correct Roman numeral analysis10 into a shortest path problem.

To achieve this, we must assign weights to each edge, such that we have a shortest path problem. One of the simpler yet effective methods involves assigning a weight denoted as w using the formula: 5.26) which means that we count the number of modulations. This way, our shortest path problem is a minimization of modulations (which makes a lot of musical sense).

w (t 1 , ξ 1 , i 1 ), (t 2 , ξ 2 , i 2 ) = 1 if ξ 1 = ξ 2 0 if ξ 1 = ξ 2 (
We add a start node S connected to the first nodes and an end node E connected to the final nodes and we have then the graph It is important to highlight that the algorithm favored V over viiº when the chord was V 7 (and thus the erosion detected both of them) solely due to the sequence order in the harmony, while both options shared an identical path length. The issue of the sevenths will be discussed later.

Application to Other Pieces

Let us show its application to more complex pieces to see its effectiveness. We start by Bach's Prelude 1 in C major from the first book of the Well Tempered Clavier. Following our established approach, we take the output from the preceding section, contract it in I, and generate a chroma roll of activations to which we apply the erosion and generate the tonal graph.

The expected result that we target is exposed in Figure 5.11. The input chroma roll11 is presented in Figure 5.12.

We choose as previously the harmony of Table 5.1. The resulting graph12 is presented in Figure 5 5.2 presents a comparative analysis between the expected analysis (the one of Figure 5.11) and the outcome derived from our shortest path algorithm. Focusing solely on the agreement between the tonic and the Roman numeral (excluding sevenths and assuming that 7º corresponds to vii) we observe 22 out of 32 matching instances, resulting in an accuracy rate of 68.75 %.

In the harmony table provided in Table 5.1, our consideration was limited to triads. However, for a thorough analysis of tonal music, the inclusion of seventh chords is necessary. Yet, the challenge arises when seventh chords can be perceived as a supremum of two triads; for instance, ii∨IV = {2, 5, 9}∨{5, 9, 10} = {2, 5, 9, 10} = ii 7 . Consequently, the erosion detects both ii and IV along with ii 7 .

To address this scenario and prioritize seventh chords, a strategic approach in-Figure 5.13: Tonal graph of Bach's Prelude in C major, BWV 846. volves assigning a reduced weight to edges leading to seventh chords 13 . For instance, we can deduct an arbitrary small amount14 from the weight of each arrow directed towards a seventh chord.

The new chords that we add to the harmony presented in Table 5.1 are shown in Table 5.3.

While we omit the presentation of the new graph due to its similarity to the previous one, the corresponding results are captured in Table 5.4 Notably, further enhancements can be made to address the remaining errors. Two notable avenues for improvement are evident: first, certain options should be ruled out as possibilities; for example, the last chord cannot be a I in C because there is a B  ; random modulations in various places can be rectified -such as the modulation occurring one measure earlier in measure 12, where we want to enforce the V 7 to resolve into the I.

We initiate the refinement process by implementing the first technique, employing the hit-or-miss transform. Subsequently, we proceed to apply the second technique involving the utilization of weights that are depend on the Roman numerals of the vertices. F: V 7

Table 5.4: Result of the analysis using seventh chords.

By adopting a hit or miss transform instead of an erosion, we gain the ability to not only specify which chromas we desire but also those we wish to exclude. To achieve this, we introduce two harmonies: one for the "contains" condition (equivalent to erosion) and another for the "contained" condition (complemented erosion). For the second harmony, we use scales like the major scale Maj = {0, 2, 4, 5, 7, 9, 11} or a combination of minor scales (natural, harmonic, and melodic). This refined approach yields an accuracy of 27 out of 32 (84.375 %), with the remaining errors attributed to timing discrepancies in modulations.

To address this timing issue, we can weight differently the chains in function of the corresponding Roman numeral. For instance, we can assign lower weights to V 7 -I and 7º -I transitions. By reducing the weight by 0.1 to edges linking these Roman numerals (in major and minor modes), we achieve a success rate of 29 out 32 (90.625 %). The remaining errors can be summarized as follows:

1. In measure 22, we get C  : 7º instead of G: 7º; this is due to the equivalence between these two chords and the fact that there is no G chord before or after.

2. In measure 23, we get iiº 7 instead of C: 7º; this is interesting because whereas we have affirmed the the correct chord is C: 7º, there may be an argument to defend the actual output 15 .

3. In measure 28, we get E: 7º instead of G: 7º; this is the same problem than the first item.

Conclusion

In conclusion, the construction of tonal graphs proves to be remarkably beneficial for the analysis of chroma rolls comprised of chords. The examples we have presented are just a glimpse of the potential; by adjusting harmonies, weights, edges, or even incorporating higher-order weights derived from the graph, we can significantly enhance their accuracy. Notably, future research may involve using machine learning to learn edge weights, offering an avenue for refinement and optimization in the analysis process.

Conclusions and Perspectives

Throughout this thesis, we have demonstrated the utility of mathematical morphology in the analysis and generation of time-frequency representations of music. Our focus has centered on two primary representations: spectrograms and piano rolls.

In the following, we will provide a summary of the main findings, their respective successes, and the contributions and future prospects of our research. Overall, MM proves effective in both time-frequency representations. Nevertheless, the approach we employed for each of them differs significantly. While we adopted a more traditional approach for spectrograms, MM reveals itself as more meaningful and adaptable when applied to piano rolls. Let us delve deeper into each application.

Spectrograms are very useful representations of sound, but they lack an important element: phase information. While this is not problematic in most cases, it does impose constraints on certain applications. For instance, the sum of two spectrograms is not the spectrogram of the sum, which makes linear analysis inappropriate. Furthermore, the spectrogram of the combination of two sounds is not the supremum of the spectrograms, which violates one of the basic assumptions for using MM. This fact challenges the applicability of the MM framework.

In particular, the presence of interferences cannot be effectively addressed by the STN model. When our processing pipeline (see Figure 3.3) is applied to spectrograms with interferences, the output proves inadequate. This raises questions about the suitability of these methods for critical cases. Future research may explore the application of MM to a different class of spectrograms: those derived from the timefrequency-scale transformation. This transformation could mitigate time-frequency uncertainty and resolve interference issues.

Despite these challenges, our current pipeline performs well at detecting both horizontal and vertical lines. Lines in the spectrogram are a crucial component of many musical instruments, and we perform correctly at detecting them in such instruments. Furthermore, the pipeline effectively identifies noise, which is particu-larly important for other types of instruments, especially percussion ones. While the outcomes of this procedure are promising, there is significant room for refinement; the results do not yet capture the purity of the original sounds. This issue could be attributed either to the synthesis model, the detection model, or most likely, a combination of both. Subsequent research may illuminate the way to integrate these two components, resulting in a more coherent and robust model.

Our current pipeline can replicate the sound of a musical instrument but does not identify the specific instrument or the number of instruments present. This task, called polyphonic multi-pitch estimation, exceeds our current capabilities and was explored at the outset of the doctoral research without success.

However, we believe that the geometry of spectrograms remains largely an uncharted territory. The presence of holes in noisy spectrograms, complementary to lines, may be well adapted to MM operators. Given our success in detecting holes and lines, a model that generates sounds while adhering to these constraints might prove more effective. We propose that future research could explore an approach based on what we might call ridges and sinks model, that would be more adapted to MM.

Transitioning to the realm of piano rolls, MM emerges as an elegant and seamless approach. Furthermore, it catalyzed the development of a new formalism of greyscale MM based on residuated triplets.

The introduction of residuated triplets aligns effectively with the demands of MM. It underscores a critical distinction between the interpretation of inputs in erosion/dilation operations and their corresponding outputs. In standard MM, both are interpreted as images, but with residuated triplets, the lattices hold distinct meanings, influencing the interpretation. Our model typically interprets the input of a dilation (or the output of an erosion) as an activation, while the input of an erosion (or the output of a dilation) represents the object itself, in our case, piano rolls.

Also important is the relationship between the space and the group that acts on it, especially in the context of music. An input piano roll represents a musical piece, whereas a structuring element represents a motif. The group elements encompass rhythms in the time dimension and Roman numerals in the frequency dimension, which are combined to generate motives. This framework enables us to understand a musical piece as a composition of building blocks (the motives) activated through the dilation operator.

While mathematical morphology has been at the forefront of our discussion, we must acknowledge the utilization of other tools in our research, notably tensor products and graphs. Let us delve into their roles and significance.

Tensor product, a versatile mathematical construct, finds application across numerous domains. In our case, it served as a means to encapsulate the concept of a chord distributed rhythmically over time. Through the amalgamation of diverse rhythms and chords, i.e., texture and harmony, we crafted intricate musical motifs and elucidated entire compositions.

We posit that the use of texture and harmony as compositional tools bears significant potential, particularly in the realm of education. The amalgamation of these parameters can serve as a fertile playground for aspiring musicians, offering a novel and meaningful approach to music creation.

While we proposed a XML approach to create music, it is not intended to be a way of writing music. Instead, we envision the development of a user-friendly interface that streamlines the process and abstracts the XML from the user. Such an interface could potentially serve as an alternative to traditional score editors and sequencers, eliminating the need for users to know staff notation. Furthermore, it could incorporate suggestions to enhance the intuitiveness of music creation. An implementation of this interface in OpenMusic [START_REF] Agón | OpenMusic : Un langage visuel pour la composition musicale assistée par ordinateur[END_REF] is ongoing.

Expanding upon the concept of texture and harmony, their integration could transcend mere composition; it may extend into the realm of Artificial Intelligence (AI) applied to music generation and analysis. An AI agent, equipped to comprehend musical pieces or segments using these factors, may find alignment with common neural networks, given the inherent array-like nature of these musical objects. Additionally, genetic algorithms, capable of manipulating parameters individually to produce incremental changes akin to musical evolution, could play a role. We ardently believe that this approach represents a significant stride in advancing the understanding of music by machines.

Another notable facet of our research is the application of graph theory, particularly the extensive use of chains of bipartite graphs. These graphs represent the concept of sequential decision-making, mirroring the process of musical piece analysis in a meaningful manner.

They perform particularly well for Roman numeral analysis. However, there are still many unexplored possibilities. A promising avenue for future research lies in determining the correct weights for the different edges using machine learning techniques with annotated corpora. Additionally, the utilization of derived graphs may prove beneficial in transitioning from sequences of two chords to more complex chord sequences.

The use of such graphs for the extraction of a minimal set of activations has shown

B.2. Common operators

The scaling operator may be defined differently depending on what norm you want to preserve. When the codomain is C, we can define the conjugate operator.

Definition B.8 (Conjugate operator). Let D be a set. Then, the conjugate operator is defined as

• : C D → C D f → f : D → C x → f (x) . (B.10)
If the domain is a group, we can define the involution operator.

Definition B.9. Let (G, +) be a group. The involution operator is defined as

• * : C G → C G f → f * : G → C x → f (-x) . (B.11)
When the domain is R and the codomain is C, we can define the modulation operator.

Definition B.10 (Modulation operator). Let ξ ∈ R. Then, the modulation operator is defined as .12)

M ξ : C R → C R f → M ξ [f ] : R → C t → f (t) • e 2πıtξ . ( B

Figure 1 :

 1 Figure 1: Time-frequency representations of Bach's Toccata and Fugue in D minor, BWV 565 bars 1-5.

Figure 2 :

 2 Figure 2: Spectrograms of music instruments.

  1. If ∀a, b ∈ L, ∃c ∈ L : c = a ∨ b then (L, ≤) is an upper semilattice. 2. If ∀a, b ∈ L, ∃c ∈ L : c = a ∧ b then (L, ≤) is a lower semilattice.

  Example of a translation action on Z 12 ; p = 11 and x = 3.

Figure

  Figure 1.1: Examples of translation actions.

FigureFigure 1

 1 Figure 1.2: Translation of sets.

Figure 1 . 4 :

 14 Figure 1.4: Illustration of binary dilation and erosion in R 2 .

Figure 1 .

 1 Figure 1.5 provides an intuitive interpretation of opening and closing operations.During the opening, the little circle outside the shape from Figure1.4 has been filtered out, while during the closing, the little hole inside the shape has been filled in. This occurs because the circle and the hole are smaller (in terms of inclusion) than the structuring element. Despite these changes, the main shape of the object

Figure 1 . 5 :

 15 Figure 1.5: Illustration of binary opening and closing of the shape from Figure 1.4.

Figure 1 . 6 :

 16 Figure 1.6: Dilations of an image for different structuring elements sizes k.

Figure 1 . 7 :

 17 Figure 1.7: Erosions of an image for different structuring elements of size k.

Figure 1 . 8 :

 18 Figure 1.8: Morphological filters with a structuring element of size k = 7.

Definition 1 .

 1 31 (Hit-or-miss transform). Let A ⊆ E and C, D ⊆ G with C ∩D = ∅. Then, the hit-or-miss transform of A by (C, D), denoted by A ⊛ (C, D), is given by

Definition 1 .

 1 33 (Thinning). Let C, D ⊆ G such that 0 ∈ C and C ∩ D = ∅. Let Ψ be either a set (Ψ ⊆ E) or a function (Ψ ∈ T E ). Then, the elementary thinning of Ψ by (C, D), denoted by Ψ (C, D), is given by:

  supremum(a: Level, b: Level) -> Level infimum(a: Level, b: Level) -> Level __mul__(self, other: Lattice) -> Lattice __truediv__(self, other: Lattice) -> Lattice

1:

  function dilation(image: Image, str_el: StructuringElement) 2: lattice = image.lattice * str_el.lattice 3: output = Image(numpy.empty_like(image.array), image.space, lattice) Algorithm for erosion 1: function erosion(image: Image, str_el: StructuringElement) 2: lattice = image.lattice / str_el.lattice 3: output = Image(numpy.empty_like(image.array), image.space, lattice) for shift in str_el.group do 7: tmp ← image[point + shift] -str_el[shift]

  All the computations are made on a PC with: OS: Ubuntu 20.04.3 LTS CPU: Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz GPU: Nvidia GeForce GTX 1050 MobileParameters:Input size: (1000, 1000) Structuring element size:[START_REF]the complex conjugate of z ∈ C, i.e., if z = a + bı then z = abı 16. Z n : the set of integer numbers modulo n[END_REF][START_REF]the complex conjugate of z ∈ C, i.e., if z = a + bı then z = abı 16. Z n : the set of integer numbers modulo n[END_REF]

  All the computations are made on a PC with: OS: Ubuntu 20.04.3 LTS CPU: Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz GPU: Nvidia GeForce GTX 1050 Mobile Parameters: Input size: (1000, 1000) Structuring element size: (15, 15) (b) Features

Figure 1 . 9 :

 19 Figure 1.9: nnMorpho compared to other Python libraries.

Figure 2 . 1 :

 21 Figure 2.1: Beginning of the Violin Sonata No.1 in G minor, BWV 1001 from Johann Sebastian Bach.

Definition 2. 7 (

 7 Inverse short-time Fourier transform). Let g, γ ∈ L 2 (R; C) such that g, γ = 0. Then, ∀S ∈ L 2 (R × R; C) the inverse short-time Fourier transform of S, denoted by iSTFT[S]

  .3 illustrates a spectrogram generated by TFST with a logarithmic scale for both frequency and amplitude. Three different scale values are shown: N l = 1024, 4096, 46384. This depiction clearly demonstrates that the TFST effectively addresses the time-frequency uncertainty by utilizing various window sizes, enabling more precise representations for different segments of the signal.

Figure 2

 2 Figure 2.2: Spectrograms of the first notes of Chopin's Nocturne nº2, Op. 9.

Figure 2 . 3 :

 23 Figure 2.3: Spectrogram of TFST for three window sizes (1024, 4096, 16384) of a piano A4.

  Piano roll from a MIDI file with time expressed in ticks.

  Piano roll from a MIDI file with time expressed in seconds.

Figure 2 . 4 :

 24 Figure 2.4: First bars of the first movement of Beethoven's Piano Sonata No.17, Op.31 No.2 in different representations.

  or a = • and b ∈ D 128 , or a ≤ 128 b with a, b ∈ D 128 .

Figure 2 . 5 :

 25 Figure 2.5: Piano roll representation of mm. 20-22 of Mozart's Eine kleine Nachtmusik, K.525.

Figure 2 . 6 :

 26 Figure 2.6: Difference between the score of mm. 17-23 from Brahms' Romanze, Klavierstücke, Op.118, Nº5 and the score with only the kept features.

Figure 2 . 7

 27 shows a representation of the first bars of Beethoven's Pathetic sonata in both representations.

Figure 2 . 7 :

 27 Figure 2.7: Piano roll representation of mm. 1-2 of Beethoven's Piano Sonata No.8, Op.13.

  .8 depicts the first bars of Chopin's Nocturne Op.48 Nº1 in both piano roll representation (Figure 2.8b) and chroma roll representation. The chroma roll is presented in both the flat representation (Figure 2.8c) and the cylindrical representation (Figure 2.8d). Notably, C and B are very close in the cylindrical representation (as expected) and appear far apart in the flat representation due to the limitations of plotting on two dimensions.

Figure 2 . 8 :

 28 Figure 2.8: Piano and chroma roll representations of mm. 1-4 of Chopin's Nocturne Op.48 Nº1. 77

  Figure 3.1: Pipeline for the synthesis of signals.

  Figure 3.2: Rotation of the time-frequency plane by F .

  (a) the number of sines I, (b) the amplitudes {a s i [m]} M -1 m=0 of each sine, (c) the frequencies {ξ i [m]} M -1 m=0 of each sine. 2. For the filtered noise: (a) the mask {Θ[m, k]} M -1, K-1 m=0, k=0 . 3. For the transients: (a) the number of transients P , (b) the amplitudes {a h p [k]} K-1 k=0 of each transient, (c) the times {t p [k]} K-1

Figure

  Figure 3.3: Pipeline for the morphological processing.

Figure 3 . 5 :

 35 Figure 3.5: Noisy part of a spectrogram and its reconstruction by erosion.

Figure 3 . 6 :

 36 Figure 3.6: Erosion of the reconstruction by erosion for avoiding temporal leakage.

Figure 3 . 7 :

 37 Figure 3.7: Opening of the reconstruction by erosion.

  Opening after the reconstruction by erosion

Figure 3 . 8 :

 38 Figure 3.8: The result of applying the morphological processing to white noise with σ = 30.

Figure 3 . 9 :

 39 Figure 3.9: Temporal leakage.

Figure 3 . 10 :

 310 Figure 3.10: Result of extracting the stochastic component of the signal.

Figure 3 . 11 :

 311 Figure 3.11: Vertical thinning transforming the ridges into one-pixel-thick lines.

Figure 3 . 12 :

 312 Figure 3.12: Vertical top-hat for recovering the horizontal lines.

Figure 3 . 13 :

 313 Figure 3.13: Threshold of 5 dB of the vertical top-hat.

Figure 3 .

 3 Figure 3.14: Removal of the small lines with a thinning followed by a reconstruction by dilation.

Figure 3 . 15 :

 315 Figure 3.15: Lines and filtered lines recovered in the process superimposed to the input spectrogram.

  .

Figure 3 .

 3 Figure 3.16: Morphological steps for recovering the transient lines. Once we recover the lines, we use the method exposed in Section 3.1.3 to generate a transient. The comparison between the input and the spectrogram of the generated

Figure 3 . 18 :

 318 Figure 3.18: Comparison between the spectrograms of the input and the generated transient.

  Spectrogram of the generated sinusoids (f) Lines and filtered lines superimposed to the input spectrogram.

Figure

  Figure 3.19: Marimba.

  Figure 3.20: Violin.

  Figure 3.21: Violin vibrato.

  Figure 3.22: Gong.

  Figure 3.23: Piano.

Figure 4 . 1 :

 41 Figure 4.1: Representation of the rhythm |  by the function from Equation (4.4).

Examples 4. 10 .

 10 Let us plot the chords from Examples 4.8.

  Figure 4.2: Illustration of the harmonies from Examples 4.13.

Example 4. 16 .

 16 Consider the left hand of the first four measures of the Mozart's Piano Sonata No. 8 in A minor, K. 310 / 300d (see Figure 4.7).

Figure 4 . 4 :

 44 Figure 4.4: Model of the left hand of the first measure of Mozart's Piano Sonata No. 8 in A minor, K. 310 / 300d through the rhythmed chord R 0 ⊗ Am. Remark 4.18. Notice that we made an abuse of notation when writing T ⊗H; indeed,

10 )

 10 Example 4.19. Let us consider the left hand of the measure 41 from the first movement ofBeethoven's Piano Sonata No.17, Op.31 No.2.

Figure 4 .

 4 Figure 4.5 shows how to render this harmonic texture through matrix multiplication. Let us explore some examples of harmonic textures. Example 4.20. We call the Alberti bass with tatum the texture

Figure 4 . 5 :

 45 Figure 4.5: Harmonic texture as a matrix multiplication representing the left hand of the measure 41 from the first movement of Beethoven's Piano Sonata No.17, Op.31 No.2.

4. 1 .Figure 4 . 6 :

 146 Figure 4.6: Mozart's Piano Sonata No.16 in C major, K.545 mm.1-4. This construction is extremely flexible and allows us to understand through the same texture chords of different sizes. Let us show that by examining the left hand of the first four measures of the Mozart's Sonata No.16 in A minor, K. 310 / 300d (Figure 4.7). Example 4.21. Let R be the rhythm of the Example 4.16 and let C 0 and C 1 be the chords C 0 = {A3, C4, E4} and C 1 = {A3, B3, D4, E4}. Then, the first four measures of the Mozart's Sonata No.16 in A minor, K. 310 / 300d are represented by

Figure 4 . 7 :

 47 Figure 4.7: Mozart's Sonata No.16 in A minor, K. 310 / 300d, mm.1-4.

7 i=0( 8 

 78 t i + T ) ⊗ H i with t i = (1, 1, 0) + 3i 8 ∈ T 12 and t + T = (t + R i ) i∈I ∈ A T12 8

Definition 4 .

 4 Figure 4.8: Lacrimosa of Mozart's Requiem in D minor, K.626, mm.1-4.

Example 4. 25 .Figure 4 . 9 :

 2549 Figure 4.9: Textures for chord textures represented by arrays.

  Figure 4.10: Incipit of Chopin's Nocturnes 1 and 2, Op. 9.

Figure 4 . 11 :

 411 Figure 4.11: Mozart's Sonata Nº11, 3 rd movement, Alla Turca -Allegretto, mm. 0-4

Figure 4 . 12 :

 412 Figure 4.12: Beethoven's Piano Sonata No.17, Op.31 No.2 mm.0-8. Let us define the textures

Figure 4 . 13 :

 413 Figure 4.13: Piano roll generated by a dilation representing the measures 0 to 8 of the 3 rd movement of Beethoven's of Piano Sonata No.17, Op.31 No.2.

  ), D5 } A 4 = { (8, 1, 0), D5 } A 5 = { (1, 1, 0), D3 , (2, 1, 0), D3 , (3, 1, 0), D3 , (8, 1, 0), D3 } A 6 = { (4, 1, 0), D3 , (5, 1, 0), D3 , (6, 1, 0), D3 , (7, 1, 0), D3 } which belong to A T

Figure 4 . 15 :

 415 Figure 4.15: Formal structure of Mozart's Piano Sonata No.16 in C major, K.545.

Figure 4 .

 4 Figure 4.16: Score tree of A 2

Figure 5 .

 5 Figure 5.1: Third movement of Beethoven's Piano Sonata No.14, Op.27 No.2 mm. 1-2.

  Morphology Applied to Analyze Piano RollsThe resulting A j = P ⊖ B j are P • B = (A 1 ⊕ B 1 ) ∨ (A 2 ⊕ B 1 ) = P .

  µ × and µ • are the measures obtained by setting (c × , c • ) to (1, 0) and (0, 1), respectively. Now, let us choose B = (B j )

Figure 5 . 2 :

 52 Figure 5.2: Third movement of Beethoven's Piano Sonata No.14, Op.27 No.2 mm. 53-56.

Figure 5 . 3 :

 53 Figure 5.3: Graph of inclusion for subsets of X with |X| = 3.

Figure 5 . 4 :

 54 Figure 5.4: Graph of the subsets of X with 2 |X| -1 arrows, where |X| = 3.

Figure 5 . 5 :

 55 Figure 5.5: Third movement of Beethoven's Piano Sonata No.14, Op.27 No.2 measure 54. The resulting activations A P T = (A i ) i∈I , I = {1, 2, 3} are

  the path S -B -D -A -A -E and calculate its weight, we get 3, which represents the number of different letters (A, B, and D) encountered along the path. However, if we choose the path S -A -C -A -A -E, we still get a weight of 3, even though there are only 2 distinct letters (A and C) in this case.

Figure 5 . 8 :

 58 Figure 5.8: Roman numeral analysis of the third movement of Beethoven's Piano Sonata No.14, Op.27 No.2 mm. 53-56.

Figure 5 . 9 :

 59 Figure 5.9: Input activations A ∈ A T ×N 2

Figure 5 . 10 :

 510 Figure 5.10: Input activations A ∈ A T ×N 12 2 up to the octave representing the excerpt from Figure 5.2.

Definition 5 . 9 (

 59 Tonal graph). LetA ∈ A T ×N 12 2

  : i, G  : iv, G  : i, G  : V, G  : i, G  : N, G  : i, G  : V) ,which gives us the expected solution.

Figure 5 . 11 :

 511 Figure 5.11: Roman numeral analysis of Bach's Prelude in C major, BWV 846.

Figure 5 . 12 :

 512 Figure 5.12: Chroma roll of Bach's Prelude in C major, BWV 846 with chords played at once.

Definition B. 6 7 .

 67 (p-unitary scaling operator). Let s ∈ K \ {0}. Let p ∈ [1, ∞]. We define the p-unitary scaling operator of scale s asD p s : C R → C R f → D p s [f ] : R → C x → |s|-The p-unitary scaling operator is unitary for the p norm, i.e., ||D p s [f ]|| p = ||f || p . (B.9)

  Structuring Elements 1. (R, +) is an additive group. It will represent the continuous translations. 2. (Q, +) is an additive group. It will represent the fractional translations. 3. (Z, +) is an additive group. It will represent the integer translations. 4. (Z 12 , +) is an additive torsion group. It will represent the integer translations modulo 12. 5. Let (G 1 , + 1 ) and (G 2 , + 2 ) be two additive groups. Then, (G 1 × G 2 , +) is an additive group, where the sum is defined by

Table 1

 1

	.1.

  1. Mathematical MorphologyImage corresponds to the input18 . The accessing and assignment operators rely on Point and Level, elements of its attributes space and lattice.

	Image: array: numpy.ndarray space: Space lattice: Lattice __getitem__(self, point: Point) -> Level __setitem__(self, point: Point, value: Level)
	StructuringElement: array: numpy.ndarray group: Group lattice: Lattice

__getitem__(self, shift: Shift) -> Level StructuringElement corresponds to the structuring element

19

. The accessing operator rely on Shift and Level, elements of the group and lattice attributes.

  1.3.4, with the dynamics lattice being D 128 (as presented in Section 2.1.3.3). There are two choices: A P D 128 or A D 128 . We recall the definitions, exposed in Equations (2.12) and (2.15), of the lattices

(A P D 128 , ≤ P 128 ) and (A D 128 , ≤ 128 ):

  oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe ˙˙oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe ™ oe oe ™ oe ae ae ˙ae ae ˙ae ae oe ae ae ˙ae ae oe ae ae ˙ae ae ˙ae ae ˙ae ae oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe ae ae ˙ae ae ˙ae ae ˙ae ae oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe w w

	Vln. 1 Vln. 2 Vla. Bass Vln. 1 Vln. 2 Vla.	sf sf sf ae ae oe # # oe p sf p sf p p sf p (p) f (p) f (p) f (p) f B # 20 24 ? # & # & # B # ? # & # & # # B # oe oe ˙˙oe oe oe ˙ae ae oe ae ae oe ae ae oe ae ae oe ae ae oe oe™ oe J oe # ™ oe oe ™ oe oe™ oe J J oe oe oe oe oe oe oe oe J oe oe J oe oe oe oe oe oe oe oe oe oe # oe oe oe oe oe oe oe oe # oe oe oe oe OE p oe J oe oe J oe oe oe oe oe oe oe oe J oe oe J oe oe oe oe oe oe oe oe J oe ae ae oe oe ae ae oe oe ae ae oe oe ae ae oe oe oe
	Bass	? #

# oe oe oe oe OE oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe # oe oe oe oe oe oe oe oe oe oe oe oe oe oe OE oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe oe # oe oe oe oe oe oe oe oe oe oe oe oe oe OE

  Table 3.1. The parameters T , N , and M depend on the input signal and are not explicitly listed in the table.

	Parameter ξ s T s g J j 0	Value 44 100 Hz 2.2 × 10 -5 s Blackman window 2048 1024	Parameter Value H 44 K 4096 T p 1 ms ξ p 10 Hz T o 46 ms
	Table 3.1: Choice of the parameters for the computations.

  .1.

	RN Chord I {0, 4, 7} IV {0, 5, 9} RN Chord i {0, 3, 7} iv {0, 5, 8} ii {2, 5, 9} V {2, 7, 11} viiº {2, 5, 11} RN Chord vi {0, 4, 9} VI {0, 3, 8} iiº {2, 5, 8} V 45 {0, 2, 7} N {1, 5, 8}

Table 5 .

 5 1: Harmony that we choose for analyzing common tonal music.

Table 5 .

 5 2: Result of the analysis of Bach's Prelude in C major, BWV 846 by means of the tonal graph.

  . In this updated analysis, we have achieved 24 accurate chord identifications out of 32, resulting in a

	RN I 7 {0, 4, 7, 11} IV 7 Chord RN i 7 {0, 3, 7, 10} iv 7 ii 7 {2, 5, 9, 0} V 7 {2, 7, 11, 5} viiº 7 {2, 5, 11, 9} Chord RN Chord {0, 5, 9, 4} vi {0, 4, 9, 7} {0, 5, 8, 3} VI 7 {0, 3, 8, 7} iiº 7 {2, 5, 8, 0} V 457 {0, 2, 7, 5} 7º {0, 2, 7, 8}

Table 5 .

 5 3: Seventh chords extending the harmony presented in Table5.1. 75 %accuracy rate.

We follow ISO standards (see[START_REF] Iso | Quantities and units -part 2: Mathematics[END_REF] vii

This notion already underlies in previous works, but we will make it explicit and central.

It is important to note that supremum and infimum are associative and commutative.

The domain is E if the lattice is either (P(E), ⊆) or (T E , ).

When using the additive notation for a group we assume that this group is commutative.

We use the set 2N + 1 = {2n + 1 ∈ N : n ∈ N} for referring the odd numbers. Whereas k may be even, we limit the structuring element sizes to odd numbers in order to get it centered at the origin.

We use the additive notation for the residuated lattice since it is closer to our applications. The correspondence is the following: we use + in the place of • , -in the place of / and 0 in the place of 1.

As far as we know, this is the only limitation of residuation when formalizing addition and subtraction in morphological operators.

This structuring element determines the notion of connectivity.

To emphasize the role of each operand, we have permuted the order of the operands.

We use the notations of Python for the methods that override operators.

For instance, we might use the constructor in the iteration.

Notice that whereas the operators have arguments and output that are Levels, they may not be the same Level type, allowing for residuated triplets.

We use the term image but another (overused) term would be function.

We use the term structuring element as is more frequent in the literature than structuring function.

A whole is a note value in music, represented by , that can be associated to the value 1 ∈ Q.

If the start of the piece is called t 0 ∈ T s , the bijection is T s → R, t → e(t 0 , t) ∈ R, where e(t 0 , t) is the time elapsed between t 0 and t measured in seconds.

In our computations, we limit the representation to millisecond precision for practical reasons. However, it is important to note that, in theory, we could refine the time precision as much as needed since the space is bijective with R.

Note the overload of the term beat, referring to both a timestamp and a time shift.

Notice that we assume the time space is defined by a single time signature. This may not be the case in a musical piece, but we left more complex cases for future reasearch.

See Section 2.1.2.2 for the notation used for notes and chords.

The first chord, Gm, has a single time associated since it is played at once.

We use the term chroma to refer to what is also called a pitch class in music theory. While the latter may be more mathematically accurate (as it refers to an equivalence class of pitches), the term chroma emphasizes the circularity of the pitch space, as proposed by[START_REF] Shepard | Circularity in judgments of relative pitch[END_REF]

Given by the midi number, exposed in Section 2.3.2.2.

See Definition 1.24 for the definitions of these concepts.

Notice that the window function cannot be a discrete function, as it needs to be sampled differently depending on the desired window size.

The quality factor is supposed to be equal for each band, but the rounding of N k introduces a slight variation that is expressed by the ≈ symbol.

Up to a phase factor, as in the continuous case (exposed in Proposition 2.14).

Complex numbers can be endowed with a partial order, for instance ∀z, ω ∈ C, z ω ⇔ |z| ≤ |ω|), but it does not fit our purpose.

The requirement of F to be countable ensures that pitches are essentially discrete, as opposed to frequencies, which can vary continuously.

The term "contraction" is borrowed from the terminology of tensors, where the contraction of one index involves summing over the values of this index. In the context of piano roll stacks, it signifies merging multiple piano rolls into a single one, taking the supremum of all the indexed elements.

We do not consider the possibility of changing the time signature, even if this is a common practice, to remain simple.

The sampling period T s is the inverse of the sampling frequency ξ s , i.e., T s = 1 ξs .

To further avoid artifacts, we actually apply a filter to this window with a Hann kernel of 5 ms.

It is worth noting that σ is dimensionless, as the amplitude range is also dimensionless.

Since we shall make a choice in the order of removal of the points, we choose this order, i.e.,N,

It is very difficult to remove all of them, if not impossible, in part because the border between signal and noise is not always well defined.

In our example, since the time resolution for the array ξ is 0.001 s, the sampling frequency is 1000 Hz and the Nyquist frequency is 500 Hz, which gives a critical frequency of 0.05 × 500 = 25 Hz.

https://theremin.music.uiowa.edu/MIS.html.

It is worth noting that we have modified the order of the inputs of • for consistency with the subsequent notations.

Notice that the duration may be zero, which is allowed for representing percussive rhythms where the duration is not featured.

The requirement for a function ι : T → Q arises from the need to establish an association between an element t 0 ∈ T and an element ι(t 0 ) ∈ Q, thereby enabling the definition of the additive inverse of t 0 , denoted as -t 0 := -ι(t 0 ).

That is, a ∈ Q + is the GCD of p ∈ Q and q ∈ Q if ∃m p , m q ∈ Z such that p = a • m p , q = a • m q ,and a is the greatest rational that satisfies this property.

This notion, whose name was given by[START_REF] Bilmes | Timing is of the essence: Perceptual and computational techniques for representing, learning, and reproducing expressive timing in percussive rhythm[END_REF], has been studied in (Romero-García, Lascabettes, et al., 2022;[START_REF] Romero-García | A model of rhythm transcription as path selection through approximate common divisor graphs[END_REF].

This statement is made without proof, since it seems clear and the proof could be technical and take us away from our subject.

The subtraction between members of F should be understood as the signed distance induced by the isomorphism with Z.

We recall that, as exposed in Section 2.1.1.3, we denote a timestamp inside a score as a triplet referring to bar, beat and offset.

It is not exclusive to chord textures; we can do that also with harmonic textures, but usually it is important to keep the notion of order in the notes when dealing with melodies (and with motives, to a less extent).

In this work, the examples are picked-up from Western tonal music, but the formalism can handle any type of music based on rational divisions of the time and integer divisions of the octave.

The names of the chords are selected based on the quality of the chords (M → major, m → minor, 7 → dominant seventh and min → minor scale) for avoiding long notations, but we shall recall that they are positioned chords.

We assume that ⊗ has priority over ⊕ as in the case of × and +.

The labels Exp., Dev. and Rec. stand for the sections of the sonata form, namely Exposition, Development and Recapitulation respectively.

All the figures that represent piano rolls in this thesis are done with this method.

https://github.com/Manza12/MMM

We recall that ⊗ has priority over ⊕.

The symbol for opening and composition is the same (•) but will be discernible from the context.

The emphasized term might here indicates the possibility that the search might not reach a conclusion if the selection of the B j is not appropriate. Furthermore, even when the search does conclude, there is no guarantee that we obtain the desired elements.

The inverse of the characteristic function.

All the graphs used in this work are in fact digraphs (directed graphs).

The frequency coordinate is shared by the covered element and the activation because of the nature of the dilation by a texture.

Note that K T depends on computational parameters.

The existence of a (single) correct interpretation may be discussed by musicologists, but there are a lot of cases where there is no doubt, as in this example.

Technically, we shall represent the chroma roll with crosses since it represents activations, but we have decided to represent it with black and white for improving clarity.

We omitted the edges (except the shortest path ones) for clarity but they are obvious because it is a chain of complete bipartite graphs.

Actually, we may also play with the order of exploration of the graph to check later the seventh chords, but is not a robust approach in our opinion.

We chose 0.1 for practical computations.

This argument is that it is followed by a V and the iiº -V movement is very present in tonal music.

Acknowledgements

Chapter 4. Mathematical Morphology Applied to Generate Piano Rolls with h n disjoint hit functions with corresponding starts and durations s n and d n , respectively. We call e n = s n + d n the ends.

Without loss of generality, we can order these functions such that n < m ⇒ s n < s m , having additionally that n < m ⇒ e n ≤ s m due to the h n being disjoint (notice that they may be equal).

There is a single degenerated case that we will not address: the case where N = 1 and h 1 ≡ (0, 0). In the other cases, let a ∈ Q + be the greatest common divisor (GCD) in the sense of rationals 4 of the set {s n , e n } N n=1 , which is commonly called in music the tatum 5 .

With a, we can rewrite all the s n , e n as m s n , m e n with s n = m s n • a and e n = m e n • a. Now, let

We give then the following values to each element of the vector R:

with t 0 = (m + m 0 ) • a ∈ Q and t 1 = (m + m 0 + 1) • a ∈ Q.

Let us illustrate this method with an example. (Figure 4.3a). If we consider the 11 chords that appear (Figure 4.3b) and we stack them in order of appearance, we have the harmony given in Figure 4.3c.

Harmonic Texture

Now that we have defined texture and harmony, let us see how we can combine them to create time-frequency elements and create music. We start by defining the combination of a rhythm and a chord. To do that, we draw upon two mathematical tools: first, we identify a chord with its characteristic 4.2. Generating Piano Rolls with Harmonic Textures and Mathematical Morphology The compatibility arises in the context of spaces with groups acting on them; when we have a space E and a group (G, +) acting on it, there is three cases of compatibility:

1. the group with itself, the output being the group, given by the sum

2. the space with the group, the output being the space, given by the sum

3. the group with the space, the output being the space, given by the sum

The only case of incompatibility is between the space and the space, where there is no meaningful 13 interpretation of a sum; what would mean C4 + C4?

The notion of compatibility might also be defined for the lattices; if we have the lattices A A , A B and A C , the compatibility would mean that there is a residuated triplet defined by the lattice multiplication • : A A × A B → A C . The paradigmatic case of that is the lattice multiplication presented in Section 2.1.3.2.

What we think might be the default choice is to use the space-like elements for activations and the group-like elements for harmonic textures. For instance, we might have chosen the patterns 5, 7, 11, 14} which belong to A Z 2 , and thus having harmonic textures belonging to A Q×Z

3

, and the 13 Of course we may artificially define a sum, but we only focus on sums that make musical sense. Morphology Definition 4.27 (Score tree). We call S a score tree of domain

(4. [START_REF]the complex conjugate of z ∈ C, i.e., if z = a + bı then z = abı 16. Z n : the set of integer numbers modulo n[END_REF] with A n ∈ A

3

, either a score tree of domain

The way of transforming a score tree into a piano roll is by using recursively the morphological dilation. Indeed, the piano roll P S associated to the score tree

is given by the formula

where P Bn is either B n if it is a harmonic texture or the piano roll associated to B n if it is a score tree.

In the following, we make the abuse of notation

that consists on identifying a score tree with its resulting piano roll. Let us show an example of such a decomposition of a score into a score tree.

Example 4.28. We call S the score tree representing first movement of Mozart's Piano Sonata No.16 in C major,K.545. We divide its structure in two parts corresponding to the sections enclosed by repeat signs, i.e., Part 1 consists on mm. 1-28 and Part 2 in mm. 29-73. The corresponding score tree description is S = {(0, 0), (28, 0)} ⊕ Part 1 ∨ {(56, 0), (100, 0)} ⊕ Part 2 (4.17)

where we factor out the point-like elements (1, 1, 0) and C4.

We can divide even further the score in the following way Finally, we can define the score tree. We need for that to define also the activations.

×J be the erosion defined in Equation (5.6), with B = (B j ) j∈J ∈ A T ×F ×J . We define A = (A j ) j∈J ∈ A T ×F ×J . Then, the adjoint operator of ε B is the dilation

.

(5.7)

Proof. We know by Theorem 1.10 that since ε B : A T ×F → A T ×F ×J is an erosion there is an adjoint dilation given by the formula

We are proving that j∈J (A j ⊕ B j ) ∈ P and that ∀P ∈ P, j∈J (A j ⊕ B j ) P , which means that j∈J (A j ⊕ B j ) = P and finishes the proof.

j∈J

Chapter 5. Mathematical Morphology Applied to Analyze Piano Rolls

To achieve this, we once again employ the erosion. However, this time we use the erosion of a piano roll by a texture. It is important to note that since

, there is no direct definition for such an erosion.

For the sake of generality, let us abstract away the specific choices for time and frequency and consider P ∈ A T ×F 3 and T ∈ A Q×I

3

. We will employ a similar approach as in the previous section and use the bijections:

where we recall that 0 = {0}.

We have now two compatible domains T × F × 0 and Q × 0 × I. Moreover, we can use the inclusion:

and treat any texture as a stack of piano rolls.

The erosion is now well defined and has the explicit expression

Let us illustrate this technique with our example. By calling In this case, we have a huge redundancy 4 . Indeed, using the reference redundan-

4 The redundancy of T over P is defined as the redundancy of T ⊗ 0 over P .

155

Chapter 5. Mathematical Morphology Applied to Analyze Piano Rolls cies, we obtain

The challenge now is to remove the redundant activations of A T such that the redundancy is 0 %.

Extracting a Minimal Set of Activations

We recall that we have a piano roll P ∈ A T ×F 3 and a texture T ∈ A Q×I

3

. Let us consider A ∈ A T ×F×I 2 such that P = A ⊕ T ⊗ 0. Then, by using the properties of mathematical morphology, we have that (5.18) which means that by taking the erosion A P T = P ⊖ T ⊗ 0 we will always have extra activations. We are focusing now on extracting A min A P T such that it is minimal, i.e., ∀A ∈ A T ×F ×I 2 : P = A ⊕ T ⊗ 0, A min A P T .

( 5.19) Since the amplitude range of A P T is the Boolean lattice A 2 , we can use the isomorphism given by the support function 5

supp :

and identify A P T with its support. From now, we make the abuse of notation A P T ⊆ T × F × I. We use then ⊆ instead of for the order.

Analyzing Piano Rolls with Textures

((1, 0), 0) consideration relies not only on the edges between two individual nodes, but also on the nodes selected along the path. Addressing this issue has led to the use of the derived graph.

((1, 0), 0)

(a) Full graph.

(3, 0), 0

(1, 1 8 ), 2

(3, 1 8 ), 0

(3, 0), 1

(2, 0), 2

The Derived Graph of a Graph

The notion of derived graph of a graph was introduced by Harary and Norman, 1960 under the name of line graph, and it is known by various other names such as interchange graph, adjoint, or edge-to-vertex dual [START_REF] Beineke | Characterizations of derived graphs[END_REF]. The idea behind this construction appeared earlier in [START_REF] Whitney | Congruent graphs and the connectivity of graphs[END_REF][START_REF] Krausz | Démonstration nouvelle d'une théoreme de whitney sur les réseaux[END_REF], and it has been extensively studied by researchers like [START_REF] Beineke | Derived graphs and digraphs[END_REF][START_REF] Beineke | Characterizations of derived graphs[END_REF]Beineke and Zamfirescu, 1982;[START_REF] Beineke | Connection digraphs and second-order line digraphs[END_REF], from who we borrow the term derived graph and the notation ∂G, used for instance by [START_REF] Beineke | Derived graphs and digraphs[END_REF][START_REF] Beineke | Characterizations of derived graphs[END_REF].

The definition of the derived graph is as follows.

Definition 5.4 (Derived graph). Let G = (V, E) be a graph where E ⊆ V 2 . Then, the derived graph of G is the graph ∂G

Since the vertex v 2 is shared, we will rather write e ′ ∈ E ′ ⊆ V 3 , e ′ = (v 1 , v 2 , v 3 ).

Notice that we can derive a graph as much as we want, and we notate ∂ k G the k th derived graph of G. The k th derived graph of G can be seen as the graph whose vertices are the paths of length k + 1 of G and the edges represent that two paths share common elements and thus can form a bigger path.

To illustrate this concept, let us consider the following example. .

Then, the derived graph of G, ∂G is represented by

(5, 6) (6, 7) [START_REF]the set of real numbers[END_REF][START_REF]the set of real numbers without the zero[END_REF] .

The 2 nd derived graph of G, ∂ 2 G is represented by

(5, 6, 7) (1, 3, 4) (3,4,[START_REF]Q + : the set of non-negative rational numbers[END_REF] (1, 3, 5)

(3, 5, 6) [START_REF]Q + : the set of non-negative rational numbers[END_REF][START_REF]the set of real numbers[END_REF][START_REF]the set of real numbers without the zero[END_REF] .

The derived graph becomes useful when we want to consider second-order information in a graph. To further explain this concept, let us consider a simple problem

Analyzing Piano Rolls with Textures

In this new graph, if we follow the path SS -SA -AC -CA -AA -AE -EE (which can be simplified to S -A -C -A -A -E), we obtain a weight of 2, which accurately represents the number of distinct letters.

By using this concept, we can iteratively differentiate the graph to consider common letters that are spaced more than two edges apart. The number of times we need to differentiate is equal to the distance (in terms of edges) between the common letters minus one.

Chain of Bipartite Graphs

In this chapter, we use a particular type of graph: chain of bipartite graphs. This concept appears in [START_REF] Sathiamoorthy | Labeling of chain bipartite graphs[END_REF] and can be understood as a chain in the sense of [START_REF] Concas | Chained graphs and some applications[END_REF] of bipartite graphs. We present the definition next.

Definition 5.6 (Chain of bipartite graphs). A graph

such that ∀l ∈ {1, 2, ..., L-1}, all the vertices of V l are connected with all the vertices in V l+1 , and there are no other edges.s Formally, for digraphs,

we say that the digraph is a chain of complete bipartite graphs.

We use the notation G n 1 →n 2 →...→n L for a chain of bipartite graphs and K n 1 →n 2 →...→n L for a chain of complete bipartite graphs, where n l = |V l |, l ∈ {1, 2, ..., K}.

Example 5.7. The chain of complete bipartite graphs K 1→3→2→4→2→1 is represented by

Analyzing Piano Rolls with Textures

elements at the same time, which gives us the order of derivation

(5.24)

In our example, where k F = 4, we have k = 4 • 4 -1 = 15, which is a huge order of derivation. Indeed, if we call

our graph and if we use the formulas from Proposition 5.8, we have

n l-m = 84 633 984

These sizes are too big to perform computations (specially because the result is only a single measure). When we filter the activations with the contraction we obtain the graph

Then, the sizes of the graph and its 15 th derivative are

which is now tractable. Indeed, we obtain as minimal activations

Let us discuss in detail the interest of this construction. Each vertex takes the form of a triplet (t, ξ, i) ∈ T × N 12 × I representing a potential interpretation of the chord as a Roman numeral. Here is how the components of this triplet shall be interpreted:

• the time value t designates the moment when the chord is present within the musical piece,

• the chroma ξ represents the tonic, that is the key to which the Roman numeral is subordinated,

• the index i corresponds to the Roman numeral, characterizing the harmonic label.

For any two successive time points t 1 and t 2 , i.e., A| ]t 1 ,t 2 [ = ∅, all vertices existing at time t 1 are linked to all vertices at time t 2 . This makes the graph a chain of complete bipartite graphs. Moreover, traversing a path within the graph signifies an interpretation of each chord present in the input as a Roman numeral in a key.

Let us illustrate that with an example.

Example 5.10. We consider the activations piano roll A ∈ A T ×N 12 2 from Figure 5.10.

2 is the harmony exposed in Table 5.1. Then, the erosion A ⊖ H is

Conclusions and Perspectives

theoretically promising but practically limited. This challenge remains for the future, possibly necessitating the development of novel tools for resolution. Furthermore, it is worth noting that our assumptions regarding prior knowledge of the texture are not warranted a priori.

The quest for a unified method to simultaneously uncover texture and harmony is, in our opinion, one of the most exhilarating challenges this thesis presents. We view it as an all-in analysis that can provide a comprehensive understanding of a musical piece at various levels: from the hierarchical perspective at a broader level, through the identification of the motives that constitute a piece at a medium level, to the recognition of individual notes as harmonic elements related to a tonic at a lower level. Such an approach could foster a compelling human-machine interaction, with machines proposing possibilities and humans making decisions.

List of publications

Appendix A

Order theory

The foundations of lattice theory is order theory. We recall here some basic concepts. In an abuse of notation, we will say that the symbol ≺ is the binary relation. A binary relation on X is a binary relation between X and X. Definition A.2 (Partial order). Let L be a set. A partial order on L is a binary relation on L that satisfies the three following properties:

We call (L, ) a partially ordered set.

We also recall the definitions of the supremum and infimum.

Definition A.3 (Supremum). Let (L, ) be a partially ordered set. Let A ⊆ L. We say that a 0 ∈ L is the supremum of A, and we notate a 0 = A, if a 0 is the least upper bound of A, i. We use the notation a ∧ b for {a, b}.

A direct consequence of the definitions that is used extensively is the following.

Proposition A.5. Let (L, ) be a partially ordered set.

Functional Analysis

Functional Analysis is the study of spaces made of functions and the operators between them. Let us recall the classical functional spaces.

B.1 Common functional spaces

Let D and C be two sets. We recall that we call C D the space of functions f : D → C that have D as domain and C as codomain.

All the L p (R; C) spaces are Banach spaces, i.e., complete normed vector spaces. Furthermore, the space L 2 is a Hilbert space with scalar product defined in Definition B.13.

B.2 Common operators

If we have a subtraction operator defined in the domain, we can define the translation operator.

Definition B.2 (Translation operator). Let C be a set and let E and G be two sets such that a subtraction -: E × G → E is defined. Let a ∈ G. Then, the translation operator is defined as

3) Definition B.3 (Reflection operator). Let C be a set and (G, +) be a group. Then, the reflection operator is defined as

When the set C has a ring structure, we can define the addition and multiplication operators. The multiplication operator is defined as

.

(B.6)

When we have a field structure in the domain, we can define the scaling operator.

B.3 Fourier theory

In this section we will recall the basics of Fourier transformations, in particular to establish the notations used throughout this thesis. The Fourier Transform is defined for functions in L 1 (R; C) as follows.