Claude Jolivalt

Ludovic Neuville

Francois D Boyer

Lucien Kerhoas

Christian Mougin

Identification and formation pathway of laccase-mediated oxidation products formed from hydroxyphenylureas

Keywords: Laccase, white rot fungi, T. versicolor, transformation pathway, hydroxyphenylurea, quinone, polyaromatic oligomers, reaction mechanism

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Phenylurea compounds have become extensively widespread in the environment since their discovery in the 1950s. Most of them are mainly used as herbicides for the selective protection of cereal crops and for the maintenance of roads and railways. Others enter the industrial processes as biocides and/or protective agents against oxidation. Nevertheless, these compounds are persistent contaminants of the environment, including soil and both surface and ground waters [START_REF] Garmouma | Spatial and temporal variations of herbicide (triazines and phenylureas) concentrations in the catchment bassin of the Marne river (France)[END_REF][START_REF] Nitschke | Surface water pollution by herbicides from effluents of wastewater treatment plants[END_REF]. As a consequence, there is an increasing interest in the biotic and abiotic degradation pathways of these chemicals [START_REF] Bolte | Cycle de vie d'un polluant. Cas particulier de pesticides de la famille des phénylurées[END_REF].

Biodegradation of phenylureas in soils is governed by many environmental factors, such as pH, temperature, and the presence of degrading micro-organisms [START_REF] Gaillardon | Changes in concentration of isoproturon and its degradation products in soil and soil solution during incubation at two temperatures[END_REF]. Only one bacterial strain, Arthrobacter sp. N2 [START_REF] Tixier | Biotransformation of phenylurea herbicides by a soil bacterial strain, Arthrobacter sp. N2: structure, ecotoxicity and fate of diuron metabolite with soil fungi[END_REF], was identified as an efficient degrader of diuron [N- [START_REF] Bolte | Cycle de vie d'un polluant. Cas particulier de pesticides de la famille des phénylurées[END_REF][START_REF] Gaillardon | Changes in concentration of isoproturon and its degradation products in soil and soil solution during incubation at two temperatures[END_REF]-dichlorophenyl)-N',N'-dimethylurea] into its corresponding aniline. In soils, fungal strains transform phenylureas to Ndemethylated and/or hydroxylated derivatives [START_REF] Tixier | Fungal biodegradation of a phenylurea herbicide, diuron: structure and toxicity of metabolites[END_REF]. Hydroxylation of the aromatic ring of phenylureas during their metabolization in higher plants has also been reported for monuron [N-(4-chlorophenyl)-N',N'-dimethylurea], which is transformed to N',N'-dimethyl-N-(4-chloro-2-hydroxyphenyl)urea [START_REF] Frear | Monuron metabolism in excised Gossypium hirsutum leaves: aryl hydroxylation and conjugation of 4-chlorophenylurea[END_REF].

Of par-ticular importance, direct and indirect photolysis of substituted phenylureas exposed to sunlight leads to several photoproducts, including those resulting from ring hydroxylation and substitution of a chlorine atom by a hydroxyl group [START_REF] Tanaka | Photolysis of 3-(4-chlorophenyl)-1,1-dimethylurea in dilute aqueous solution[END_REF][START_REF] Aguer | Transformation of fenuron induced by photochemical excitation of humic acids[END_REF][START_REF] Richard | PH effect in the photocatlytic transformation of a phenylurea herbicide[END_REF][START_REF] Jirkovsky | Photolysis of diuron[END_REF][START_REF] Boulkamh | Phototransformation of metoxuron [3-(3-chloro-4-methoxyphenyl)-1,1-dimethylurea] in aqueous solution[END_REF][START_REF] Galichet | Iron(III) photoinduced degradation of isoproturon: correlation between degradation and toxicity[END_REF][START_REF] Richard | Transformation of Monuron photosensitized by soil extracted humic sub-stances: energy or hydrogen transfer?[END_REF]. Functionalization by hydroxylation of the aromatic ring considerably extends the range of metabolic reactions able to further transform the pollutants [START_REF] Tixier | Biotransformation of phenylurea herbicides by a soil bacterial strain, Arthrobacter sp. N2: structure, ecotoxicity and fate of diuron metabolite with soil fungi[END_REF][START_REF] Tixier | Fungal biodegradation of a phenylurea herbicide, diuron: structure and toxicity of metabolites[END_REF][START_REF] Frear | Monuron metabolism in excised Gossypium hirsutum leaves: aryl hydroxylation and conjugation of 4-chlorophenylurea[END_REF].

In this context, white rot fungi, such as T. versicolor, offer many advantages in secreting nonspecific extracellular oxidases. Among these enzymes, laccases (EC 1.10.3.2, p-diphenol: dioxygen oxidoreductases) have been reported to catalyze the transformation of a wide range of xenobiotics, including phenols and chlorinated phenols. They convert these chemicals to radicals, which react to form stable polymers by coupling to each other or with humic substances in soils [START_REF] Park | Dehalogenation as a consequence of binding to humic materials[END_REF][START_REF] Ullah | Reactions of pentachlorophenol with laccase from Coriolus versicolor[END_REF][START_REF] Bollag | Enzymatic transformation of chlorophenol compounds[END_REF]. Such a oxidative coupling process is a key mechanism for the formation of bound residue in the soil and for the precipitation of insoluble oligomers in waters. Because these polymers can be easily removed from water by filtration, such an enzymatic approach can be of particular interest in the decontamination of aqueous effluents [START_REF] Jolivalt | Immobilization of laccase from Trametes versicolor on a modified PVDF microfiltration membrane: characterisation of the grafted support and application in removing a phenylurea pesticide in wastewater[END_REF].

Up to now, our work was mainly focused on the transforma-tion of nonchlorinated derivatives by fungal laccases [START_REF] Jolivalt | Transformation of N',N-4-dimethyl-N-(hydroxyphenyl)ureas by laccase from the white rot fungus Trametes versicolor[END_REF]. In the present study, we included chlorohydroxylated phenylurea compounds, so that the hydroxylated phenylurea derivatives studied in this work are representative of the range of hydroxylated metabolites of some phenylurea herbicides found on the fields.

The potential of fungal laccases in catalyzing the transforma-tion reactions of these pollutants as a function of the pH was investigated. In addition, we determined the chemical structures of the main products formed from nonchlorinated and chlorinated hydroxyphenylureas during the enzymatic reactions, allowing us to propose transformation pathways and reaction mechanisms. Our results better define (i) the fate of phenylureas in the environment and (ii) the potential application of laccases in decontamination efforts.

Materials and methods

Chemicals

Acetonitrile (HPLC grade) was obtained from Carlo Erba (Val de Reuil, France). Chemicals used as buffers and silica for chromatography (Gerudan Si 60, 40-63 µm) were purchased from VWR (Fontenay sous bois, France). Chlorzoxazone (5-chloro-2-benzoxazolone) was from Acros (Noisy le Grand, France). All other chemicals or reagents used for synthesis and assays, including 2,2'azinobis(3-ethylbenzthiazoline-6-sulfonic) acid (ABTS), were obtained from Sigma (Saint Quentin Fallavier, France).

Production of T. versicolor laccase.

The production and purifica-tion of laccase have been described elsewhere [START_REF] Bertrand | Purification and preliminary crystallographic study of Trametes versicolor laccase in its native form[END_REF]. Briefly, the enzyme was produced from Trametes versicolor cultures, induced by 2,5-xylidine in a 5-L bioreactor. The purification included two steps of chromatography, a first one on a DEAE 52 anion exchange column, and a second one on a Phenyl Sepharose (Pharmacia HiTrap) column. The purification procedure led to a homogeneous sample, as checked by electrophoresis, with a specific activity of 300 U/mg, where one unit of enzyme activity (U) is defined as the amount of enzyme that oxidizes 1 µmol of ABTS in 1 min.

Assays for laccase activity were performed by measuring the enzymatic oxidation of ABTS at 420 nm ( = 3.6 10 4 cm -1 M -1 for the oxidation product). The reaction mixture contained 1 mM ABTS in 0.1 M Na2HPO4/citric acid buffer (pH 3.0) in a final volume of 1 mL. The buffer solution was saturated with air by bubbling prior to the experiment. After the enzymatic solution was added, the increase in absorbance was followed during the first 2 min, at 30°C. Enzymatic reactions with hydroxyphenylurea derivatives pH Activity profile of T. versicolor laccase. Determination of the optimum pH was conducted in 50 mM citrate/phosphate buffer in the range of pH 2.5-7. The chlorinated hydroxyphenylureas were solubilized at 0.1 g/L, and 0.05 U of laccase was added in a total volume of 5 mL. Samples were incubated and shaken (150 rpm) at 30°C for 15 min. The reaction was then stopped by adding 0.5 mL of concentrated HCl. Quantitative analysis of phenylurea transformation was conducted using an HPLC system (Varian) with a reverse phase column (NovaPak C8, Waters, 2.3 mm 150 mm). Elution was carried out at a flow rate of 1 mL min -1 using a solvent gradient from 100% water to 100% acetonitrile within 20 min. Detection was performed at 244 nm using a UV detector. Retention times for the hydroxyphenylureas tested ranged from 7 to 11 min. Experiments with 2HF-5Cl were duplicated. Time course of hydroxyphenylurea transformation. Hydroxyphenylureas (1 mM) were shaken at 150 rpm at 30°C for 2 h with a specified amount of enzyme in 10 mL of 25 mM citrate/phosphate buffer at various pH values. Incubations without enzyme served as the control. After defined periods of incubation, the reaction assays were stopped with 10% (v/v) concentrated HCl and filtered (Millipore 0.22 µm) before HPLC analysis.

Incubation with phenylureas and extraction of oxidation products. Hydroxyphenylureas (25-50 mg) were incubated with 5-10 U of laccase in 25-50 mL of aerated citrate/phosphate buffer (50 mM) at 30°C in darkness. The extent of transformation was checked by HPLC, and the incubation stopped when the totality of the initial phenylurea was transformed. Reaction mixtures were acidified to pH 2 by addition of 1 M HCl. The reaction mixture (aqueous phase and precipitate) was treated with sodium chloride (2.5 mol L -1 ) final concentration) and extracted with dichloromethane (25 mL) three times.

The extracts were combined, dried over anhydrous sodium or magnesium sulfate, and evaporated on a vacuum evaporator at 40°C. The residues were dissolved in 5 mL of ethyl acetate and separated on a silica column using ethyl acetate as an elution solvent. Identification of pure compounds isolated from the column was performed by mass and NMR spectroscopy.

Analytical and spectroscopic methods. MS and MSMS spectra were obtained using a triple quadripole instrument Nermag R 30-10 (Quad Service, Poissy, France) in electronic impact (EI) and chemical ionization (CI) modes. A Quattro LC (Micromass, Manchester, U.K.) equipped with an electrospray source ionization (ESI) (Z-spray from Micromass) was used for LC-MS determinations. Nermag R 30-10 source conditions were set as follows: temperature, 130°C; electron energy, 70 eV (EI) or 95 eV (CI). Samples were introduced either by gas chromatography or with a direct insertion probe for ionization by EI or by desorption chemical ionization (DCI). NH3 was used as reagent gas at 10 -4 Torr pressure in the source housing, and for MS-MS experiments argon was used as collision gas (4 x 10 -2 Torr).

The Quattro LC electrospray source parameters were as follows: capillary, 3.25 kV; extractor, 2 V; source block temperature, 120°C; desolvation gas, 500 L/h N2; temperature, 400°C. The sampling cone voltage was varied usually from 20 to 40 V. Data acquisition and processing were carried out using the software MassLynx version 4.0. The compounds were introduced by infusion (Harvard Apparatus, Holliston. MA). shifts are reported in δ ppm relative to CHCl3 (CDCl3) as internal reference: 7.27 ppm for 1 H (77.14 ppm for 13 C) or measured with SiMe4 as internal reference following standard techniques. For the other cases, residual solvent was also used as internal standards [START_REF] Gottlieb | NMR chemical shifts of common laboratory solvents as trace impurities[END_REF]. Coupling constants (J) are given in hertz (Hz). Multiplicities are recorded as s (singlet), d (doublet), t (triplet), q (quadruplet), m (multiplet), and br (broad).

Results

Influence of pH on the transformation of chlorinated hydroxyphenylureas by laccase pH-curves (Figure 1) exhibit the same bell shaped profile for all the chlorinated hydroxyphenylureas with a maximum between pH 4 and 5. A slight difference was observed for 2HF-5Cl, whose pH-curve was found to be broader. In the case of 4HF-3Cl and 2HF-4Cl, laccase showed only residual activity at pH 7, while, with 2HF-5Cl, the enzyme retained up to 15% of its maximal activity.

Time dependence of the transformation of 4HF-3Cl by laccase

The analytical method used to follow the transforma-tion of the chlorinated hydroxyphenylurea derivatives as a function of the pH showed that pH influences the nature of the reaction products. A similar behavior was observed for all the tested compounds. Consequently, transformation of 4HF-3Cl was chosen here as a typical example. The time course of 4HF-3Cl transformation was carried out in the presence of laccase at two pH values: 3 and 5 (Figure 2). At pH 3 (Figure 2A), the enzymatic transformation of 4HF-3Cl led to the formation of one predominant product, I, whose retention time (tR = 3.4 min) was smaller than that of 4HF-3Cl (tR )=7 min), indicating a higher polarity of the product.

At pH 5 (Figure 2B), the transformation of 4HF-3Cl was more rapid than that at pH 3 and led to the formation of two products, I and V (tR = 7.4 min). After 45 min, 97% of the 4HF-3Cl was transformed whereas the concentration of each of the two products reached a maximum at this time. The amount of compound V slightly decreased thereafter, whereas that of product I remained constant. The formation of both products, thus, seems to proceed via a competitive process, since product V did not seem to result from the further transformation of product I.

Transformation of all chlorinated hydroxyphenylureas displayed a similar behavior to that observed with 4HF-3Cl: a quinonic predominant product at pH 3, more polar than its parent compound. When increasing the reaction pH, the quinonic compound concentration decreased, favoring one or more less polar oxidation products.

Identification of products formed from hydroxyphenylureas by laccase

Proposed structures for the transformation products are shown in Tables 123, including their 1 H NMR and mass spectroscopic data.

The 1 H NMR spectrum of compound I showed only one singlet (δ 6.74 ppm), suggesting a symmetric localization of hydrogen atoms on the aromatic ring. The molecular mass was found to be 108.

Comparison (spectroscopic data, retention time) with commercial standards led to its identification as 1,4-benzoquinone (Table 1).

The 1 H NMR spectrum of compound II (Table 1) showed only the presence of three aromatic hydrogen atoms. According to the observed coupling constants and the multiplicity of the peaks, the signal pattern is consistent with the presence of three protons in the 3, 5, and 6 positions on the ring. From mass spectroscopy, we know that the compound retained its chlorine atom during the transformation.

Together with a molecular mass of 142, this information is in agreement with the structure of 2-chloro-1,4-benzoquinone, as confirmed by comparison with the spectroscopic data of the commercial standard.

The 1 H NMR spectrum of compound III (Table 1) indicated the presence of a N(CH3)2 group (δ 3.08 ppm). Chemical ionization mass spectrometry indicated that the parent molecule loses its chlorine atom during the oxidation process, leading to a compound with a molecular mass of 194, which is in accordance with the structure shown in Table 1.

Compound IV was the major product obtained by oxidation of 4HF at pH 5. The NMR spectrum (Table 2) indicated the presence of a N(CH3)2 group and of 2 x 2 aromatic protons. The MS fragmentation pattern (EI spectrum) gave a m/z of 178 with fragments 134 and 106 representative of the successive fragmentations of the substituted urea group on the parent molecule. On the basis of these large similarities with the parent molecule, together with published 1 H NMR data on acetyl p-benzoquinone imine [START_REF] Novak | Hydrolysis of the model carcinogen N-(pivaloyloxy)-4methoxyacetanilide: involvement of N-acetyl-p-benzoquinone imine[END_REF], it was concluded that compound IV was the 1,1-dimethyl-3-(4-oxocyclohexa-2,5dienylidene)-urea resulting from the oxidation of 4HF.

Compound V was obtained from transformation of 4HF-3Cl

From 1 H NMR spectrum analysis (Table 2), it can be assumed that the molecule retained the dimethylated substituted phenylurea motif. The aromatic hydrogen substitution pattern on the 1 H NMR spectrum is very similar to that observed for II. Data from mass spectrometry indicated a molecular mass of 212, i.e., loss of two mass units compared to the parent molecule and the presence of one chlorine atom. The 13 C spectrum showed three CH (140.92, 134.04, 127.79) and two quaternary carbons (179.02 and 161.96), whose chemical shift is very close to that observed for a 2-chloro-1,4quinonimine cycle, confirming the structure of V as (3E)-3-(3-chloro-4-oxocyclohexa-2,5-dienylidene)-1,1-dimethylurea. It was noticed that the 13 C NMR and 1 H NMR spectra contained additional peaks (around 40% of the total intensity) assigned to the corresponding (3Z) isomeric form of the molecule.

The oxidation products formed during the transformation of ortho-hydroxylated phenylureas 2HF and 2HF-4Cl at pH 5 were identified as dimeric species resulting from a carbon-carbon coupling of the parent molecules (Table 2). Mass spectrometry analysis (positive and negative modes) of the 2HF transformation extract indicated a molecular mass of 356 and exhibited fragments at (M -45) and (M -2 x 45) in the negative mode, which is representative of two successive losses of the dimethylated amine fragment. 1H NMR spectrum resulted from the superposition of the signals of two regioisomers, compound VI (more than 80%) and compound VIbis. The 1 H NMR spectrum of the major isomer, VI, was representative of a symmetrical dimer. The signal assignment is consistent with the presence of a dimethylated phenylurea fragment and of three aromatic protons in the 3, 4, and 6 positions. 13 C NMR confirmed the presence of the urea carbonyl (154.71 ppm) and indicated the presence of one aromatic quinone (180.81 ppm), together with two quaternary carbons (138.62, 135.17) and three CH (134.12, 128.16, 111.26) on each aromatic cycle. These results are in agreement with the structures of VI and VIbis proposed in Table 2, with the E conformation being assigned to the major product (compound VI), which is sterically favored.

Because of the poor chromatographic separation of the crude extract after 2HF-4Cl transformation at pH 5, identification was based on the mass spectrometry analysis of fractions collected from HPLC.

The purest fraction contained, together with unreacted parent 2HF-4Cl, one major product compound (VII). Mass spectra indicated a molecular mass of 426 for VII, including two chlorine atoms. Two successive losses of a dimethylated amine in the fragmentation pattern could be attributed to the presence of two urea groups on VII. The 1 H NMR spectrum contained four singlets (Table 2). The signal at δ 7.89 ppm disappeared in the presence of D2O and so can be assigned to NH, by analogy with the chemical shift of the parent molecule 2HF-4Cl (δ 7.79 ppm). Two singlets were assigned to aromatic hydrogen atoms in the para position. To account for the molecular mass, compound VII was thus identified as the dimeric hydroxyl compound, derived from the coupling of the parent phenylurea in the para position, with reference to the hydroxyl group.

At pH 5, oxidation of 2HF-5Cl resulted in the formation of 2,5-benzoquinone II as a final product.

Increasing the pH to pH 6.5 allowed shifting to the formation of three different products, identified as chlorzoxazone and compounds VIII and IX (Table 3). They were recovered from three successive fractions collected from the silica column. In addition to a distinctive Rf, each compound was characterized by a different color. The fraction with the higher Rf (0.8, ethyl acetate), chlozoxazone, was yellow, the intermediate fraction (Rf 0.61, ethyl acetate, product VIII) was orange, and the most retained fraction (Rf 0.32, ethyl acetate, product IX) was pink. Chlorzoxazone was identified by comparison with an authentic sample (spectroscopic data and retention time). From MS data, the molecular mass of product VIII was 406, which corresponds to a dimeric oligomer derived from 2HF-5Cl, and the molecule contains one chlorine atom. 13 C NMR indicated the presence of 12 aromatic carbons in compound VIII, which confirmed the oligomerization extent. 13C NMR showed typical signals for quinones in the range of 180 ppm. Compound VIII contained two carbonyls from a urea group in the range of 150 ppm, showing that each aromatic cycle kept its urea moiety. The 1 H NMR signal at δ 8.11 ppm disappeared in the presence of D2O and was thus assigned to NH protons. On the basis of the assignment of the two 1 H NMR singlet protons in 2-acetamido-5-ethoxy-1,4-benzoquinone [START_REF] Calder | N-hydroxylation of p-acetophenetidide as a factor in nephrotoxicity[END_REF], it was hypothesized that VIII could contain two hydrogens in the para position on a disubstituted 1,4-benzoquinone (δ 8.01 and 6.02 ppm in C6D6). The 1 H NMR multiplets and coupling constants of the aromatic protons of VIII are representative of a 1, 2, 5 aromatic substitution pattern. A NOE experiment showed correlation between H3 and H3' protons, which is consistent with assigning H3, H4, H6 on one hand and H3', H6' on the other hand to two different aromatic cycles.

From MS data, IX has a mass of 584, corresponding to a trimer of its parent molecule. It contains only one chlorine atom. Two successive losses of 45 units followed by loss of fragments of 44 units indicate that each of the aromatic cycles of IX bears the urea group of the parent molecule 2HF-5Cl, which is confirmed by three 13 C NMR signals in the range of urea carbonyl (155.85, 154.09, 153.25 ppm) as well as the three singlets (8.38, 8.19, 8.02 ppm), corresponding to NH in the 1 H NMR spectrum (Table 3). The 13 C NMR spectrum also showed the presence of 18 aromatic carbons, including two quinonic carbons (182.47, 181.08 ppm). Both 1 H NMR singlets (6.94 and 5.97 ppm in CDCl3) are supposed to be representative of two aromatic protons in the para position on a disubstituted 1,4-benzoquinone, as encountered in compound VIII. The six remaining 1 H NMR signals can be attributed to the three protons of each of the two aromatic cycles in the 3, 4, 6 positions on each cycle. The highest chemical shifts were assigned to the chlorinated ring, in accordance with related aromatic values [START_REF] Pretsch | Structure Determination of Organic Compounds[END_REF]. All results led to the identification of IX as drawn in Table 3.

Discussion

Study of the oxidative transformation of hydroxyphenylurea derivatives by laccase from T. versicolor clearly demonstrated the influence of the pH of the incubation mixture. First, the extent of degradation of the substrates greatly depended on the pH. The activity curve exhibited a bell shaped profile with a maximum between pH 4 and 5 for all three chlorinated hydroxyphenylureas tested. Such behavior is representative of phenolic substrates and was previously observed, among numer-ous examples, with syringic and vanillic acids [START_REF] Leonowicz | The effect of pH on the transformation of syringic and vanillic acids by the laccases of Rhizoctonia praticola and Trametes versicolor[END_REF], sinapic acid (28), and 2,4,6-trichlorophenol [START_REF] Leontievsky | Transformation of 2,4,6-trichlorophenol by free and immobilized fungal laccase[END_REF]. Xu [START_REF] Xu | Effects of redox potential and hydroxide inhibition on the pH activity profile of fungal laccases[END_REF] postulated that this bell shape profile is a consequence of two opposite effects: the ascending part of the curve is generated by the redox poten-tial difference between the reducing substrate and the type 1 copper of laccase, which increases with the pH, whereas the descending part results from the binding of an hydroxide anion to the type 2/type 3 coppers of laccase, which inhibits the activity at higher pH.

For all the tested phenylureas, the para-quinonic derivative was obtained as the predominant product at pH 3. At pH 5, the amount of 2-chlorobenzoquinone in the incubation mixture was only 20% of that at pH 3 (Figure 2). An earlier report of our group in regard to the transformation products of 4HF as a result of T. versicolor oxidation (19) led to a similar conclusion: at pH 3, the only products observed during the transformation of HF derivatives were p-benzoquinones. The formation of quinones in laccase-catalyzed reactions has been observed before at low pH for the enzymatic oxidation of vanillic and syringic acids [START_REF] Leonowicz | The effect of pH on the transformation of syringic and vanillic acids by the laccases of Rhizoctonia praticola and Trametes versicolor[END_REF] or chlorophenolic compounds [START_REF] Leontievsky | Transformation of 2,4,6-trichlorophenol by free and immobilized fungal laccase[END_REF]. Recently, Niedermeyer reported that amination of alkylated p-hydroquinones with aromatic amines using laccases proceeded through the inter-mediary of the formation of the corresponding quinone. Scheme 2 suggests a possible mechanism for the formation of a p-benzoquinone from 4HF and 4HF-3Cl. First, a laccase-catalyzed one-electron oxidation generates a phenoxy radical by removal of hydrogen and one electron from the phenoxy group of the parent hydroxyphenyl. A second enzymatic oxidation is then assumed to proceed, leading to the formation of a carbocation intermediate at the 4 position, favored due to a possible stabilization by resonance via the imine form of the molecule. At higher pH, the carbocation can lose a proton to form an imine which becomes the major final product of the reaction. In slightly acidic conditions, hydrolysis of the cationic group in the para position proceeds via a nucleophilic addition of water, leading to the formation of the corresponding pbenzoquinone. Such a mechanism involving two different pathways with a common intermediate species, i.e. the carbocation, is consistent with the experimental evidence that at a given pH the concomitant formation of the imine and the p-benzoquinone is observed.

At higher pH, 2HF and 2HF-4Cl, which are not substituted at the para position, do not lead to the formation of an imine but to dimeric compounds. As is the case in the previous mechanism, two successive laccase-mediated oxidations are leading to a cationic intermediate (Scheme 3). In a third step, the nucleophilic addition of the parent molecule at the carbocation leads to the formation of a carbon-carbon coupled dimer. Once the cation is generated, coupling is completed without further involvement of the enzyme. Preferential coupling at the C4 position might be due to less sterical hindrance of the corresponding cation, which is stabilized due to the presence of an urea group in the meta position.

An alternative mechanism involving a laccase-catalyzed one-electron oxidation of the substrate (2HF or 2HF-4Cl) to form a phenoxy radical and its subsequent dimerization cannot be ruled out (Scheme 3, radical dimerization route). In that case, assuming that the preferred entry of a new substituent is at the position ortho or para to the hydroxyl group, with the meta position being unreactive, three isomers of dimers with C-C bonds are expected to be formed from 2HF and 2HF-4Cl, as derived from the reaction between two radicals. However, only one of these, the 5,5-di(2-hydroxyphenylurea) dimer could be identified, which is consistent with the cation formation route rather than the radical dimerization one.

There are only a few examples of laccase-catalyzed formation of a C-C bond in the literature since the pioneering work of Bollag et al. [START_REF] Sjoblad | Polymerisation of 1-naphthol and related phenolic compounds by an extracellular fungal enzyme[END_REF][START_REF] Jonas | Transformation of 2-hydroxydibenzofuran by laccases of the white rot fungi Trametes versicolor and Pycnoporus cinnabarinus and characterization of oligomerization products[END_REF]. However, similar preferential coupling in the para position to the hydroxyl group was reported for sinapic acid [START_REF] Lacki | Transformation of 3,5-dimethoxy-4-hydroxy cinnamic acid by polyphenol oxidase from the fungus Trametes versicolor: product elucidation studies[END_REF], although 21 dimeric products could be expected if a random coupling took place. Recently, Ciecholewski et al. [START_REF] Ciecholewski | Laccase-catalyzed carbon-carbon bond formation: oxidative dimerization of salicylic esters by air in aqueous solution[END_REF] performed the oxidative dimerization of salicylic esters with laccase in an attempt to provide a new route for the synthesis of functionalized biaryls. Selective carbon-carbon bond formation was observed at the ortho position to the hydroxyl group of the salycilate, likely because it was the only unsubstituted position on the ring. Dimers formed by coupling via C-C bond formation remained phenolic compounds, thus likely to be potential substrates for T. versicolor laccase, which is known for its weak specificity. Tridimensional structure studies [START_REF] Bertrand | Purification and preliminary crystallographic study of Trametes versicolor laccase in its native form[END_REF] have shown that the enzyme possesses a large cavity (around 10 Å) at the reducing substrate active site. In the case of 2HF, it can thus be suggested that a second biocatalytic oxidation takes place following the coupling step, leading to the corresponding quinones VI and VIbis. However, no quinonic compound was found from 2HF-4Cl oxidation. The reason that the 2HF phenolic dimer is more readily oxidable than the corresponding 2HF-4Cl one is most likely because of the electron-withdrawing property of the chlorine, reducing the electron density at the phenoxy group, thus making 2HF-4Cl more difficult to be further oxidized. It should be noted that a dimeric quinonic compound was previously found as the final predominant product of enzymatic oxidation of sinapic acid [START_REF] Lacki | Transformation of 3,5-dimethoxy-4-hydroxy cinnamic acid by polyphenol oxidase from the fungus Trametes versicolor: product elucidation studies[END_REF].

Finally, it was found that the oligomers VIII and IX resulting from the coupling of aromatic rings via an ether bond were obtained from the laccase-catalyzed oxidation of 2HF-5Cl at pH 6.5. The proposed pathway (Scheme 4) includes the laccase-catalyzed formation of the corresponding p-quinone as the first step, followed by a subsequent nonenzymatic addition at the para position to the urea substituent via the hydroxyl group of the parent 2HF-5Cl. The presence of the p-quinone in the reaction medium is of course a prerequisite in this pathway and is supported by the experimental observation that the quinone formation is the major transformation route of 2HF-5Cl, even at pH 5. Although limited at similar pH for the other hydroxy-phenylureas such as 4HF or 2HF, the presence of p-quinone in the reaction medium can be explained by the more electrophilic character of the carbocation formed after the two-electron oxidation of 2HF-5Cl, due to the presence of chlorine, favoring the subsequent addition of water. A second argument is that, thanks to the hydrogen bond with the hydrogen of the urea amine, the delocalization of the nonbinding doublet of its nitrogen atom is favored, enhancing the leaving character of the chlorine atom (Scheme 4). However, quinone formation is only the first step of the 2HF-5Cl transformation pathway. As it is known that at pH 6.5 the transformation of the substrate is very low, the quinone is in the presence of a rather high concentration of unreacted substrate in solution, which is likely to undergo a nucleophilic addition on the quinone. A similar mechanism of addition on paraquinone was recently described for the synthesis of fungal laccase-catalyzed aminoquinones, resulting from the amination of phydroquinones with primary aromatic amines [START_REF] Niedermeyer | Nuclear amination catalyzed by fungal laccases: reaction products of p-hydroquinones and primary aromatic amines[END_REF] at an ortho position to the quinoic carbonyl, as observed in the present work.

Compound IX resulted from a nucleophilic substitution on the aromatic ring taking place para to the first C-O coupling site. One interesting point to note is the release of chlorine ions as a result of the coupling reaction. Chlorine atoms were released if they happened to be attached to carbon atoms engaged in a coupling or nucleophilic addition reaction. A similar observation was reported for laccasecatalyzed transformation of chlorophenols [START_REF] Park | Dehalogenation as a consequence of binding to humic materials[END_REF].

Conclusion

The hydroxyphenylurea derivatives studied in this work are metabolites or photochemical degradation products formed from phenylurea compounds widespread in the environment, including soils and waters. Functionalization of their aromatic ring through hydroxylation makes phenylureas potential substrates for oxidative enzymes and thus for a further degradation. This paper offers the first data on both the transformation mechanisms and the structure of the metabolites formed by transformation with laccase from T. versicolor, an oxidase produced in many soil fungi.

We have shown that laccase-mediated transformation of hydroxyphenylureas led to quinones at pH 3.

By contrast, at higher pH values, these compounds are further transformed into oligomers by chemical processes. The polymerization process is of special importance for the environmental fate of HF derivatives. First, when the oxidation coupling reaction occurs at the chlorinated site of the substrate, the induced dehalogenation contributes to the overall detoxification effect since it is generally recognized that toxicity decreases after dechlorination. Second, as we postulated that the mechanism of covalent bond formation involves the addition of a nucleophile, leading to the formation of heterodimers, the same mechanism is likely to take place between quinones resulting from the two electron oxida-tion of hydroxyphenylureas and humic acids present in soils. Such a polymerization process of binding to humic acid has been previously reported for chlorophenols [START_REF] Park | Dehalogenation as a consequence of binding to humic materials[END_REF]. Potentially toxic quinones formed from laccase-catalyzed oxidation of HF derivatives are thus likely to be bound to the organic matter of soils and thus to lose their bioavailability and, as a consequence, their ecotoxicity.

  N',N'-Dimethyl-N-(4-hydroxyphenyl)urea (4HF), N',N'-dimethyl-N-(2-hydroxyphenyl)urea (2HF), N',N'-dimethyl-N-(4-chloro-2-hydroxy-phenyl)urea (2HF-4Cl), N',N'-dimethyl-N-(5-chloro-2hydroxyphenyl)-urea (2HF-5Cl), and N',N'-dimethyl-N-(3-chloro-4-hydroxyphenyl)urea (4HF-3Cl) were prepared according to procedures adapted from the literature (19-21) (see Supporting Information). The chemical structures of these compounds are shown in Scheme 1.

  1 H and 13 C NMR spectra were recorded at 293 K, on Bruker AC 200 equipment or on a Varian Mercury plus 300 instrument in deuteriochloroform (CDCl3), benzene-[D6], or DMSO-[D6]. Chemical

Figure 1 .

 1 Figure 1. Influence of pH on the transformation of chlorinated hydroxyphenylureas by laccases: () 2HF-5Cl; (•) 4HF-3Cl; () 2HF-4Cl. Substrate initial concentration: 0.1 g/L in 0.1 M citrate/phosphate buffer; T ) 30°C. Laccase concentration: 0.011 units for 2HF-5Cl; 0.05 units for 4HF-3Cl and 2HF-4Cl. Incubation duration: 15 min.

Figure 2 .

 2 Figure 2. Time course of 4HF-3Cl and product concentrations during the transformation of 4HF-3Cl by T. versicolor laccase (0.03 units/mL) at pH 3 (A) and 5 (B): (b) 4HF-3Cl; (•) product I; (▲) product V.

Table2.[START_REF] Garmouma | Spatial and temporal variations of herbicide (triazines and phenylureas) concentrations in the catchment bassin of the Marne river (France)[END_REF] H NMR and mass spectroscopic data for hydroxyphenylurea transformation products at pH 5

Scheme 3. Possible reaction mechanism in the laccase-catalyzed oxidation of 2HF and 2HF-4Cl at pH

Scheme 4. Possible reaction mechanism in the laccase-catalyzed oxidation of 2HF-5Cl at pH 6.5

Acknowledgment

We thank Dr. Debrauwer (UMR 1089 Xénobiotiques, INRA-ENVT, Toulouse) for acquisition of the APCI mass spectrum of compound VII and Dr. David Tepfer for the careful reading of the manuscript. Supporting Information Available: Synthesis procedure and spectroscopic data ( 1 H and 13 C NMR, mass spectrometry) of compounds 2HF, 4HF, 4HF-3Cl, 2HF-4Cl, and 2HF-5Cl and compounds I-IX. This material is available free of charge via the Internet at http://pubs.acs.org.

Table 3. 1 H NMR and mass spectroscopic data for 2HF-5Cl transformation products at pH 6.5 Scheme 2. Possible reaction mechanism in the laccase-catalyzed oxidation of 4HF and 4HF-3Cl


Yuancheng Wang
email: wangyuancheng@seu.edu.cn

Yuyang Jing
email: jingyuyang@nua.edu.cn

Wei Wei
email: weiwei@njxzc.edu.cn

Dorian Cazau
email: dorian.cazau@imt-atlantique.fr

Olivier Adam
email: olivier.adam@sorbonne-universite.fr

Qiao Wang
email: qiaowang@seu.edu.cn

PipaSet and TEAS: A Multimodal Dataset and Annotation Platform for Automatic Music Transcription and Expressive Analysis dedicated to Chinese Traditional Plucked String Instrument Pipa

Keywords: Multimodal Dataset Creation, Automatic Music Transcription, Playing Technique Analysis, Optical Sensoring, Annotation System

Music Information Retrieval (MIR) develop rapidly these years, Automatic Music Transcription (AMT) and Expressive Analysis is increasingly gaining momentum on both Western and non-eurogenic music. However, the annotated datasets for non-eurogenic instruments remain scarce on quantity and feature diversity so that general evaluations and data-driven models on various tasks cannot be well-explored. As one of the most popular traditional plucked string instruments in Asia barely analyzed in MIR community, pipa has lots of distinctive national and local characteristics mainly including 4 classes of sophisticated playing techniques greatly enhancing the music expressiveness. Our work aims to systematically clarify an efficient procedure for the multi-modal pipa dataset creation which consists of audio, musical notations and multi-view videos of Chinese traditional solos. The use of 4-track string vibration signals captured by optical sensors paves a path to high quality annotation. A Transcription and Expressiveness Annotation System (TEAS) is transparently implemented to ensure the scalability of dataset. Finally, a series of existent and new MIR tasks enabled by this dataset are enumerated to explore in the future.

I. INTRODUCTION

A. BACKGROUND AND TREND TO THE MULTIMODAL AUTOMATIC MUSIC TRANSCRIPTION AND EXPRESSION ANALYSIS

A utomatic Music Transcription (AMT) and polyphonic music analysis have emerged for almost 50 years [START_REF] James | On the segmentation and analysis of continuous musical sound by digital computer[END_REF]. As a fundamental task of the MIR, it's still very challenging on the basis of results in the MIR community [START_REF]Music Information Retrieval Evaluation eXchange (MIREX[END_REF]. The purpose of AMT system narrowly defined in [START_REF] Anssi | Automatic music transcription as we know it today[END_REF] aims to indicate basically a low-level quadruple i.e. pitch, intensity, onset time, and duration of each sound that was played. [START_REF] Benetos | Automatic Music Transcription: An Overview[END_REF] summarizes the frame-level, note-level, stream-level and notation-level transcriptions in which the notation-level one, also referred to as Complete Notation Transcription (CNT), converts an audio signal into some form of music notations, such as sheet music [START_REF] Ycart | Investigating the Perceptual Validity of Evaluation Metrics for Automatic Piano Music Transcription[END_REF]. However, the music notations cannot completely reflect the characteristics of different performers.

The prior knowledge, such as the music, acoustic features of particular instruments, is always taken into account at the beginning of the research. The AMT studies on piano [START_REF] Emiya | Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle[END_REF] and guitar [START_REF] Barbancho | Automatic transcription of guitar chords and fingering from audio[END_REF] introduce the prior knowledge of the occidental instruments and music theory, the others on vocals in Indian VOLUME 4, 2021 music [START_REF] Miryala | Automatically Identifying Vocal Expressions for Music Transcription[END_REF], Ney and Tanbur in Turkish Makam music [START_REF] Benetos | Automatic transcription of turkish microtonal music[END_REF] and Marovany zither in Madagascar music [START_REF] Cazau | An automatic music transcription system dedicated to the repertoires of the marovany zither[END_REF] investigate the the non-eurogenetic music transcriptions which manifest the characteristics distinct from the assumptions of the occidental music conventions, e.g. non-equal temperament, free tempo, specific playing techniques and musical notations which are highly relative to the regional culture, styles and genres. These works greatly expand the scope of the MIR research and have developed an inter-disciplinary field, computational ethnomusicology [START_REF] Tzanetakis | Computational ethnomusicology: A music information retrieval perspective[END_REF], [START_REF] Holzapfel | Automatic Music Transcription and Ethnomusicology: a User Study[END_REF]. De-centering the western music has become a path to the cultural diversity in MIR field [START_REF] Huang | De-centering the west: East asian philosophies and the ethics of applying artificial intelligence to music[END_REF].

In recent decade years, the expression analysis [START_REF] Fabian | Expressiveness in Music Performance: Empirical Approaches across Styles and Cultures[END_REF] and the multimodality [START_REF] Müller | Cross-Modal Music Retrieval and Applications: An Overview of Key Methodologies[END_REF] have become promising trends MIR community. Depicting a rich variety of individual emotions, styles and culture, the expression analysis serves to playing technique detection and parameter estimation in which the latter promotes the concept of CNT. The multi-modalities provide a wider perspective and more direct perception to the music features, particularly in the complex scenarios [START_REF] Leman | On the role of the hand in the expression of music[END_REF], [START_REF] Gainza | Automating ornamentation transcription[END_REF] (for more details, please see section IV.C). Audioonly playing technique analyses on electric guitar [START_REF] Reboursière | Left and right-hand guitar playing techniques detection[END_REF]- [START_REF] Abeßer | Instrument-Centered Music Transcription of Solo Bass Guitar Recordings[END_REF], bowed strings [START_REF] Barbancho | Transcription and expressiveness detection system for violin music[END_REF]- [START_REF] Kruger | Playing technique classification for bowed string instruments from raw audio[END_REF] and Chinese Bamboo flute [START_REF] Wang | Adaptive time-frequency scattering for modulation recognition in music signal[END_REF], [START_REF] Wang | Playing technique recognition by joint time-frequency scattering[END_REF] enrich the functionality in a single angle. The multimodality analysis covers the synchronization, similarity, timedependent representation and classification in MIR tasks [START_REF] Simonetta | Multimodal Music Information Processing and Retrieval: Survey and Future Challenges[END_REF], [START_REF] Zhonghua | Multimodal music information retrieval: From content analysis to multimodal fusion[END_REF], most of works usually focus on audio, video, score modalities which can accompany with expression analysis [START_REF] Perez-Carrillo | Estimation of Guitar Fingering and Plucking Controls based on Multimodal Analysis of Motion, Audio and Musical Score[END_REF], [START_REF] Li | Creating a Multitrack Classical Music Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications[END_REF]. More generally, the sensoring components like the key pressure transducer in piano [START_REF] Emiya | Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle[END_REF] and the strain gauge to acquire bowing motion for string quartet [START_REF] Marchini | The sense of ensemble: A machine learning approach to expressive performance modelling in string quartets[END_REF] seen as the physical layer modalities are commonly applied to efficient music analysis and annotation.

B. DESCRIPTION OF PIPA AND PIPA MUSIC

Pipa, the undisputed head of the traditional plucked string instruments in China, has over 2000 years of history. The most ancient straight-necked pipa can be traced back to the Qin Dynasty. The bent-neck pipa was then re-introduced through the Silk Road from Persia and horizontally played as the on-the-horse musical instrument. With the evolution of the playing techniques and repertories, the pentatonic pipa became obliquely played with plectrum then vertically played with fingers at Tang Dynasty. It has been progressively integrated into Chinese music cultures and spread to the other countries in East and Southeast Asia as the most popular plucked string instrument.

1) Pipa Construction

In this paper, we are interested in the modern Chinese pipa standardized in the middle of last century which are chromatic and unified as pear shape. The fake nails substitute for fingernails and plectrums so as to more agilely perform the crisp sound. The research object in our work is a six-ledged (Xiang, 相), twenty-four fretted (Pin, 品), four-stringed pipa [35] and a suit of celluloid-made translucent fake nails (See Figure 1). Ledges count into frets in what follows.

Similarly to the acoustic guitar, the modern pipa is often equipped by the steel strings with different thickness in which the 3 thick strings are copper string-wrapped or nylonwound. The thickness and materials physically determinate the amplitude distribution and inharmonicity of harmonic series [START_REF] Rossing | The science of string instruments[END_REF] which allow the estimation of the string, fret and plucking positions from audio [START_REF] Møller | Estimation of guitar string, fret and plucking position using parametric pitch estimation[END_REF], [START_REF] Dittmar | Real-time guitar string detection for music education software[END_REF]. Furthermore, according to the pipa construction and performance rules in [START_REF] Zhuang | Pipa Handbook[END_REF], the playable pitch range are shown in Table 1. Concretely, the typical tuning method, Standard D, i.e. A2, E3, D3, A3, is chosen upon the open strings to simplify the following recordings. Notice that the pitch interval between two adjacent frets corresponds to approximately a semitone except those of the lowest 2 frets of 1st and 2nd strings, the register ranges thus from A2 to E6. Considering both register and partials of high tones, the frequency range covering at least from 110 Hz to 14000 Hz is recommended to take up the subsequent research.

2) Music Notations and Taxonomy of Playing Techniques

The ancient pipa notations, e.g. GongChe (工尺) notations, contains the symbols written with Chinese characters and radicals, traditional musicians pay attentions to oral transmission which more or less deprives of key music features in ancient notations like tempo and rhythm. The modern pipa notation is a tonic sol-fa notation combining with both Western and Chinese musical symbols of which a font example could be found in [START_REF]ZPshen pipa fonts[END_REF]. Besides the basic normal plucks, the fingerings, string number, playing techniques are versatile and especially highlighted in pipa music. Shown in Figure 2, lots of articulation symbols all over the notation reflect the indispensable needs for pipa music transcription. The nomenclature of pipa, originated from two fingerings Pi and Pa, respectively meaning the plucking forwards and backwards, implies the diversity and complexity of playing techniques.

Above all, over 60 techniques [START_REF] Feng | Some acoustical measurements on Chinese musical instrument pipa[END_REF] must be categorized into the identical or similar notions in the sense of Western music theory. Firstly, we distinguish the basic fingerings and articulations in pipa music. The plucking string, position, finger, direction, inclination for right hand and the termination position (fingertips upon the fret or beside the fret) for left hand are classified into the basic fingerings, so are the techniques constituted by basic pluck(s) like doublepluck (ShuangTan, 双弹) which represents the simultaneous plucks on two different strings. Vibrato, sliding, tremolo, strumming and those perfectly correspond to the articulation types in Western music framework. Furthermore, since the performance parameters have a major impact on a listener's perception of the music [START_REF] Clarke | Listening to Performance[END_REF], the names are often accompanied by parameters in pipa techniques, e.g. the triple tremolo (SanLun, 三轮) which contains 3 intensive plucks sequentially played with index, middle and annular fingers. This signifies not only the detection to indicate the articulation intervals but also the parameter estimation within the intervals must be involved in pipa analysis framework. The left hand techniques (i.e. vibrato, trilling, bending, sliding, harmmeron/pull-off), the right hand ones (i.e. tremolo, strumming), as well as their corresponding parameters are gathered in

C. PRINCIPLE OF DATASET CREATION AND ORGANIZATION OF THE PAPER

The collection and analysis framework for PipaSet follows the principles described by Su and Yang [START_REF] Su | Escaping from the Abyss of Manual Annotation: New Methodology of Building Polyphonic Datasets for Automatic Music Transcription[END_REF] and Xi [START_REF] Li | Creating a Multitrack Classical Music Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications[END_REF]:

• Generality: This dataset covers realistic, complex, polyphonic musical phrases, the performers' motions and scores without any abridgement on original pipa repertoires. The self-interpretation on tempo, rubato, dynamic variations and ornamentation is allowed to highlight the natural charm of Chinese music. The recording progress is developed in popular style. • Quality: In order to preserve nuances in the performance, including the pitch deviation and articulations, we elaborate a vibration sampling device for individual string and provide multiple annotation formats to ensure high quality annotations. • Efficiency: The annotation and visualization are so automated through the highly interpretable mapping algorithms that the music experts could focus on manual corrections in most of cases. PipaSet can be easily extended for this reason. • Cost: The equipment is either common for recording studio or very affordable to acquire like a small piece of Printed Circuit Board (PCB) with optical switches.

The remainder of paper is organized as follows. The multimodal recordings, including the optical sensors to capture vibration signal from individual string, are present in Section II. Section III shows an annotation platform implemented using MATLAB GUI that benefits from optical vibration recordings. In Section IV, dataset statistics, detailed acoustic and music features for target pipa and repertories used in our study and applications empowered by this dataset are present. Finally, Section V contains the conclusion and the future work for this dataset.

II. MULTIMODAL RECORDINGS

The dataset creation, always motivated by the research objectives, has begun to extend to many non-audio modalities in MIR tasks. The recording devices and music instruments for VOLUME 4, 2021

Datasets

Microphone(s) Camera(s) Sensor type(s) Instrument/Genre MAPS [START_REF] Emiya | Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle[END_REF] Single-channel Key Pressure Transducer Piano MAESTRO v3 [START_REF] Hawthorne | Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset[END_REF] Dual-channel Key/pedal Pressure Transducer Piano Yang [START_REF] Yang | AVA: A Graphical User Interface for Automatic Vibrato and Portamento Detection and Analysis[END_REF] Single-channel Erhu, Violin. EEP [START_REF] Marchini | The sense of ensemble: A machine learning approach to expressive performance modelling in string quartets[END_REF] Single-channel contact Piezoelectric pickup/Strain gauge/ String quartet Electromagnetical position sensor CBF dataset [START_REF] Wang | Adaptive time-frequency scattering for modulation recognition in music signal[END_REF], [START_REF] Wang | Playing technique recognition by joint time-frequency scattering[END_REF] Single-channel Chinese bamboo flute Multimodal Guitar [START_REF] Perez-Carrillo | Estimation of Guitar Fingering and Plucking Controls based on Multimodal Analysis of Motion, Audio and Musical Score[END_REF] Dual-channel contact Multi-view Electric guitar Guitar playing technique [START_REF] Su | Sparse cepstral, phase codes for guitar playing technique classification[END_REF] Dual-channel Electric guitar IDMT-SMT [START_REF] Kehling | Automatic tablature transcription of electric guitar recordings by estimation of score-and instrument-related parameters[END_REF] Dual-channel Electric guitar URMP [START_REF] Li | Creating a Multitrack Classical Music Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications[END_REF] Single-channel Single-view Multi-instruments Marovany Zither [START_REF] Cazau | Information retrieval of Marovany zither music with an original optical-based system[END_REF] Single-channel Optical switch Marovany zithers GuitarSet [START_REF] Xi | GuitarSet: A Dataset for Guitar Transcription[END_REF] Single-channel Magnetic pickup Guitars CTIS dataset [START_REF] Liang | Constructing a multimedia Chinese musical instrument database[END_REF] Dual some related datasets can be found in Table 3. The manual annotation from polyphonic and complex audio recordings has low-efficiency and uncontrollable deviation for different annotators, the convenience and resolution for high-quality annotation must be considered within selected modalities. In this section, the sensors to capture the signal from individual string, the device deployment for pipa polyphonic recording are described as follows.

1) Single-string Vibration Recording using Optical Sensors

The string vibration sensing, physically filtering the external noise like those from fake nails and knocks, can be mainly divided into three types. Usually located on the bridge, the piezoelectric pickups [START_REF] Marchini | The sense of ensemble: A machine learning approach to expressive performance modelling in string quartets[END_REF], [START_REF] Angulo | Hexaphonic guitar transcription and visualization[END_REF] for guitar and violin are not adapted to the pipa construction in which the strings are directly dragged through the tailpiece without need of saddle support. Xi [START_REF] Xi | GuitarSet: A Dataset for Guitar Transcription[END_REF] utilizes the hexaphonic pickups to capture the vibration but no similar products for our instrument.

Cazau [START_REF] Cazau | Information retrieval of Marovany zither music with an original optical-based system[END_REF] provides a simple reference optical system for unmovable Marovany zithers using an external support and proves the distortion impact of optical sensors is minor and acceptable. To achieve detachability, we customize a system using small optical switches mounted within the small gaps between the strings and the soundboard unlike those on Zithers so as to prevent from disturbing the performance (See Figure 3). The principle of the optical switches is to detect the surface shadowed by strings, the string vibrate within a small sensing interval to avoid the clipping effect and waveform warping. The switches are thus placed close to tailpiece to ensure the absence of low frequency wave nodes and the amplitude range appropriate to the optical switch. A lobulated PCB is designed to independently adjust switch heights to strings using the fine-tuning screws. The waterproof tape wrapping around the screws avoids the loose noise.

2) Multimodal Device Deployment and Synchronization

A pair of sE8 condenser microphones [START_REF]sE Electronics sE8 Small Diaphragm Microphone[END_REF] with a A/B stereo mode is placed at 0.3 m of distance pointed to the sound hole under the tailpiece. The 6-track analog signals from optical switches and microphones are all connected to a single RME fireface UFX sound card [START_REF]RME fireface UFX sound card[END_REF] to ensure the synchronization. All 6-channels signals are sampled with rate of 48 kHz and depth of 24. In addition, 3 industrial cameras using Sony IMX214 CMOS image chips are selected to capture the multi-view finger behaviors to avoid the occlusion from both hands with the sampling rate of 30 fps and resolution of 1080p. Two adjacent ones have 75 degree difference oriented to the performer with 0.7 m of distance. Landtop Faist anechoic chamber provides an ambience preventing from the spatial acoustics during recording, in which the green background curtain facilitates video processing like character matting. The layout is shown in Figure 4. The different colors of nail stickers adhered on the fake nails of each finger are chosen as visually conspicuous markers (See right plot of Figure 1). Synchronization among different devices is still challenging for multi-modal recordings. The externally triggered cameras are often misaligned with the audio system. In our study, the weak microphones embedded in cameras are activated to align audio-visual devices using "synchronize clips" function of the Final Cut Pro software.

III. TEAS: A TRANSCRIPTION AND EXPRESSIVENESS ANNOTATION SYSTEM

The annotation is a time-consuming and labor-intensive work which often requires the field expertise. In order to maximize the efficiency and automation, we greatly enhance the AVA platform [START_REF] Yang | AVA: A Graphical User Interface for Automatic Vibrato and Portamento Detection and Analysis[END_REF] which only focuses on the continuous pitch, vibrato and portamento. The design notion of TEAS platform is driven by principles below:

• The pipeline is clear, concise and reserves the vast majority of performance-level rather than note-level attributes.

• Distinct from transcription system, the annotation system aims to a new instrument and accompanied with high interpretability and controllability (lowparameter). A suitable low-precision for mapping algorithms is favorable as the deletion is easier than adding. • The user interaction modules, e.g. visualization, efficient manual correction strategies and import/export in different formats, need to be involved at arbitrary step. Shown in Figure 5, the annotation usually operates from the the low-level to the high-level music representations as a progressive dimension reduction procedure. First we try to remove the inevitable inter-stringed resonance with Kernel Additive Method Interference Reduction (KAMIR) source separation method [START_REF] Pratzlich | Kernel additive modeling for interference reduction in multichannel music recording[END_REF] referred to [START_REF] Xi | GuitarSet: A Dataset for Guitar Transcription[END_REF]. Embedded in TEAS, KAMIR module decomposes 4 channels of string vibration signals into 4 tracks of debleeded signals, meanwhile the diagonal elements in initial interference matrix are configured as 1 and 0.8 for the rest to ensure channel matching. Although the MMSE-LSA adaptive filter [START_REF] Ephraim | Speech enhancement using a minmum meansquare error log-spectral amplitude estimator[END_REF] could also reduce the parasite noise from electronics and the mutual resonances of strings [START_REF] Cazau | Information retrieval of Marovany zither music with an original optical-based system[END_REF], the separated signal sounds much more likely to the realistic audio recording. Figure 6 shows an example of the raw optical signal and debleeded signal of first string, the significant resonance note removal from another strings can be found located between 6.8-7.8s.

Similarly to Tony [START_REF] Mauch | Computer-aided Melody Note Transcription Using the Tony Software: Accuracy and Efficiency[END_REF] and Melodyne [START_REF]Celemony Melodyne Studio[END_REF] softwares, the monophonic AMT modules in TEAS indicate boundaries, pitch contours, note segments and corresponding intensity on each string as the second step. Furthermore, playing technique analysis modules using the above-achieved features are elaborated. The screenshots of TEAS are shown in Figure 7.

A. MONOPHONIC AUTOMATIC MUSIC TRANSCRIPTION

1) Pitch Estimation and Tracking

Pitch detection always confronts with the challenges of the noise and time-variance in music and speech recordings. The traditional pitch tracking algorithms can be mainly classed into non-parametric and parametric approaches. In most of recent dataset creation works, the pitch contours are extracted by the non-parametric state-of-the-art pYIN [START_REF] Mauch | pyin: A fundamental frequency estimator using probabilistic threshold distributions[END_REF].

A parametric method, Bayesian pitch tracker using Nonlinear Least Square (BNLS) [START_REF] Shi | Robust Bayesian Pitch Tracking Based on the Harmonic Model[END_REF], is provided. The input signal is sliced into N frames and expressed as Assuming the partial frequencies as the integral multiples of the fundamental frequency, the main idea of parametric methods is to reconstruct the y n with an ensemble of Fourier bases in terms of harmonic series within preset register which can be formulated as follows:

Y N = VOLUME 4, 2021
y n = u n K k=1 Z(ω k,n )a k,n + ε n = u n K k=1 Re(d k,n Z c (ω k,n )) + ε n (1)
where

Z(ω k,n ) = 0 sin(ω k,n ) ... sin((M -1)ω k,n ) 1 cos(ω k,n ) ... cos((M -1)ω k,n ) T (2)
Z c (ω k,n ) = [1, exp(iω k,n ), ..., exp(i(M -1)ω k,n )] T (3)
u n ∈ {0, 1} determines whether n th frame is voiced or not, ω k,n = kω 1,n denotes the normalized radian frequency at k th partial and n th frame and K the total order of wave function, ε n a Gaussian white noise vector respects N (0, σ 2 n I), the real amplitude vector

a k,n = [α k,n , β k,n ]
T and d k,n ∈ C contain the amplitude and initial phase. The g-prior for order constraint and Viterbi decoder are proven effective to reduce the octave errors.

A dimension reduction approach allows a shorter window size [START_REF] Nielsen | Grid size selection for nonlinear least-squares optimization in spectral estimation and array processing[END_REF] which is more appropriate to the pitch shift based techniques. Furthermore, comparing to pYIN, BNLS returns experimentally more consistent pitch contours under the voice activity smoothness, particularly in case of tremolo. Finally, since the pitch resolution is always limited by the searching bins, Fibonacci search for de-quantification [START_REF] Antoniou | Practical Optimization: Algorithms and Engineering Applications[END_REF] and the iterative methods [START_REF] Morfi | A computationally efficient refinement of the fundamental frequency estimate for the Adaptive Harmonic Model[END_REF] for each frame don't only take time but also lose the smoothness from Viterbi decoder. Alternatively, inspired by pYIN algorithm, we refine the pitch using parabolic interpolation on loss function of the BNLS.

2) Onset and Offset Detection

As another primary features in audio analysis usually fetch the critical structural prior, the onsets denotes the plucking points and offsets reflects the duration of a note event. Despite the voice activity detection (VAD) often bound with pitch estimators [START_REF] Drugman | Voice Activity Detection: Merging Source and Filter-based Information[END_REF], the estimated boundaries still have a better a time-domain alignment and improve accuracy for subsequent tasks, e.g. note segmentation and playing technique detection. Energy peak detection generally has a sufficient performance for monophonic onset detection. Involved with simultaneously amplitude pitch shift, the coarse onset detectors, e.g. SpecFlux [START_REF] Reboursière | Left and right-hand guitar playing techniques detection[END_REF], SuperFlux [START_REF] Widmer | Maximum filter vibrato suppression for onset detection[END_REF] and ComplexFlux [START_REF] Böck | Local Group Delay Based Vibrato and Tremolo Suppression for Onset Detection[END_REF], always follow with energy-based time alignment step [START_REF] Jehan | Creating music by listening[END_REF].

Specially, the fake nail attack on the string brings a pair of adjacent peaks for a single tone, namely attack peak and natural transient, which respectively serve to note segmentation and velocity estimation. The above-mentioned onset detectors may introduce ambiguity of local peaks. To overcome this issues, we directly employ the log-energy envelope of high-pass filtered signal (See the bottom plot of Figure 6) and set parameters of boundary detectors as follows:

• Peak-picking: we use the peak picker in [START_REF] Widmer | Maximum filter vibrato suppression for onset detection[END_REF] and decrease the peak-picking parameters for the short intervals of peak pairs. The n th frame with respect of Logenergy Envelope (LE) is selected as an potential onset if it fulfills 3 following conditions with the thresholds δ 1 = 25ms, δ 2 = 25ms, δ 3 = 25ms, δ 4 = 25ms, δ 5 = 0.025 and δ 6 = 25ms.

1. LE(n) = max(LE(n -δ 1 : n + δ 2 )) 2. LE(n) ≥ mean(LE(n -δ 3 : n + δ 4 )) + δ 5 3. n -n P reviousN ote > δ 6 (4)
• Offset detection: Given two types of correct onsets, a tone end until signal energy remaining 5% or next pluck starts.

• Playing technique cases: The offset of last pluck at the tail of tremolo interval respects above law, the detected onsets within tremolo interval needs to be eliminated. In case of pitch fluctuation techniques, the last one with 5% of local maximal energy or the end of vibrato is selected as offset. The pitch transition don't count into the offset of a note otherwise another pluck starts.

3) Note segmentation, Intensity, String and Fret Position

The note segments maintain the most direct connection to music scores. The pitch in unit of integer Musical Instrument Digital Interface (MIDI) value is converted from the median of pitch contours within note segment. The pitch discretization for pitch shift techniques like vibrato and sliding increase is necessary in our platform. To this end, the pitch2note algorithm in Tony which employs Hidden Markov Model (HMM) with 3 states, attack, stable and silence is chosen. In parallel, the intensity or the velocity in MIDI format denotes the loudness of a note embodied at natural transient. Notice that the intensity of the destination of sliding tones are set as 0. Given track index and note annotation, the string and fret position can be easily achieved through the presets.

B. PLAYING TECHNIQUE ANALYSIS AND ANNOTATION

Playing technique analysis generally concerns about both detection and parameter extraction within the detected ranges.

In TEAS, the intervals and parameters of pitch fluctuation techniques (vibrato, trilling and bending), pitch transition techniques (sliding, slide in/out hammer-on, pull-off), tremolo and multi-track techniques (strumming, arpeggio) which occupy predominant portion of articulations, are automated to facilitate the annotation.

1) Pitch Fluctuation Analysis

Based on the pitch contour in continuous MIDI values, the pitch fluctuation parameter estimation is often viewed as a single harmonic inversion problem. In early work, Abeßer [START_REF] Abeßer | Score-informed analysis of intonation and pitch modulation in jazz solos[END_REF] proposes a note-level autocorrelation-based method which doesn't work to extent estimation and bending technique. The early short time non-parametric approaches like Time Fourier Transform (STFT) [START_REF] Herrera | Vibrato extraction and parameterization in the spectral modeling synthesis framework[END_REF], [START_REF] Ventura | Accurate analysis and visual feedback of vibrato in singing[END_REF] and crosscorrelation of amplitude and frequency [START_REF] Coler | Vibrato detection using cross correlation between temporal energy and fundamental frequency[END_REF] have a related low performance on both detection and parameter estimation.

Yang [START_REF] Yang | The filter diagonalisation method for music signal analysis: frame-wise vibrato detection and estimation[END_REF] utilizes a parametric damped sinusoidal estimator, Filter Diagonalization Method (FDM), that achieves a better fitting to pitch fluctuation compared to early works. Since the vibrato detection realized by hard thresholding or Decision Tree (DT) always relies on the instrumentation and playing styles, in TEAS we provide a single parameter method that could be steered by scroll between 0-1. Power Ratio (PR) [START_REF] Sohn | A voice activity detector employing soft decision based noise spectrum adaptation[END_REF], [START_REF] Fisher | Generalized Likelihood Ratio Test for Voiced-Unvoiced Decision in Noisy Speech Using the Harmonic Model[END_REF] widely used in VAD and its variant Power Difference (PD) to vibrato detection formulated with ωn as singlecomponent version of the Eq. ( 9):

ωn = max ωn |y T n Z c (ω n )| 2 (5) P R per = y T n ŷn y T n y n = |y T n Z c (ω n )| 2 y T n y n > η per (6)
P D per = |y T n Z c (ω n )| 2 -σ 2 vib > M λ per (7)
Under the hypothesis of the power subtraction method, the noise is independent to the signal and respects a Gaussian distribution with the standard deviation σ vib .

σ 2 vib = y T n y n -Re( dn Z c (ω n )) T Re( dn Z c (ω n )) (8)
Thus the P D per can be computed with

P D per = 2|y T n Z c (ω n )| 2 -y T n y n > M λ per (9)
the PR and PD on FDM can be constructed in same manner:

P R F DM = y T n Re( dn Z c (ω c n )) y T n y n > η F DM ( 10
)
P D F DM = 2Re( dn Z c (ω c n )) T Re( dn Z c (ω c n )) -y T n y n > M λ F DM (11) VOLUME 4, 2021
where the complex amplitude dc and complex frequency ωc n are estimated by the FDM. All the thresholds λ per , λ F DM , η per and η F DM mostly distribute between 0 and 1 that could be controlled by a scroll in practical application.

2) Pitch Transition Analysis

Wang proposes an SVM based model for 3 pitch evolution techniques, acciacatura (Short sliding ornament), portamento (Continuous pitch shift), glissando (Discrete pitch shift) for Chinese bamboo flute [START_REF] Wang | Playing technique recognition by joint time-frequency scattering[END_REF]. Yang also proposes a portamento detector in AVA [START_REF] Yang | AVA: A Graphical User Interface for Automatic Vibrato and Portamento Detection and Analysis[END_REF] and parameter [START_REF] Yang | Logistic Modeling of Note Transitions[END_REF] on the vibrato-free pitch contours for non-fretted instruments. Similarly to the 3-stated HMM for note segmentation, the down-steady-up states models the pitch evolution to detect the portamento. Logistic regression is used to estimate the parameters of note transition: inflection point, growth rate, antecedent and subsequent pitches. Strictly speaking as a fretted instrument, pipa has only glissando for note transition and slide in out around note boundary produced for pipa instrument and count in sliding technique. After smoothing on the vibrato-free pitch contour, we find the portamento modeling in AVA still works for pipa.

3) Tremolo Analysis

The tremolo in pipa, with 3 fingerings termed as wheel (Lun, 轮) on first or forth string, rolling (Gun, 滚) horizontally played by index finger, shaking (Yao, 摇) alternatively played by thumb and index mainly on middle two strings, is the most common technique in pipa articulations. This effect is physically constituted by a succession of rapid plucks on same string often accompanied with pitch shift techniques and dynamic variation. Hence, we parametrize the tremolo by the initial intensity, rate, number of plucks and fingering types within a segment of note (See Table 2). Different from the literal tremolo rates for violin [START_REF] Barbancho | Transcription and expressiveness detection system for violin music[END_REF], acoustic guitar [START_REF] Freire | Study of the tremolo technique on the acoustic guitar: Experimental setup and preliminary results on regularity[END_REF] and flute [START_REF] Wang | Adaptive time-frequency scattering for modulation recognition in music signal[END_REF] respectively range between 3-6 Hz, 3-8 Hz, 5-12 Hz, pipa has an minimal pluck interval down to 60 ms for synthesis [START_REF] Chen | Sound synthesis of the pipa based on computed timbre analysis and physical modeling[END_REF] and 30 ms for real recording. Inspired by [START_REF] Freire | Study of the tremolo technique on the acoustic guitar: Experimental setup and preliminary results on regularity[END_REF] which estimates the parameters after onset detection, we leverage the approach in III.B.2) the for plucking points and natural transients within each note. Given a series of corrected plucked points P i within a tremolo note, the number of plucks P N > 2 defines the characteristic like the triple tremolo. The tremolo rate T R is computed by:

T R = P N -1 i=1 P i+1 -P i P N -1 (12)

4) Strumming Analysis

The automatic recognition of the strumming technique in TEAS platform are realized by the plucking points imported from each track. In our research, we propose a rule-based method which allows to aggregate the neighbouring plucks from orderly strings into a strum interval. The starting/ending strings and rate similarly defined to Eq. ( 12) within a interval are used to characterize the types of techniques. For example, the down tremble arpeggio represents the arpeggio performed from 3rd string to 1st one.

C. VISUALIZATION, CORRECTION, IMPORT/EXPORT AND VALIDATION

A multitrack tab (See plot (f) in Figure 7) incorporates an overall visualization for articulations, source separation (via KAMIR module), import/export modules of the whole project for quick global load. In order to better recover musicians' performance and simplify the validation from auditory perspective, we propose to export MIDI file as the load file of synthesizer like Ample China Pipa synthesizer [START_REF] Ample | [END_REF]. The playing techniques are supported by key switch beyond that of pipa register, the more exquisite expressions like pitch fluctuation can be realized by the Continuous Controller (CC) channels. The MIDI module in Intelligent Sound Processing (ISP) toolbox [START_REF]Intelligent Sound Processing (ISP) toolbox[END_REF] bridges the sophisticated annotations and MIDI files. Although the synthesized timbre and envelope sound different from that in our pipa but sufficiently natural for validation between synthesized audio and microphone recordings. From visual point of view, the tempo is indispensable for music score generation. Through the mean local autocorrelation approach in [START_REF] Grosche | Cyclic tempogram -A mid-level tempo representation for music signals[END_REF], the precise performance BPM (Beat per minute) can be computed, the initial BPM via the marked music score or musicians' guess. Different tempo of sections may occur so we split pieces by section.

Besides the fundamental event boundary fine-tuning, the correction for each module is described as follows:

• Pitch: Octave up/down is provided as the octave error take a large proportion of the errors. A pitch point or area could be selected to customize frequency or voice activity. The corrected notes help pitch curve clipping and padding in export module.

• Onset/offset: The selected points can be rectified to the closest peak to speed-up the correction. • Note segments: A pair of adjacent onset peaks and estimated offset determine note attributes. • Pitch shifts: Pitch fluctuation intervals are limited by corresponding note segments. • Other techniques: The remaining techniques, like harmonic tone, twist tone or percussion non-identifiable through the string vibration, take up a small quantity. We manually annotate them as an additional playing technique sequence. The non-attack noise like that from friction is ignored.

IV. DATASET OVERVIEW A. DATASET STATISTICS AND COMPARISON

Table 4 re-aggregates the related datasets aforementioned in Table 3 in terms of the polyphony, annotation types, playing techniques and basic materials. Among these datasets, GuitarSet [START_REF] Xi | GuitarSet: A Dataset for Guitar Transcription[END_REF] has pitch contours and discrete notes but no playing techniques. Multimodal Guitar [START_REF] Perez-Carrillo | Estimation of Guitar Fingering and Plucking Controls based on Multimodal Analysis of Motion, Audio and Musical Score[END_REF] works on the fingerings in sense of that in I.B.2). Guitar playing technique dataset [START_REF] Su | Sparse cepstral, phase codes for guitar playing technique classification[END_REF] focuses on monophonic recordings. The performed pieces in IDMT-SMT [START_REF] Kehling | Automatic tablature transcription of electric guitar recordings by estimation of score-and instrument-related parameters[END_REF] sound stiff and lack

MAPS [START_REF] Emiya | Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle[END_REF] Notes × Single notes, chords MAESTRO v3 [START_REF] Hawthorne | Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset[END_REF] Notes, velocity Pedal × Yang [START_REF] Yang | AVA: A Graphical User Interface for Automatic Vibrato and Portamento Detection and Analysis[END_REF] × Performance-level playing technique intervals and parameters

Vibrato/portamento × EEP [START_REF] Marchini | The sense of ensemble: A machine learning approach to expressive performance modelling in string quartets[END_REF] Bow MoCap data Vibrato × CBF dataset [START_REF] Wang | Adaptive time-frequency scattering for modulation recognition in music signal[END_REF], [START_REF] Wang | Playing technique recognition by joint time-frequency scattering[END_REF] × Performance-level playing technique intervals Playing techniques Multimodal Guitar [START_REF] Perez-Carrillo | Estimation of Guitar Fingering and Plucking Controls based on Multimodal Analysis of Motion, Audio and Musical Score[END_REF] × × Fingerings Guitar playing technique [START_REF] Su | Sparse cepstral, phase codes for guitar playing technique classification[END_REF] × × Multiples Single notes, playing techniques IDMT-SMT [START_REF] Kehling | Automatic tablature transcription of electric guitar recordings by estimation of score-and instrument-related parameters[END_REF] Notes/string, note-level playing technique Multiples Single notes, chords, playing techniques URMP [START_REF] Li | Creating a Multitrack Classical Music Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications[END_REF] Pitch/notes, note-level vibrato with parameters Vibrato × Marovany zither [START_REF] Cazau | Information retrieval of Marovany zither music with an original optical-based system[END_REF] Pitch/notes × Single notes GuitarSet [START_REF] Xi | GuitarSet: A Dataset for Guitar Transcription[END_REF] Pitch of annotated expression parameters. As the most massive dataset for Chinese traditional instruments, CTIS dataset [START_REF] Liang | Constructing a multimedia Chinese musical instrument database[END_REF] consists of more than 700 audio clips with playing techniques, but the absence of transcription annotation restricts its application. PipaSet has the relatively more comprehensive labels for transcription and expressive analysis tasks. During recordings, the musicians are asked to play music as naturally as possible so that the music score may not be completely followed. According to description in above sections, the detailed data types in PipaSet comprise:

• Audio and string vibration signals: Recordings from microphones and optical switches are saved in WAV format including the selected pieces in [START_REF]Grade examination committee of China Conservatory of musicNational general textbook for social art grade examination for Pipa[END_REF], [START_REF]The collection for Pipa grade examination of China[END_REF] as well as the single-note scale from each string and playing technique examples as basic audio materials.

B. ACOUSTIC AND MUSIC FEATURES

In this section, we reveal common prior knowledge of particular instruments and non-eurogenetic music under the basic materials:

• Fake nail noise: Like the plucking peak for note segmentation in string vibration signal (See Figure 6), a more crisp attack noise triggered by the fake nails also occurs in audio recording. • Non-equal temperament: From point of view of pipa construction, the fret positions often associated with pipa makers' craft probably produce an intrinsic pitch deviation from the equal temperament even though the open strings are well-tuned (See Figure 8). • Body response effect: The modulation of body response affects the amplitude distribution of harmonic series. For example, the F0 amplitude of A3 tones may be seriously weaker than those of high order partials (See figure 9). Hence, the frequency of maximal amplitude and the interval between the zero-crossing points at the two ends of a waveform element may physically introduce the octave errors for pitch detection. Meanwhile, the sympathetic resonance between body and string may distort the pitch up to a semitone at tone tail of normal plucks around F5 tones (≈ 700Hz).

The approximate inharmonicity coefficients from 1st to 4th strings computed by the approach in [START_REF] Møller | Estimation of guitar string, fret and plucking position using parametric pitch estimation[END_REF]: 2.25e-5, 1.264e-4, 4.99e-5, 6.6e-4 which are thus negligible in most of applications irrelevant to strings.

C. APPLICATIONS OF THE DATASET

In this section, we enumerate a variety of conventional and promising emerging tasks empowered by our versatile dataset from single-modal and multi-modal perspectives. First, the single-modal and intra-modal applications are present as follows:

• Annotation: The expressive music and performance generation [START_REF] Lerch | Music Performance Analysis: A Survey[END_REF]- [START_REF] Carnovalini | A Multilayered Approach to Automatic Music Generation and Expressive Performance[END_REF] appears barely studied with complex articulations of plucked string instruments and promising for the future. The higher music representations like chord and key can be recognized by the musicological toolkits like Music21 [START_REF] Cuthbert | music21: a toolkit for computer-aided musicology and symbolic music data[END_REF]. Since the repertoires originate from famous and classic collections so that the external information like the key, style, emotion, even aesthetic comments is easy fetch. The annotations have potentials to the tasks beyond the transcription and hierarchical multi-tasks [START_REF] Lerdahl | An Overview of Hierarchical Structure in Music[END_REF]- [START_REF] Panda | Novel audio features for music emotion recognition[END_REF]. • Audio: As the fundamental MIR tasks for polyphonic expressive music, multi-pitch estimation [START_REF] Duan | Multiple fundamental frequency estimation by modeling spectral peaks and non-peak regions[END_REF], [START_REF] Bittner | Deep salience representations for f0 estimation in polyphonic music[END_REF] and playing technique analysis [START_REF] Kehling | Automatic tablature transcription of electric guitar recordings by estimation of score-and instrument-related parameters[END_REF], [START_REF] Driedger | Template-Based Vibrato Analysis in Complex Music Signals[END_REF] are the most direct applications to the dataset. Higher-level music feature analysis can be realized under extended annotation. The expressive annotation-synthesis frameworks can be explored with the basic material [START_REF] Vincent | Analysis and synthesis of expressive guitar performance[END_REF]- [START_REF] Wu | MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling[END_REF].

• Video: Multi-view shots allows the visual 3D construction for robust hand pose estimation [START_REF] Khaleghi | Multi-View Video-Based 3D Hand Pose Estimation[END_REF] and performance analysis [START_REF] Kwon | Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering[END_REF]. The pitch, fret, string position on fingerboard [START_REF] Koepke | Visual pitch estimation[END_REF], [START_REF] Paleari | A multimodal approach to music transcription[END_REF], vibrato [START_REF] Li | Videobased Vibrato Detection and Analysis for Polyphonic String Music[END_REF] could be more directly perceived. The colored fake nails help to mark the finger motion and provides a way for fingering analysis. • Music sheet: The works on Optical Character Recognition (OCR) [START_REF] Rebelo | Optical music recognition: state-ofthe-art and open issues[END_REF], [START_REF] Calvo-Zaragoza | Understanding Optical Music Recognition[END_REF] for special notation or called Optical Music Recognition (OMR) and the scoreperformance alignment for non-eurogenetic music [101] are new topics. The analysis and generation of expressions including tempo and dynamics could be realized between score and annotation including the parameters of articulations [START_REF] Cancino-Chacón | Computational Modeling of Musical Expression Perspectives, Datasets, Analysis and Generation[END_REF]. Multimodality analysis has become increasingly popular in various fields [START_REF] Wang | A Comprehensive Survey on Cross-modal Retrieval[END_REF], [START_REF] Baltrusaitis | Multimodal Machine Learning: A Survey and Taxonomy[END_REF]. Video [START_REF] Ventura | Accurate analysis and visual feedback of vibrato in singing[END_REF] and score [START_REF] Abeßer | Score-informed analysis of intonation and pitch modulation in jazz solos[END_REF], [START_REF] Li | Analysis of expressive musical terms in violin using score-informed and expression-based audio features[END_REF] based works are also proposed to deal with polyphonic scenarios. The audio-visual generation for fingering and gestures [START_REF] Chen | Deep cross-modal audio visual generation[END_REF], sight-to-sound [START_REF] Koepke | Sight to Sound: An End-to-End Approach for Visual Piano Transcription[END_REF] for the motion-based music generation, audio-sheet correspondence analysis [START_REF] Dorfer | Learning audio-sheet music correspondences for score identification and offline alignment[END_REF] and cross-modal query [START_REF] Li | Query by Video: Cross-modal Music Retrieval[END_REF] can be realized through PipaSet as multimodal transformations.

V. CONCLUSION AND FUTURE WORK

To the best of our knowledge, PipaSet is the first annotated dataset for transcription and expressive analysis dedicated to Chinese traditional instrument pipa. The main contribution of this paper is to present a complete creation procedure of prototype dataset including multimodal recording and multitask annotation. The motivation of this work is to promote the development of the research in ethnomusicology and polyphonic string instruments with expressive features. Licensed under GNU general public license, the preview of datasets and the beta version of TEAS platform are opensourced respectively on Zenodo 1 on Github 2 .

Although a series of applications are introduced in section IV, some limitations in our dataset could be improved from different aspects in the future. The annotation for gestures and fingerings, e.g. termination position, plucking finger and position, inclination, can be captured from video or the other sensors like foil gauge mounted on fake nail for the plucking deflection, Time of Flight (ToF) camera for depth estimation, etc. Finally, we will extend this dataset using TEAS platform to cover more pieces of music, pipa models, players even the other plucked string instruments in the future.

ACKNOWLEDGEMENT

We greatly thank to Yuanbo Tang, M.S. from SEU, for his constructive contribution to the device deployment. We are also grateful to the virtuosic performance of Xiaoyue Zhang from Nanjing XiaoZhuang University. 1 Website's coming soon 2 Website's coming soon

FIGURE 1 :

 1 FIGURE 1: The six ledged, twenty-four fretted, four-stringed Pipa (Left) and celluloid-made fake nails (Right) used in our work.

FIGURE 2 :

 2 FIGURE 2: Pipa notation of Swan excerpt.

FIGURE 3 :

 3 FIGURE 3: The optical sensors used to capture string vibrations in our work.

FIGURE 4 :

 4 FIGURE 4: Player and audio-video equipment layout in anechoic room.

FIGURE 5 :

 5 FIGURE 5: The flowchart of TEAS used to assist the transcription and expressive annotation.

FIGURE 6 :

 6 FIGURE 6: Raw optical signal and debleeded signal of 1st string for Jasmine Flower excerpt.

  (a) Pitch detection for debleeded first string vibration signal of Jasmine Flower recordings (b) Note segmentation for debleeded first string vibration signal of Jasmine Flower recordings (c) Tremolo analysis for debleeded first string vibration signal of Ambush from Ten Sides recordings (d) Annotations overview for all tracks of Jasmine Flower

FIGURE 7 :

 7 FIGURE 7: TEAS screenshots for 2 pipa masterpieces Ambush from Ten Sides and Jasmine Flower.

FIGURE 8 :FIGURE 9 :

 89 FIGURE 8: (a) Focus on 500-1000 Hz. (b) Red curve denotes a single waveform element in time domain and 5 zeros crossing points within. (c) The triangle and correspond to the F0 and F3

10 VOLUME 4

 104 , 2021

TABLE 1 :

 1

The pitch ranges for each string under Pipa standard tuning method.

Table 2 (

 2 See Section III for more details). Harmonics, staccato, different percussive and noise techniques like string scratch, slapping and board flick, twisting (Jiao, 绞) and combining (Bing, 并) count into the 'Others' item in the table.

	Categories	Articulations	Feature parameter(s)
	Pitch	Vibrato,	Extent, rate.
	fluctuation	trilling,
		bending
	Pitch	Sliding,	Inflection point, growth
	transition	hammer-on,	rate.
		pull-off, pull,
		push
	Tremolo	Wheel,	Plucking number, rate,
		rolling,	dynamic.
		shaking
	Strum	Strumming,	Rate, strings of depar-
		arpeggio	ture and destination.
	Others	Harmonics,	-
		percussion,
		special
		noises

TABLE 2 :

 2 Articulation information in dataset.

TABLE 3 :

 3 Recording devices and music instruments of datasets discussed in this paper.

		-channel	Signle-view		Multiple Chinese instruments
	PipaSet	Dual-channel + Camera	Multi-view	Optical Switch	Pipa
		embedded microphones

TABLE 4 :

 4 Content characteristics of Datasets.

		/notes/string, beats, chords	Strumming	×
	CTIS dataset [76]	×	Multiples	Single notes, playing
				techniques
	PipaSet	Pitch/notes/string/velocity, performance-level	Multiples	Single notes, playing
		playing technique intervals and parameters		techniques

  Videos from left, middle and right views pointed to the performers are encoded by MP4 with resolution of 1080p and sampling rate of 30 fps. The pipa numbered tonic-solfa scores are scanned and saved in PDF format. All annotated features are manually corrected through audiovisual contrasts, then crossvalidated to ensure the quality by musicians, finally saved as CSV files. MIDI files with an estimated tempo are converted from annotations, the pieces of multiple sections are split apart by the different tempo.

• Video: • Sheet music: • Annotation:

  Instead of T 60 in[START_REF] Chen | Sound synthesis of the pipa based on computed timbre analysis and physical modeling[END_REF], the values of T 26 corresponding to the duration from natural transients to offsets of 5% of maximal energy remaining from 1st to 4th open strings are 0.546, 1.318, 1.4834, 1.0674 which don't always decrease with the pitch.• Tempo and rubato: Except different tempos in music sections like those in Ambush from Ten Sides, selfinterpretation[START_REF] Muller | Towards automated extraction of tempo parameters from expressive recordings[END_REF] often alters the performance speed in Chinese music.

• Time-domain envelope:

 VOLUME 4, 2021

 VOLUME 4, 2021

 VOLUME 4, 2021

 VOLUME 4, 2021

 VOLUME 4, 2021

OLIVIER ADAM was born in Paris, France, in He received M.S. degree in applied physics and did his PhD thesis on artificial neural networks. He is currently Professor at Sorbonne University, Paris, France. Specialist in signal processing and acoustics, his main research topic is the analysis of sounds emitted by cetacean species in order to describe their behaviors, interactions and habitats. He was also involved in automatic Music transcription, and was interested by analysing traditional music from Malagasy zithers. He published more than 50 research articles in scientific revues QIAO WANG received the BS, MS, and PhD degrees in mathematics from Wuhan University, China, in 1988China, in , 1994China, in , and 1997, respectively. He is currently professor of the School of Information Science and Engineering, and ShingTung Yau Center, Southeast University, China. He joined the School of Information Science and Engineering, Southeast University, Nanjing, China, in 1997, and was appointed as associate professor, in 1999, then full professor, in 2001. He was a visiting scientist with Harvard University, Cambridge, Massachusetts from January 2003 to January 2004. His main research interests include applied harmonic analysis and information theory, data analysis and image analysis in the areas of urban science, urban planning and design, clinical medicine and public health, and sports analysis. He was awarded Excellence Design by the International Society of the Built Environment, in 2019, and Best Editor Award of ICT Express, 2019.