V C Kieu

Alicia Fornés

Muriel Visani
email: muriel.visani@univ-lr.fr

Nicholas Journet
email: journet@labri.fr

Anjan Dutta
email: adutta@cvc.uab.es

The ICDAR/GREC 2013 Music Scores Competition on Staff Removal

Keywords: Competition, Music scores, Staff Removal

The first competition on music scores that was organized at ICDAR and GREC in 2011 awoke the interest of researchers, who participated both at staff removal and writer identification tasks. In this second edition, we propose a staff removal competition where we simulate old music scores. Thus, we have created a new set of images, which contain noise and 3D distortions. This paper describes the distortion methods, metrics, the participant's methods and the obtained results.

I. INTRODUCTION

Optical Music Recognition (OMR) has been an active research field for years. Many staff removal algorithms have been proposed [START_REF] Dalitz | A Comparative Study of Staff Removal Algorithms[END_REF], [START_REF] Santos Cardoso | Staff Detection with Stable Paths[END_REF] as a first step in the OMR systems. However, there is still room for research, especially in the case of degraded music scores. At ICDAR [START_REF] Fornés | The ICDAR 2011 Music Scores Competition: Staff Removal and Writer Identification[END_REF] and GREC 2011, we organized the first edition of the music scores competition. For the staff removal task, we created several sets of distorted images (each set corresponding to a different kind of distortion) and compared the robustness of the participants' methods. After GREC 2011, we extended the staff removal competition [START_REF]The 2012 Music Scores Competitions: Staff Removal and Writer Identification[END_REF] by generating a new set of images combining different distortions at different levels. The results demonstrated that most methods significantly decrease the performance when coping with a combination of distortions.

In this second edition of the competition, we have generated new images that emulate typical degradations appearing in old handwritten documents. Two types of degradations (local noise and 3D distortions) have been applied on the 1000 images from the original CVC-MUSCIMA database [START_REF]CVC-MUSCIMA: A Ground Truth of Handwritten Music Score Images for Writer Identification and Staff Removal[END_REF].

The rest of the paper is organized as follows. Firstly, we describe the degradation models and the dataset used for the competition. Secondly, we present the participants' methods, the evaluation metrics, and we analyze the results.

II. ICDAR/GREC 2013 DATABASE

For comparing the robustness of the different participants' staff removal algorithms, we have applied the 3D distortion and the local noise degradation models described hereafter to the original CVC-MUSCIMA database [START_REF]CVC-MUSCIMA: A Ground Truth of Handwritten Music Score Images for Writer Identification and Staff Removal[END_REF], which consists of 1000 music sheets written by 50 different musicians.

A. Degradation Models 1) 3D Degradation Model:

This degradation model aims at mimicking some challenging distortions for staff removal algorithms, such as skews, curvatures and rotations. Differing from the 2D model used for GREC 2011, our new 3D model [START_REF] Kieu | Semisynthetic Document Image Generation Using Texture Mapping on Scanned 3D Document Shapes[END_REF] generates much more realistic images containing dents, small folds, torns. . . This 3D degradation model can distort the staff lines, making their detection and removal more difficult. It is based on 3D meshes and texture coordinate generation. The main idea is that we get multiple 3D meshes of old document pages using real ancient documents and a 3D scanner. Then, we wrap any 2D image on these meshes using some wrapping functions which are specifically adapted to document images.

2) Local Noise Model: Some old documents' defects such as ink splotches and white specks or streaks might lead for instance to disconnections of the staff lines or to the addition of dark specks connected to a staff line which can be confused with musical symbols. In order to simulate such degradations, which are very challenging for staff removal algorithms, we apply our local noise model described in [START_REF] Kieu | A Character Degradation Model for Grayscale Ancient Document Images[END_REF]. It consists in three main steps. Firstly, the "seed-points" (i.e. the centres of local noise regions) are selected so that they are more likely to appear near the foreground pixels (obtained by binarizing the input grayscale image). Then, we add arbitrary shaped greylevel specks (in our case, the shape is an ellipse). The greylevel values of the pixels inside the noise regions are modified so as to obtain realistic looking bright and dark specks.

B. ICDAR/GREC 2013 Degraded Database

For the ICDAR/GREC 2013 staff removal competition, we generate a semi-synthetic database by applying the two degradation models presented above to the 1000 images from the original CVC-MUSCIMA database. The obtained degraded database consists in 6000 images: 4000 images for training and 2000 images for testing the staff removal algorithms.

1) Training Set:

The training set consists in 4000 semisynthetic images generated from 667 out of the 1000 original images in the CVC-MUSCIMA database. This training set is split into three subsets corresponding to different degradation types and levels of degradation, as described hereafter:

TrainingSubset1 contains 1000 images generated using the 3D distortion model (c.f. sub-section II-A1) and two different meshes. The first mesh contains essentially a perspective distortion due to the scanning of a thick and bound page, while the second mesh has many small curves, folds and concaves. Both meshes are applied to the 667 original images. Then, 1000 images (500 images per mesh) are randomly selected from those 2 × 667 = 1334 degraded images.

TrainingSubset2 contains 1000 images generated with three different levels (i.e low, medium, and high levels) of local noise. The different levels of noise are obtained by varying the number of seed-points and the average size of the noise regions (see sub-section II-A2).

TrainingSubset3 (see Fig. 1) contains 2000 images generated using both the 3D distortion and the local noise model. We obtain six different levels of degradation (the two meshes used for TrainingSubset1 × the three levels of distortion used for TrainingSubset2).

For each image in the training set, we provide to the participants of the competition its grey and binary version and the associated ground-truth, under the form of its binary staff-less version (such images containing only binarized music symbols but no staff lines), as illustrated in Fig. 2.

2) Test Set: The test set consists in 2000 semi-synthetic images generated from the 333 original images from the CVC-MUSCIMA database that are not used for the training set.

TestSubset1 contains 500 images generated using the 3D distortion model. Two meshes -distinct from the ones used in the training set -are used. 500 images (250 for each mesh) are randomly selected among the 2x333=666 degraded images.

TestSubset2 contains 500 images generated using three different levels of local noise, using the same values of the parameters as in TrainingSubset2.

TestSubset3 contains 1000 images equally distributed between six different levels of degradation using both 3D distortion models (and the same 2 meshes as in TestSubset1), and the same 3 different levels of local noise as in TrainingSubset2.

For each image in the test set, we provide to the participants of the competition its gray and binary version. The groundtruth associated to the test set, consisting of binary staff-less images, was made public after the contest.

III. EXPERIMENTAL PROTOCOL AND RESULTS

The competition was organized as follows. First, we provided to the participants (see section III-A) the training set and its grount-truth for training their algorithms. 46 days later, we sent them the test set. They returned us their outputs as binary staff-less images within 23 days. We compared their outputs to the test set ground-truth using the metrics presented in section III-B, obtaining the results presented in section III-C.

A. Participants Information 1) TAU-bin:

The method was submitted by Oleg Dobkin from the Tel-Aviv University, Israel. It is based on the Fujinaga's method [START_REF] Fujinaga | Adaptive Optical Music Recognition[END_REF]. The method is based on an estimation of the staff-line height and the staff-space height and vertical runlengths. It consists in removing black pixels which are part of a short vertical run of black pixels (these pixels being more likely to be part of a staff line).

2) NUS-bin: This method [START_REF] Su | An Effective Staff Detection and Removal Technique for Musical Documents[END_REF] was submitted by Bolan Su (National University of Singapore), Umapada Pal (Indian Statistical Institute, Kolkata, India) and Chew-Lim Tan (National University of Singapore). It predicts the lines' direction and fits an approximate staff line curve.

3) NUASi: Christoph Dalitz and Andreas Kitzig, from the Niederrhein University of Applied Sciences -Institute for Pattern Recognition (iPattern), Krefeld, Germany, submitted two different methods [START_REF] Dalitz | A Comparative Study of Staff Removal Algorithms[END_REF] for which the source code is available at http://music-staves.sourceforge.net/. In the NUASi-bin-lin method, all short vertical runs are removed from the skeleton image, and a function filters the staffline pixels that belong to a crossing symbol. The NUASi-bin-skel method is a refinement of the previous method where the skeleton of the staff line is considered locally, at branching and corner points so as to remove more efficiently the crossing music symbols and to join staff line segments corresponding to the same staff line.

4) LRDE:

Thierry Géraud, from the EPITA Research and Development Laboratory (LRDE), Paris, France, submitted two methods described in http//www.lrde.epita.fr/cgibin/twiki/view/Olena/Icdar2013Score. These methods rely on morphological operators and can handle respectively binary images (in its version LRDE-bin) and grayscale images (in its version LRDE-gray using Sauvola's binarization).

5) INESC:

Ana Rebelo and Jaime S. Cardoso (Universidade do Porto, Portugal) propose two graph-based methods [START_REF] Santos Cardoso | Staff Detection with Stable Paths[END_REF]. In the INESC-bin method, a graph is created from predetected strong staff-pixels (SSPs). Some SSPs are labeled as staff-line pixels, according to heuristic rules. Then, a global optimization process gives the final staff lines. The INESCgray method applies a sigmoïd-based weight function that favors the luminance levels of staff. Then, the image is binarized and the INESC-bin method is applied.

B. Measures Used for Performance Comparison

At the pixel level, the staff removal problem is considered as a two-class classification problem. For each test subset and each level of noise, we compare the participant's images to their corresponding ground-truth. We compute the number of True Positive pixels (TP, pixels correctly classified as staff lines), True Negative pixels (TN, pixels correctly classified as non-staff lines), False Positive pixels (FP, pixels wrongly classified as staff lines) and False Negative pixels (FN, pixels wrongly classified as non-staff lines). Then, from these measures, we compute the Accuracy (also called Classification Rate), Precision (also called Positive Predictive Value), Recall (also called True Positive Rate or sensitivity), F-measure and Specificity (or True Negative Rate).

Since the first step of a staff removal system is usually the detection of the staff lines, the overall performance highly depends on the accuracy of this preliminary staff detection. It may occur that one system obtains very good results but "misses" (rejects) many images containing staff lines. Therefore, for each participants' method, for each test subset and each level of noise, we provide the number of rejected pages and the average values of the 5 measures described above. If there are some missing pages, the average measures are computed 1) only on the detected images and 2) taking into account the rejected pages (every staff line pixel being considered as a False Negative and every non-staff line pixel being considered as a False Positive).

C. Performance Comparison

Table I presents the results obtained by the participants. We compare these results to those obtained by a baseline algorithm proposed by Dutta et al. [START_REF] Dutta | An Efficient Staff Removal Approach from Printed Musical Documents[END_REF] and based on the analysis of neighboring components. For each line, the best method is in bold. Since the Precision is higher in some methods but with a lower Recall, we select the winners according to the Accuracy and F-measure metrics. INESC-bin is the winner on the TestSubset2 containing local noise, while LRDE-bin is the winner on the TestSubsets 1 and 3, containing respectively 3D distortions and a combination of 3D distortions and local noise. It must also be noticed that most methods (including the baseline method) obtain quite similar performances.

We can also analyze the scores according to the kind and level of degradations. Concerning the 3D distortion (TestSub-set1), most methods seem less robust to perspective deformation defects (Mesh 1) than to the presence of small curves and folds (Mesh 2). In addition, the precision scores of every participants decrease (on average of 13%) when the local noise in TestSubset2 is getting higher. Therefore, all the participants' methods are sensitive to the local noise degradation. The tests carried out with images from TestSubset3, generated by combining local noise and 3D distortions confirm that the results decrease when the level of degradation is important.

IV. CONCLUSION

The second music scores competition on staff removal held in ICDAR and GREC 2013 has raised a great interest from the research community, with 8 participant methods. The submitted methods have obtained very satisfying performance, although most methods significantly decrease their performance when dealing with a higher level of degradation. The presence of both sources of degradation (3D distortion + local noise) is especially challenging. We hope that our semisynthetic database will become a benchmark for the research on handwritten music scores in the near future.

Fig. 1 .

 1 Fig. 1. From left to right: original image from the CVC-MUSCIMA database, and two images from TrainingSubset3 of the ICDAR/GREC 2013 database generated using a high level of local noise and (respectively) mesh#1 and mesh#2.

Fig. 2 .

 2 Fig. 2. From left to right: an image from TrainingSubset3, its binary version and its binary staff-less version (ground-truth)

TABLE I .

 I COMPETITION RESULTS FOR THE 5 MEASURES (IN %) FOR EACH TEST SUBSET AND EACH DEGRADATION LEVEL. WHEN NEEDED, WE GIVE THE NUMBER # OF REJECTED IMAGES, AND THE VALUES OF THE MEASURES COMPUTED WITH AND WITHOUT REJECTION.

	Deformation	Level	Measure	TAU-bin	NUS-bin	NUASi-bin-lin	NUASi-bin-skel	LRDE bin	LRDE gray	INESC bin	INESC gray	Baseline
			Precision	75.51	98.75	99.05	98.58	98.89	87.26	99.76	32.50	98.62
	TestSubset1: 3D distortion	Mesh 1 (M1)	Recall F-Measure Specificity Accuracy Precision	96.32 84.65 98.81 98.721 82.22	52.80 68.81 99.97 98.25 99.50	#2 89.90 (89.77) 94.25 (94.18) 99.96 (99.96) 99.60 (99.60) 99.70	#3 90.26 (90.03) 94.24 (94.11) 99.95 (99.95) 99.60 (99.60) 99.39	96.19 97.52 97.52 99.82 99.52	98.41 92.50 99.45 99.42 86.59	85.41 92.03 99.99 99.46 99.90	50.91 39.67 95.97 94.32 34.36	79.86 88.26 99.95 99.22 99.29
		Mesh 2 (M2)	Recall F-Measure	91.90 86.79	55.05 70.88	#4 92.07 (91.38) 95.73 (95.36)	#2 89.63 (89.36) 94.26 (94.11)	96.39 97.93	97.76 91.83	76.33 86.54	40.85 37.33	75.47 85.76
			Specificity	99.26	99.99	99.98 (99.99)	99.97 (99.97)	99.98	99.44	99.99	97.12	99.98
			Accuracy	99.01	98.39	99.71 (99.68)	99.61 (99.60)	99.86	99.38	99.16	95.12	99.10
			Precision	65.71	95.37	98.41	97.28	95.54	53.22	97.63	38.81	95.65
		High (H)	Recall F-Measure Specificity	97.01 78.35 98.59	92.27 93.79 99.87	90.81 94.46 99.95	89.35 93.15 99.93	96.65 96.09 99.87	98.58 69.12 97.58	96.62 97.13 99.93	79.35 52.13 96.51	96.53 96.09 99.87
			Accuracy	98.55	99.67	99.71	99.64	99.79	97.61	99.85	96.05	99.78
			Precision	69.30	97.82	99.24	98.38	97.50	68.10	98.95	39.61	97.26
	TestSubset2: Local Noise	Medium (M)	Recall F-Measure	97.34 80.96	96.97 97.39	#3 91.94 (91.41) 95.45 (95.16)	#4 90.56 (89.80) 94.31 (93.90)	97.13 97.32	98.77 80.62	97.19 98.07	74.83 51.81	97.10 97.18
			Specificity	98.71	99.93	99.97 (99.97)	99.95 (99.95)	99.92	98.61	99.96	96.58	99.91
			Accuracy	98.67	99.85	99.75 (99.73)	99.68 (99.66)	99.84	98.62	99.89	95.96	99.83
			Precision	77.07	98.56	99.25	98.07	97.89	80.65	99.42	40.13	98.52
		Low (L)	Recall F-Measure Specificity	96.88 85.85 99.12	96.58 97.56 99.95	90.48 94.66 99.97	90.17 93.95 99.94	96.47 97.17 99.93	98.47 88.67 99.28	96.52 97.95 99.98	75.48 52.40 96.59	96.45 97.47 99.95
			Accuracy	99.06	99.86	99.70	99.66	99.84	99.26	99.88	95.98	99.85
			Precision	66.01	94.31	96.88	96.37	96.14	56.19	97.63	31.70	96.41
		H + M1	Recall	96.35	50.00	88.03	87.93	96.13	98.59	85.79	#17 55.21 (50.48)	85.98
			F-Measure	78.34	65.35	92.25	91.96	96.14	71.58	91.33	40.27 (38.94)	90.90
			Specificity	98.30	99.89	99.90	99.88	99.86	97.37	99.92	95.93 (96.27)	99.89
			Accuracy	98.24	98.25	99.51	99.49	99.74	97.41	99.46	94.58 (94.76)	99.43
			Precision	73.40	97.50	98.55	98.07	97.61	57.18	98.35	33.11	97.62
		H + M2	Recall	92.42	53.56	#4 90.99 (90.32)	#3 89.15 (88.68)	96.66	98.00	75.17	#12 42.15 (39.19)	81.26
			F-Measure	81.82	69.14	94.62 (94.25)	93.40 (93.14)	97.13	72.22	85.22	37.09 (35.90)	88.69
			Specificity	98.86	99.95	99.95 (99.95)	99.94 (99.94)	99.92	97.51	99.95	97.11 (97.31)	99.93
			Accuracy	98.65	98.43	99.66 (99.64)	99.59 (99.57)	99.81	97.53	99.14	95.31 (95.41)	99.32
	TestSubset3: 3D distortion + Local Noise	M + M1	Precision Recall F-Measure Specificity Accuracy	69.26 96.44 80.62 98.47 98.406	95.45 49.07 64.81 99.91 98.168	97.52 89.15 93.15 99.91 99.549	96.93 87.98 92.24 99.90 99.491	97.11 95.98 96.54 99.89 99.763	67.44 98.46 80.05 98.30 98.312	98.51 85.63 91.62 99.95 99.461	32.34 #16 53.52 (48.76) 40.31 (38.88) 96.01 (96.36) 94.556 (94.730)	97.29 85.96 91.27 99.91 99.43
			Precision	77.50	98.39	99.02	98.53	98.42	68.09	99.06	33.76	98.35
		M + M2	Recall	91.83	53.47	#4 91.57 (90.85)	#3 88.43 (87.94)	96.52	97.92	75.21	#10 41.64 (39.13)	81.08
			F-Measure	84.05	69.29	95.15 (94.76)	93.20 (92.93)	97.46	80.27	85.50	37.29 (36.25)	88.88
			Specificity	99.06	99.96	99.96 (99.96)	99.95 (99.95)	99.94	98.38	99.97	97.12 (97.30)	99.95
			Accuracy	98.87	98.39	99.68 (99.66)	99.56 (99.54)	99.83	98.37	99.13	95.24 (95.32)	99.31
			Precision	73.28	96.75	98.06	97.50	97.92	79.32	99.14	32.77	97.96
		L + M1	Recall	96.38	50.22	88.96	88.74	95.92	98.38	85.48	#17 53.83 (48.83)	85.23
			F-Measure	83.26	66.12	93.29	92.92	96.91	87.83	91.80	40.74 (39.22)	91.15
			Specificity	98.70	99.93	99.93	99.91	99.92	99.05	99.97	95.93 (96.30)	99.93
			Accuracy	98.62	98.17	99.55	99.52	99.78	99.03	99.46	94.44 (94.62)	99.41
			Precision	80.17	99.00	99.39	98.94	99.02	78.81	99.53	34.31	98.84
		L + M2	Recall	91.98	54.01	#4 91.97 (91.22)	#3 89.14 (88.63)	96.46	97.85	75.18	#8 41.34 (39.08)	80.14
			F-Measure	85.67	69.89	95.54 (95.13)	93.78 (93.50)	97.72	87.30	85.66	37.50 (36.54)	88.52
			Specificity	99.17	99.98	99.97 (99.98)	99.96 (99.96)	99.96	99.04	99.98	97.13 (97.28)	99.96
			Accuracy	98.92	98.37	99.70 (99.67)	99.59 (99.57)	99.84	99.01	99.12	95.18 (95.25)	99.27
	Total rejected images	#0	#0	#21	#18	#0	#0	#0	#80	#0

ACKNOWLEDGEMENTS

This research was partially funded by the French National Research Agency (ANR) via the DIGIDOC project, and the spanish projects TIN2009-14633-C03-03 and TIN2012-37475-C02-02. We would also like to thank Anjan Dutta for providing the baseline results.


Muriel Visani
email: muriel.visani@univ-lr.fr

V C Kieu
email: vkieu@labri.fr

Alicia Fornés
email: afornes@cvc.uab.es

Nicholas Journet
email: journet@labri.fr

The ICDAR 2013 Music Scores Competition: Staff Removal

Keywords: Competition, Music Scores, Staff Removal

The first competition on music scores that was organized at ICDAR in 2011 awoke the interest of researchers, who participated both at staff removal and writer identification tasks. In this second edition, we focus on the staff removal task and simulate a real case scenario: old music scores. For this purpose, we have generated a new set of images using two kinds of degradations: local noise and 3D distortions. This paper describes the dataset, distortion methods, evaluation metrics, the participant's methods and the obtained results.

I. INTRODUCTION

The recognition of music scores has been an active research field for decades [START_REF] Blostein | Structured Document Image Analysis[END_REF], [START_REF] Rebelo | Optical Music Recognition: State-of-the-Art and Open Issues[END_REF]. Many researchers have proposed staff removal algorithms in order to improve the recognition of music symbols [START_REF] Dalitz | A Comparative Study of Staff Removal Algorithms[END_REF], [START_REF] Santos Cardoso | Staff Detection with Stable Paths[END_REF]. However, the staff removal task cannot be considered as a solved problem, especially when dealing with ancient/degraded handwritten music scores.

At ICDAR [START_REF] Fornés | The ICDAR 2011 Music Scores Competition: Staff Removal and Writer Identification[END_REF] and GREC 2011, we organized the first edition of the music scores competition. For the staff removal task, we generated distorted images in order to test the robustness of the staff removal algorithms. Thus, we created several sets of distorted images, where each set corresponded to a different kind of distortion (e.g. Kanungo noise, rotation, curvature, staff line interruption, typeset emulation, staff line thickness variation, white speckles, etc.).

After GREC 2011, we extended the staff removal competition [START_REF]The 2012 Music Scores Competitions: Staff Removal and Writer Identification[END_REF]. The goal was to simulate a real scenario, in which music scores usually contain more than a single kind of distortion. For this purpose, we generated a new set of images, by combining different distortions at different levels. Not surprisingly, the new results demonstrated that for most methods, the performances were significantly decreased when coping with a combination of distortions.

By organizing a second edition of this competition at ICDAR/GREC 2013, we aimed at fostering the interest of researchers we attracted during the preceding edition. For this second edition, we have generated new images that emulate typical degradations appearing in old and degraded handwritten documents. In this new edition, two types of degradations have been considered: local noise and 3D distortions.

The rest of the paper is organized as follows. First, we describe the original dataset, the degradation models, the training and test sets and the evaluation metrics. Afterwards, we present the participants' methods and analyze the results.

II. THE ICDAR/GREC 2013 MUSIC SCORES DATABASE

For the music scores competition on staff removal, we generated a semi-synthetic database (presented in section II-C) using the original CVC-MUSCIMA database (see section II-A) and two degradation models we introduced recently (see section II-B).

A. Original CVC-MUSCIMA database

The original CVC-MUSCIMA1 database [START_REF]CVC-MUSCIMA: a Ground Truth of Handwritten Music Score Images for Writer Identification and Staff Removal[END_REF] consists of handwritten music score images. An extract of this database can be seen in Fig. 1(a). The dataset contains 1,000 music sheets written by 50 different musicians. Each musician has transcribed exactly the same 20 music pages, using the same pen and the same kind of music paper. The 20 selected music sheets contain monophonic and polyphonic music.

B. Degradation Models 1) 3D Degradation Model:

Since the geometric distortions such as skews and curvatures generated by 2D distortion models are challenging for detecting staffs, we used them for the ICDAR/GREC 2011 staff removal competition [START_REF] Fornés | The ICDAR 2011 Music Scores Competition: Staff Removal and Writer Identification[END_REF], [START_REF]The 2012 Music Scores Competitions: Staff Removal and Writer Identification[END_REF]. However, these 2D distortions models cannot reproduce the geometric distortions commonly encountered in real old documents such as dents, small folds, torns. . . . In this 2013 edition of the ICDAR/GREC staff removal competition, we use the 3D distortion model we introduced in [START_REF] Kieu | Semisynthetic Document Image Generation Using Texture Mapping on Scanned 3D Document Shapes[END_REF]. It can generate more realistic and challenging distortions of the staff lines, making their detection and removal more difficult. This 3D degradation model is based on 3D meshes and texture coordinate generation. The main idea is to wrap any 2D (flat) image of a document on a 3D mesh acquired by scanning a non-flat old document using a 3D scanner. The wrapping functions we use are specifically adapted to document images. In our case, we wrap the original images of the MUSCIMA database on different 3D meshes we previously acquired.

2) Local Noise Model:

The local noise model we introduced in [START_REF] Kieu | A Character Degradation Model for Grayscale Ancient Document Images[END_REF] is specifically designed to mimic some old documents' defects such as ink splotches and white specks or streaks. These defects might break the connectivity of strokes or, inversely, add a connection between separated strokes. In our case, the application of this kind of degradation to the MUSCIMA database can lead to disconnections of the staff lines, or the addition of dark specks connected to a staff line. In the latter case, the dark specks might be confused with musical symbols. More generally, local noise can lead to many degradations which are very challenging for staff removal algorithms.

As detailed in [START_REF] Kieu | A Character Degradation Model for Grayscale Ancient Document Images[END_REF], the local noise is generated in three main steps. Firstly, the "seed-points" (i.e. the centres of local noise regions) are selected so that they are more likely to appear near the foreground pixels (obtained by binarizing the input grayscale image). Then, we add arbitrary shaped grey-level specks (in our case the shape is an ellipse). The grey-level values of the pixels inside the noise regions are modified so as to obtain realistic looking bright and dark specks.

C. The ICDAR/GREC 2013 Database

The ICDAR/GREC 2013 database consists in 6000 images (4000 for training, 2000 for testing), as detailed hereafter:

1) Training Set:

The training set consists in 4000 semisynthetic images generated from 667 out of the 1000 original images in the CVC-MUSCIMA database. It is split into three subsets corresponding to different degradation types:

• TrainingSubset1 contains 1000 images generated using the 3D distortion model (c.f. sub-section II-B1) and two different meshes (with 500 images per mesh).

As it can be seen in Fig. 1, the first mesh contains essentially a perspective distortion due to the scanning of a thick and bound page, while the second mesh has many small curves, folds and concaves.

• TrainingSubset2 (see Fig. 2) contains 1000 images generated with three different levels of local noise, where a higher level of noise corresponds to an increasing number of seed points and larger noise regions (see section II-B2).

• TrainingSubset3 (see Fig. 3) contains 2000 images from six degradation levels obtained by combining the same three levels of local noise and 3D distortions (with the two 3D meshes used in TrainingSubset1).

For each image in the training set, we provide to the participants of the competition its grey and binary versions and the associated ground-truth, under the form of its binary staff-less version, as illustrated in Fig. 4.

2) Test Set:

The test set consists in 2000 semi-synthetic images generated using the 333 original images from the CVC-MUSCIMA database that are not used for the training set.

• TestSubset1 contains 500 images generated using the 3D distortion degradation model and two meshes (distinct from the ones used in the training set).

• TestSubset2 contains 500 images generated using the same three levels of local noise as in TrainingSubset2.

• TestSubset3 contains 1000 images equally distributed between six different levels of degradation obtained by combining the same three levels of local noise and 3D distortions (with the 2 meshes used in TestSubset1).

For each image in the test set, we provide to the participants of the competition its gray and binary versions. The groundtruth associated to the test set, consisting of binary staff-less images, was made public after the contest.

III. EXPERIMENTAL PROTOCOL AND RESULTS

The competition was organized in three steps. First, we provided to the participants (see section III-A) the training set and its grount-truth for training their algorithms. Second, we sent them the test set 46 days later. Third, they returned us their outputs as binary staff-less images within 23 days. Further, we compared their outputs to the ground-truth of the test set using the evaluation measures in section III-C. The obtained results are confronted to the results of a baseline method (see section III-B) and analyzed in section III-D.

A. Participants Information

In this section, we briefly describe the methods used by the participants for the ICDAR/GREC2013 competition. Methods 1-3 work on binary images (in that case the participants used the binary versions we provided for the competition), while methods 4-5 can handle both binary and grayscale images.

1) TAU-bin:

This method was submitted by Oleg Dobkin from the Tel-Aviv University, Israel. It is based in the Fujinaga's method [START_REF] Fujinaga | Adaptive Optical Music Recognition[END_REF]. First, the staffline height and staffspace height are estimated using vertical scans. Then, the vertical black runs which are longer than the staffspace height are removed. Afterwards, the music page is deskewed, and the staff lines are located using a projection on the y-axis. Finally, the staff lines are removed using masks.

2) NUS-bin: This method was submitted by Bolan Su (National University of Singapore), Umapada Pal (Indian Statistical Institute, Kolkata, India) and Chew-Lim Tan (National University of Singapore). The method, detailed in [START_REF] Su | An Effective Staff Detection and Removal Technique for Musical Documents[END_REF], first estimates the staffline height and staffspace height using the vertical run length histogram. These estimated values are used to predict the lines' direction and fit an approximate staff line curve for each line. Then, the fitted staff line curve is used to identify the exact location of staff lines, and those pixels belonging to these staff lines are removed.

3) NUASi: Christoph Dalitz and Andreas Kitzig, from the Niederrhein University of Applied Sciences (iPattern Institute), Krefeld, Germany, submitted two different methods described shortly hereafter:

• NUASi-bin-lin: This method is described in Section II of [START_REF] Dalitz | A Comparative Study of Staff Removal Algorithms[END_REF]. First, the staffline height is estimated as the most frequent black vertical run length. Then, the skeleton of the staff lines is extracted, and all vertical foreground runs shorter than 2* staffline height are removed. The source code is available at http://music-staves.sourceforge.net/ (class Mu-sicStaves linetracking)

• NUASi-bin-skel: This method, detailed in the Section III.D of [START_REF] Dalitz | A Comparative Study of Staff Removal Algorithms[END_REF], first splits the skeleton of the staff lines at branching and corner points. Each segment is considered as a staff line segment if it satisfies some heuristic rules. Then, two staff segments are horizontally linked if their extrapolations from the end points with the least square fitted angle are closer than staffline height/2. To check for the false positives, non-staff segments which have the same splitting point with a staff segment are extrapolated by a parametric parabola. If the staff segment is tangent with the • LRDE-bin: This method relies on mathematical morphological operators. First, a permissive hit-or-miss with a horizontal line pattern as structuring element extracts some horizontal chunks. Second, a horizontal median filter cleans up the result, and a dilation operation is applied using a horizontal neighbourhood in order to enlarge the connected components. A binary mask is obtained thanks to a morphological closing with a rectangular structuring element. Last, a vertical median filter, applied inside the largest components of this mask, removes the staff lines.

• LRDE-gray: After removing the image border, Sauvola's binarization and a dilation using a horizontal neighbourhood are applied. That image serves as a mask in which a two-level thresholding with hysteresis of the original image is applied. Then, some spurious horizontal parts of staff are erased.

5) INESC:

Ana Rebelo and Jaime S. Cardoso (INESC Porto and Universidade do Porto) submitted the following methods (binary and grayscale versions) based on graphs of Strong Staff-Pixels (SSP: pixels with a high probability of belonging to a staff line):

• INESC-bin: First, the staffline height and staffspace height are estimated by the method presented in [START_REF] Cardoso | Robust Staffline Thickness and Distance Estimation in Binary and Gray-Level Music Scores[END_REF]. Then, all the pixels of the black runs of staffline height pixels followed or preceded by a white run of staffspace height pixels are set as the SSPs.

To decide if a SSP belongs to a staff line, the image grid is considered as a graph with pixels as nodes, and arcs connecting neighbouring pixels. Then, SSPs are classified as staff line pixels acording to some heuristic rules. Then, the groups of 5 staff lines are located among the shortest paths by using a global optimization process on the graph (see [START_REF] Santos Cardoso | Staff Detection with Stable Paths[END_REF] for details).

• INESC-gray: It applies a sigmoid-based weight function that favors the luminance levels of staff. Then, the image is binarized and the previous method is applied.

B. Baseline

For comparison purposes, we have computed some baseline results using the staff removal method proposed by Dutta et al. [START_REF] Dutta | An Efficient Staff Removal Approach from Printed Musical Documents[END_REF], which is based on the analysis of neighbouring components. Basically, it assumes that a staff line candidate segment is a horizontal linkage of vertical black runs with uniform height. Then, neighbouring properties (e.g. proximity, orientation) are used to validate or discard these segments.

C. Measures Used for Performance Comparison

At the pixel level, the staff removal problem can be seen as a two-class classification task. We compare the participant's images to the test set ground-truth (for each level of degradation), and we compute the number TP of True Positive pixels (pixels correctly classified as staff lines), the number FP of False Positives (wrongly classified as staff lines) and the number FN of False Negatives (wrongly classified as nonstaff lines). Then, we compute the F-measure F M :

F M = 2×
precision × recall precision + recall = 2×

T P T P +F P × T P T P +F N T P T P +F P + T P T P +F N

(1) Four additional measures were computed and analyzed in [START_REF] Kieu | The ICDAR/GREC 2013 Music Scores Competition on Staff Removal[END_REF] but are not detailed here for space reasons.

Since the first step of a staff removal system is usually the detection of the staff lines, the overall performance highly depends on the accuracy of this preliminary staff detection. Indeed, if the staff line system is not detected (i.e. the image is rejected by the system), then the staff lines cannot be removed. Therefore, we compute the F-measures with rejection, i.e. for any rejected image, we consider that every staff line pixel is a False Negative and every non-staff line pixel is a True Negative (correctly classified as a non-staff line pixel). Furthermore, we also provide the number of rejected pages.

D. Performance Comparison

Figure 5 presents the F-measures obtained by the eight participant methods (see section III-A), and the baseline method (see section III-B). These results were computed from each of the 3 Test Subsets and 11 levels of degradation (see section II-C). For more details, please refer to [START_REF] Kieu | The ICDAR/GREC 2013 Music Scores Competition on Staff Removal[END_REF].

The numbers on top of the bars correspond to the number of rejected images for a given method and a given level of degradation of each TestSubset. When no number is given, it means that the corresponding method does not reject any image on the corresponding subset. From these numbers, we can see that the NUASI-bin-lin and NUASI-bin-skel methods often reject some images (especially in the presence of 3D distortion and Mesh #2), but in a relatively low number (respectively 21 and 18 in total). On the other hand, the INESC-gray method rejects a higher number of images (80 in total) in the presence of a combination of 3D distortion and local noise. This is certainly due to the weighting function or binarization, as its binary version INESC-bin does not reject any image.

Concerning the 3D distortion (TestSubsets 1 and 3), most methods are less robust to perspective deformation (Mesh #1) than to the presence of small curves and folds (Mesh #2), except LRDE-gray, INESC-bin and INESC-gray which perform better in the presence of Mesh #1.

In the presence of local noise, the average F-measure (over the nine methods) on TestSubset2 drops of almost 4% when the level of noise increases from Low to High.

When combining local noise and 3D distortions (TestSub-set3), the robustness of most methods drops drastically. Indeed, when mixing a high level of local noise and 3D distortions, the average F-measure (over the nine methods) drops of almost 6% compared to a high level of local noise only (for both meshes), and it decreases of respectively 3.88% and 3.22%, compared to 3D distortions with Meshes #1 and #2.

From Fig. 5, we can conclude that INESC-bin is the best on TestSubset2 containing local noise, while LRDE-bin is the winner on TestSubsets 1 and 3, containing respectively 3D distortions and a combination of 3D distortions and local noise.

IV. CONCLUSION

The second music scores competition on staff removal held in ICDAR2013 has raised a great interest from the research community, with 8 participant methods in the competition. For this competition, we generated a database of 6000 semi-synthetic images using the 1000 images from the CVC-MUSCIMA database and two models of degradation specifically designed to mimic the defects that can be seen in historical documents. This database contains 3 subsets both for training and testing, with separate and combined degradations, and in total 11 different levels of degradation. The performances of the 8 methods proposed by the 5 participants are analyzed and compared to a baseline method relying on the analysis of neighbouring connected components.

The methods submitted by all participants have obtained satisfying performance, even though the degradations in the proposed images are severe. But, most methods significantly decrease their performance when dealing with a higher level of degradation, especially when combining 3D distorsions and local noise. The semi-synthetic ICDAR/GREC 2013 database is now available on the internet and labelled with different types and levels of degradation for both the training set and the test set; we hope the community will adopt it as a benchmark database for the research on handwritten music scores.

2 Fig. 1 .Fig. 2 .Fig. 3 . 2 Fig. 4 .

 212324 Fig. 1. From left to right: original (real) image and two semi-synthetic images from the TrainingSubset1 generated from this image using different 3D meshes

Fig. 5 .

 5 Fig. 5. F-measures of the eight participant methods (plus the baseline method) on the 3 Test Subsets and 11 levels of degradation.

Available at http://www.cvc.uab.es/cvcmuscima/

ACKNOWLEDGEMENTS

This research was partially funded by the French National Research Agency (ANR) via the DIGIDOC project, the spanish projects TIN2009-14633-C03-03 and TIN2012-37475-C02-02. We thank Anjan Dutta for providing the baseline results.


V C Kieu

Nicholas Journet

Muriel Visani

Rémy Mullot
email: remy.mullot@univ-lr.fr

Jean Phillipe Domenger
email: domenger@labri.fr

Semi-synthetic Document Image Generation Using Texture Mapping on Scanned 3D Document Shapes

Keywords: distortion model, synthetic document image, performance evaluation

 to generate the 6000 semi-synthetic degraded images of the music score removal staff line competition of ICDAR 2013 1 .

I. INTRODUCTION

The performance of document analysis and recognition methods depends on the size and the quality of the training data, no matter whether the data is obtained from real life or synthesized [START_REF] Baird | The State of the Art of Document Image Degradation Modeling[END_REF]. Using synthetic data (in addition to real life data) has some advantages: rapid generation process, low cost and high accuracy in the ground-truth. Therefore, degradation models in general and geometric distortion models in particular are widely used to generate synthetic images. These images are then used to evaluate or compare the robustness of various document analysis methods (i.e. for symbol recognition & spotting system evaluation in [START_REF] Delalandre | Generation of Synthetic Documents for Performance Evaluation of Symbol Recognition & Spotting Systems[END_REF], handwriting recognition algorithm in [START_REF] Mori | Generating New Samples from Handwritten Numerals Based on Point Correspondence[END_REF], segmentation, restoration. . . ) and to enrich the training database in order to improve the system performance [START_REF] Varga | Effects of Training Set Expansion in Handwriting Recognition Using Synthetic Data[END_REF].

There are three main approaches to degrade an image document: adding noise on an entire or a part of document, degrading characters, and distorting the shape of documents. The noise models are early and widely used to test the robustness of different document image analysis methods towards noise. The noise has both global and local effects. In [START_REF] Jian Zhai | A Line Drawings Degradation Model for Performance Characterization[END_REF], the document degradation model which simulates 1 http://www.cvc.uab.es/cvcmuscima/competition2013/index.htm four types of classic noises (Gaussian noise, high frequency noise, hard pencil noise and motion blur noise) is proposed by Zhai et al. This model is used for assessing the robustness of their line detection algorithm. The noise resulting from the physics of the image acquisition process is modelled and used to evaluate the performance, as the perturbation model of Loce et al [START_REF] Loce | Halftone Banding due to Vibrations in A Xerographic Image Bar printer[END_REF]. In addition, the effects of ten parameter model for character degradation are studied by Baird [START_REF] Baird | Structured Document Image Analyse[END_REF]. He also described the most frequently observed degradations and their causes in [START_REF] Baird | The State of the Art of Document Image Degradation Modeling[END_REF]. Based on his work, a degradation model and a method for estimating its parameters have been proposed in [START_REF] Smith | Modeling Image Degradations for Improving OCR[END_REF] for improving OCR. Kanungo et al [START_REF] Kanungo | Global and Local Document Degradation Models[END_REF] proposed a popular nonlinear local document deformation model mimicking additive noise due to the scanning process. This model adds "salt and pepper" noise, i.e. flipping pixels from background to foreground and vice versa, in the neighbourhood of the characters. However, most of these models only work with binary images. Therefore, several authors simulate most common grayscale defects due to the age of the document itself and the printing/writing process for performance evaluation works. For example, in [START_REF] Rabeux | Document Rectoverso Registration Using a Dynamic Time Warping Algorithm[END_REF], the bleed-through deformation model is presented. This model is used to compare the robustness of two OCR algorithms when the bleed-through intensity increases. We also proposed a grayscale character degradation model in [START_REF] Kieu | A Character Degradation Model for Grayscale Ancient Document Images[END_REF] which simulates most common defects in ancient documents such as ink splotches, white specks or streaks.

The geometric distortions also impact on the system's performance. The causes of these defects are numerous, for example, the thick and bound document, the incorrect position of camera or document. . . . In [START_REF] Kanungo | Global and Local Document Degradation Models[END_REF], Kanungo also presented a perspective distortion model. He considered the physical deformed form of thick and bound documents near the spine as a circular arc segment. This model takes into account the defects of the illumination and optical distortion process as well. Jian Liang et al [START_REF] Liang | Geometric Rectification of Camera-Captured Document Images[END_REF] generalized the model by using the reverse mapping process in which the thick and bound documents are modelled as the developable surfaces. These surfaces are approximately divided into a group of planar quadrilaterals to simplify the reverse mapping process. The authors then used synthetic images to evaluate the robustness of their geometric rectification algorithm. Almost all of the geometric distortion models use simple geometric transformation functions and approximate calculations to deal with complex shapes that lead to non-realistic results. Indeed, it is difficult to generate synthetically many geometric distortions observed in real documents (e.g. dents, small folds, torns in Fig. 1) by using geometric transformation functions because they follow no mathematical rules. However, since the ultimate goal of all

Curve

Convexo-concave Dent Fold Skew Torn algorithms is to handle with the real data, we really need to generate these distortions for evaluation works. Thus, we propose a method based on the texture mapping. By using this method, we can take advantage of effectively simulating real-life of 3D texture mapping technique to reproduce real defects. The method has three steps as mentioned in This paper is organized as follows: in Section II, the proposed method is described. Experimental results for visual validation are given in Section III. Conclusions and future works are provided in Section IV.

II. PROPOSED METHOD

A. Real degraded document 3D scan process

Hardware. The Kréon Aquilon laser 3D scanner (scan rate up to 1.2 million points per second and the accuracy for distances up to 60µm) is employed to create a polygon mesh that represents the real document shape (see Fig. 3). The result mesh M is a collection of vertices, edges, and triangles. Every vertex of the mesh will be mapped to a point in the plane in the next step.

B. Texture coordinate generation

The texture coordinate generation process maps every point in the curved surface to the plane. In the field of computer graphics, it generally involves many tasks to deal with high quality images (i.e. segmentation, optimization). In mathematics, a curved surface can be firstly divided into planar strips (i.e quadrilaterals, triangles). And then, each strip is mapped to the plane by solving the reverse mapping equation as described in [START_REF] Liang | Geometric Rectification of Camera-Captured Document Images[END_REF]. These processes, nevertheless, take a lot of time when the sampling result needs to be smooth. The other method presented in [START_REF] Ulges | Document Capture Using Stereo Vision[END_REF] projects directly each single triangle in the surface onto texture coordinates without considering the distance relation of vertices, which may lead to errors (e.g. stretching or a triangle with the acute angle less than 5 degrees becomes straight). Thus, we propose a technique that takes into account the distance between vertices by unfolding directly a 3D line into the texture coordinates.

In our approach, the mesh of previous step is transformed to the position in which the fore-edge or the spine of document shape is parallel to the plane Oyz before 3D lines are defined. Let P y be a plane containing at least one vertex v i provided that P y ⊥ Oy (see Fig. 4). Let y be the ycoordinate of the vertex v i . The plane P y intersects the mesh M in a 3D line L y = {S 1 , S 2 , . . . , S k } where S k is an intersection point of P y with an edge e l . Because P y ⊥ Oy, all the points S k have the same y-coordinate. For example, in Fig. 5, the intersection of the plane P 300 and the mesh M is the 3D line L 300 = {S 1 , S 2 , S 3 , S 4 , S 5 , S 6 , S 7 , S 8 , S 9 } in which S 6 , S 8 are two vertices and its y-coordinate is 300. In the next step, the intersection points in the 3D line will be mapped with points in the texture coordinates (u, v).

The 3D line L y can be then unfolded by projecting it onto the texture coordinates (u,v) on condition that the distance of two adjacent points is conserved. So, let L ′ y be the projected line in the texture coordinates and {S ′ 1 , S ′ 2 , . . . , S ′ k } be the corresponding projected points. As a result, the coordinates S ′ k (u, v) are the texture coordinates of the vertex S k . In our example, the line

L ′ 300 = {S ′ 1 , S ′ 2 , S ′ 3 , S ′ 4 , S ′ 5 , S ′ 6 , S ′ 7 , S ′ 8 , S ′
9 } is the projection of the line L 300 provided that S 1 S 2 = S ′ 1 S ′ 2 , S 2 S 3 = S ′ 2 S ′ 3 , . . . , S 8 S 9 = S ′ 8 S ′ 9 (see Fig. 6). We keep only the texture coordinates of vertices. In the example, S ′ 6 and S ′ 8 are saved for the next step. The plane P y slides from the first vertex to the last vertex of the mesh M until all the texture coordinates of the vertices are calculated. These coordinates are normalized for fitting with the image input in the texture image mapping step.

C. Texture image mapping

Once the mesh has been unfolded, any 2D document image can be overlayed on the mesh by using the texture mapping technique already supported in the library OpenGL. As a consequence of the previous step, the triangles in the mesh have one by one correspondences in the texture coordinates. Generally, in order to pack a document image on the mesh, we put the image on the texture coordinates as a mask (see Fig. 7). Then, we consider each triangle in the texture coordinates. The image pixels inside of that triangle are used to fit in the mesh. For the unfilled pixels, we need an interpolation procedure. The bi-linear interpolation is used Figure 6. Projection of the 3D line L 300 onto the texture coordinates in this case. Thanks to the texture mapping technique and Phong reflection model [START_REF] Phong | Illumination for Computer Generated Pictures[END_REF], we can calculate exactly where a pixel of the input image is mapped to the output, and we can estimate the grey-level value of this pixel. The real document 3D scan process takes a long time because of scan operations and document collection steps (i.e. finding books, selecting representative pages, installing pages on the scanner. . . ). At the first time, we have considered 20 books among which we selected and scanned four pages that contain almost all of geometric defects. Therefore, we have four meshes (four document shapes) to assess the effectiveness of our approach. First, preliminary visual validation results are given to illustrate the presence of distortions in semi-synthetic generated images. And then, a set of 6000 images is provided as a database for benchmarking in the context of ICDAR 2013 contests.

A. Preliminary visual validation result

Synthetic grayscale images from our database DIGIDOC are used to pack on scanned meshes. The observed defects in the real documents are reproduced. In Fig. 8, the generated documents illustrate the presence of convexo-concaves, folds, torns. These defects appear frequently in the real document image because of the ageing process and the physical impact of the utilization of the documents.

In Fig. 9, both the original mesh and the semi-synthetic degraded image of a right page are given. We can see clearly illumination defects due to the convexo-concave defects, one side being darker and the other one brighter (see the bottomleft images in Fig. 9). The effect of light makes the synthetic images more realistic. Besides, the dents may appear in the ancient documents (e.g. when the wood-character stamps have strongly been pressed on the sheet of paper).

Fig. 10 provides a visual comparison of the proposed model with the global distortion model of Kanungo [START_REF] Kanungo | Global and Local Document Degradation Models[END_REF] (see the left image of Fig. 10). Only the region near the document spine is distorted in Kanungo's model. In addition, the non-linear optical point spread is modelled by applying a Gaussian blur function; thus, the result of the model is blurred.

Fig. 11 provides a visual comparison of the proposed model with J. Liang's model [START_REF] Liang | Geometric Rectification of Camera-Captured Document Images[END_REF], which uses the reverse mapping process (left image in Fig. 11). In our result (the right image in Fig. 11), the real distortions such as dents, folds or irregular curves can be simulated, which makes the curved surface more realistic.

B. Semi-synthetic images for performance evaluation

The music score CVC-MUSCIMA database, which contains handwritten music score images at both gray and binary levels, is largely used to test the robustness of different optical music recognition and removal staff line algorithms [START_REF] Fornes | The ICDAR 2011 Music Scores Competition: Staff Removal and Writer Identification[END_REF]. For the removal staff line competition of ICDAR 2013, a set of 6000 semi-synthetic versions of the 1000 images original images has been generated (among which 4000 for Result of Kanungo's model Result of our method training and 2000 for test). These images are generated by using jointly the proposed method and the local character degradation model described in [START_REF] Kieu | A Character Degradation Model for Grayscale Ancient Document Images[END_REF] (see Fig. 12). They are challenging for staff removal algorithms, especially because the 3D degradations can distort the staff lines, making their detection and removal more difficult. Indeed, the addition of a white speck in the middle of a staff line can lead to disconnections within the lines, which could be tricky for most staff line extraction algorithms. Similarly, adding a dark speck connected to a staff line makes segmentation more difficult, as specks might be confused with musical symbols. These examples in Section III-A and III-B show that the proposed method can be applied in various contexts for evaluating the performance of systems. These experimental results (see more results on2 ) are encouraging. We are

IV. CONCLUSION

In this paper, we present a method based on 3D meshes and texture coordinate generation for simulating geometric distortions in semi-synthetic images. This method is adapted to document images to deal with the unwrapping errors (i.e. stretching); therefore, the obtained images are very realistic for observed geometric distortions such as dents, folds, convexo-concaves, torns. These distortions can be reproduced by using three steps in the method. First, real documents are scanned to obtain document shapes (meshes). Then, we run (once per mesh) the new texture coordinate generation method, specially fitted for document images, and therefore more effectively in our context than most existing techniques. Finally, any 2D document image can be packed in the document shape by using an existing texture mapping technique. The Phong reflection model is used to improve the image quality. The generation process of this method enables to create synthetic images quickly; the only limitation in the variety of documents we can produce only lies in the number of 3D pages available (meshes obtained by 3D scanning of ancient document pages). The method is integrated with other degradation models in a system dedicated to a semi-synthetic old document image generation. This system can generate benchmark databases that will be further used to evaluate or compare the performances of different systems (e.g. the musical score competition at ICDAR 2013) and to enrich the training data in various contexts.

Figure 1 .

 1 Figure 1. Geometric distortions in real document images

Fig. 2 :

 2 (A) Real degraded document 3D scan, (B) Texture coordinate generation and (C) Texture image mapping.

Figure 2 .

 2 Figure 2. Texture mapping on real 3D document shape process

Figure 3 .

 3 Figure 3. An mesh of a document shape and triangles inside

Figure 4 .Figure 5 .

 45 Figure 4. The plane Py, which passes through the vertex v i , intersects with the mesh M in 3D view

Figure 7 .

 7 Figure 7. Packing an image on the mesh with extracted texture coordinates

Figure 8 .Figure 9 .

 89 Figure 8. Generated synthetic images with convexo-concaves, fold and torn Original mesh Generated image

Figure 10 .Figure 11 .

 1011 Figure 10. Example of Kanungo's model in [10] (left) and result of our method (right)

Figure 12 .

 12 Figure 12. Example of synthetic images of music score removal staff line ICDAR 2013: (a) An original image of the CVC-MUSCIMA database, (b) gray distorted image, (c) binary distorted image with staff lines

http://www.labri.fr/perso/vkieu/content/Databases/music removalStaff.html

ACKNOWLEDGEMENT

Authors would like to thank Jérôme Charton for discussions on the 3D scan process. This research is done within the DIGIDOC project financed by the ANR (Agence Nationale de la Recherche).


François Portet
email: fportet@csd.abdn.ac.uk

Feng Gao
email: fgao@csd.abdn.ac.uk

Jim Hunter
email: jhunter@csd.abdn.ac.uk

René Quiniou
email: quiniou@irisa.fr

Reduction of Large Training Set by Guided Progressive Sampling: Application to Neonatal Intensive Care Data

Although large training sets are supposed to improve the performance of learning algorithms, there are limits to the volume of data such an algorithm can handle. To overcome this problem, we describe an improvement to a progressive sampling method by guiding the construction of a reduced training set. The application of this method to neonatal intensive care data shows that it is possible to reduce a training set to a third of its original size without decreasing performance.

Introduction

Intensive Care Units generate large volumes of data -about 1 MB per patient per day. However, such large volumes are difficult to analyze, so data mining or machine learning techniques are often used to learn classifiers for prediction and decision support. Although the general approach is to learn classifiers from the largest possible dataset, learning a classifier from too large a dataset can be computationally impossible or time-consuming and thus the training set must be reduced.

'Data reduction' aims at aggregating the information contained in large datasets into manageable smaller information chunks, using simple tabulation, clustering, principal component analysis (PCA), etc. However, these methods need either data pre-processing or modification of the example datasets in such a way that it is more difficult to interpret the model which has been learned (e.g. PCA). Progressive Sampling (PS) [Provost et al. 1999] incrementally constructs a training set from a larger dataset without decreasing the classification performance and without altering the initial format of the examples. In this paper, we propose a variant of PS and show its application to the domain of Neonatal Intensive Care.

Progressive Sampling

Progressive Sampling (PS) starts with a small training subset (TS) of the full dataset (FDS) and incrementally extends TS until the learning accuracy satisfies some convergence criteria. The resulting dataset is expected to be smaller than FDS and to lead to (at least) the same performance. Figure 1 shows the general algorithm. Before starting the learning process, the progressive sizes of TS are scheduled (planned). Then TS is used to learn the classifier model M (by a decision tree, neural network, etc.) which is tested until convergence is attained. The optimal training set is computed by mean of a learning curve which is used to retain the best balance between size and learning performance. Provost et al. [1999] have showed that when dealing with large volume of data, PS is more efficient than using the entire dataset. However, PS does not explicitly deal with unbalanced datasets. To face this problem, [START_REF] Ng | [END_REF] introduced a method to improve the relative distribution of each class by oversampling the minor class in computeTS(FDS). But, as they emphasized, replicating examples from the smaller classes (over-sampling) leads to over-fitting.

Let

These approaches select the examples to be added into TS at random. We believe that it is possible to speed up the convergence by using a priori information to select the most appropriate examples to add.

Guided Progressive Sampling

Guided PS (GPS) uses a distance measure d between the samples in TS and the samples in FDS to guide the selection of samples to add to TS. Once M is learned, each e i ∈FDS is tested to form the triple (e i , m(e i ), d(e i )) where m(e i ) is the result of the classification of e i using M (m(e i )∈{correct, incorrect}) and d(e i ) gives the distance from e i to the centroid of the class to which it actually belongs. This set of triples is used in computeTS(FDS) according to one of two strategies: 1. GPS adds to each class in TS, the worst misclassified examples i.e. those with the highest values of d(e i ). This is intended to improve learning robustness by considering the difficult cases. 2. GPS+ extends GPS by additionally adding the best correctly classified examples i.e. with the lowest values of d(e i ). This is intended to reinforce learning stability which can be distorted by only including the worst misclassifications. These choices rest on the assumption that learning is most influenced by the extreme examples of each class (correct classifications and misclassifications). The distance measure d does not need to be exact (otherwise it would be directly used to learn the model!) but is a heuristic estimate of how much the classification is wrong.

Case study: bradycardia detection

The method has been tested on the detection of bradycardias by decision tree learning (C4.5 with pruning). The dataset consists of thirteen heart rate (HR) time series each covering 24-hours recorded from premature babies receiving intensive care. The episodes of bradycardia were annotated by two clinical experts. Each example in FDS is described by 25 attributes (raw HR value and min, max, slope etc. over several centered windows). The size of the complete 13 record dataset is more than 80MB. Given such a large dataset, learning on the entire set is impossible. Moreover, the dataset is completely unbalanced. For example, in record #16234, the bradycardia class contains 533 examples whereas the no-bradycardia class contains 79875 examples, the bradycardias representing only 0.66% of the total dataset. However, this is to be expected, as bradycardia is defined as a short transient event. In addition, the records contain episodes of artifact that can perturb learning.

Random sampling (RS), GPS, and GPS+ have been used. To try to balance the large difference between bradycardia and no-bradycardia, the initial TS contained 100% of the bradycardias and 1% (selected at random) of the no-bradycardias. On each iteration, 3.33% more of the no-bradycardias were selected from FDS according to the particular strategy in use. Learning was stopped when the learning curve become sufficiently stable [Provost et al., 1999].

Fig. 2 shows the number of classification errors against the training size for record #16234. GPS converged faster than GPS+ and RS. GPS+ and RS converged at the same iteration however GPS+ led to higher accuracy. GPS and GPS+ produced more errors at the beginning of the process as they initially selected the most difficult examples to classify but this led rapidly to a more stable plateau (fewer oscillations) than RS. The figure shows that after reaching the beginning of the plateau, the examples added do not provide information that has not already been learned by the decision tree. Results found with GPS led to 111 errors for TS=9317 examples. Thus around 90% of the dataset is not useful for learning. The decision tree learned with GPS led to the same performance (111 errors) as the decision tree learned from FDS but with a slightly smaller tree. The proportion of bradycardias is still not equally distributed but increased from 0.66% to 5.72%.

Mean accuracy over the 13 datasets was 99.66% for RS (20 runs), 99.84% for GPS+ and 99.85% for GPS, with significant differences between GPS (or GPS+) and RS in the worst case, p<0.0001 for #16234).

Discussion

Guided progressive sampling has shown to be more efficient than random progressive sampling for learning from a massive training set. Using a priori knowledge to guide the sampling leads to a faster convergence and a better selection of the "relevant" examples to use for learning. Further experiments will be undertaken to improve bradycardia detection with the reduction of larger datasets. This approach can also be useful in a situation with a small dataset to capture the "best" training examples.

Fig. 1

 1 Fig. 1 Progressive sampling algorithm.


Alicia Fornés
email: afornes@cvc.uab.es

V C Kieu
email: vkieu@labri.fr

Muriel Visani
email: muriel.visani@univ-lr.fr

Nicholas Journet
email: journet@labri.fr

Anjan Dutta

The ICDAR/GREC 2013 Music Scores Competition: Staff Removal

The first competition on music scores that was organized at ICDAR and GREC in 2011 awoke the interest of researchers, who participated in both staff removal and writer identification tasks. In this second edition, we focus on the staff removal task and simulate a real case scenario: old and degraded music scores. For this purpose, we have generated a new set of images using two degradations: local noise and 3D distortions. In this paper we provide an extended description of the dataset, degradation methods, evaluation metrics, the participant's methods and the obtained results that could not be presented in ICDAR and GREC proceedings due to page limitations.

Introduction

The recognition of music scores has been an active research field for decades [START_REF] Blostein | Structured Document Image Analysis[END_REF][START_REF] Rebelo | Optical music recognition: state-of-the-art and open issues[END_REF]. Many researchers in Optical Music Recognition have proposed staff removal algorithms in order to make easier the segmentation and enhance the accuracy of music symbol recognition [START_REF] Dalitz | A comparative study of staff removal algorithms[END_REF][START_REF] Santos Cardoso | Staff detection with stable paths[END_REF]. However, the staff removal task cannot be considered as a solved problem, especially when dealing with degraded handwritten music scores. This task is even defined as one of the "three challenges that should be addressed in future work on OMR as applied to manuscript scores" in the 2012 survey of Rebelo et al. [START_REF] Rebelo | Optical music recognition: state-of-the-art and open issues[END_REF].

At ICDAR [START_REF] Fornés | The icdar 2011 music scores competition: Staff removal and writer identification[END_REF] and GREC 2011, we organized the first edition of the music scores competition. For the staff removal task, we created several sets of distorted images in order to test the robustness of the staff removal algorithms. Each set corresponded to a different kind of distortion (e.g. Kanungo noise, rotation, curvature, staffline interruption, typeset emulation, staffline y-variation, staffline thickness ratio, staffline thickness variation and white speckles). The staff removal competition woke up the interest of researchers, with eight submitted participant methods. Most staff removal methods showed good performance in front of severe distorted images, although the detection of the staff lines still needed o be improved.

After GREC 2011, we extended the staff removal competition [START_REF]The 2012 music scores competitions: staff removal and writer identification[END_REF]. The goal was to simulate a real scenario, in which music scores usually contain more than one single kind of distortion. For this purpose, we combined some of the ICDAR 2011 distortions at different levels to create new sets of degraded images. We then asked the participants to run their algorithms on this new set of images. Unsurprisingly, the new results demonstrated that the performances of most methods were significantly decreased because of the combination of distortions.

By organizing a second edition of this competition at ICDAR / GREC 2013, we aimed at fostering the interest of researchers and focusing on the challenging problem of old document image analysis and recognition. For this second edition, we have generated realistic semi-synthetic images that emulate typical degradations appearing in old handwritten documents such as local noise and 3D distortions.

The rest of the paper is organized as follows. Firstly, we will describe the original dataset, the degradation models, and the generated training and test sets. Then, we will present the participants' methods. Finally, we will detail the the evaluation metrics, the results analysis, and conclude the paper.

Database

In this section we describe the original database, the degradation methods, and the semi-synthetic database for the competition.

Original CVC-MUSCIMA database

The original CVC-MUSCIMA1 database [7] consists of 1,000 handwritten music score images, written by 50 different musicians. The 50 writers are adult musicians, in order to ensure that they have their own characteristic handwriting music style. Each writer has transcribed exactly the same 20 music pages, using the same pen and kind of music paper. The 20 selected music sheets contain music scores for solo instruments, choir or orchestra.

Degradation Models

3D Degradation Model Since the geometric distortions such as skews and curvatures are challenging for detecting staffs, we used them for the 2011 staff removal competition [START_REF] Fornés | The icdar 2011 music scores competition: Staff removal and writer identification[END_REF][START_REF]The 2012 music scores competitions: staff removal and writer identification[END_REF]. However, these distortion models were only 2D models which are unable to reproduce the geometric distortions commonly encountered in real old documents such as dents, small folds, and torns. . . (see Fig. 1). Therefore, in this 2013 edition, we use the 3D degradation [START_REF] Kieu | Semi-synthetic document image generation using texture mapping on scanned 3d document shapes[END_REF] that can generate more realistic and more challenging distortions of the staff lines, making their detection and removal more difficult. This 3D degradation model is based on 3D meshes and texture coordinate generation. It can wrap any 2D (flat) image of a document on a 3D mesh acquired by scanning a non-flat old document using a 3D scanner. The wrapping functions we use are specifically adapted to document images. In our case, we wrap the original music score images on different 3D meshes. For more details, please refer to [START_REF] Kieu | Semi-synthetic document image generation using texture mapping on scanned 3d document shapes[END_REF].

Curve

Convexo-concave Dent Fold Skew Torn

Fig. 1: Geometric distortions in real document images

Local Noise Model This model, introduced in [START_REF] Kieu | A character degradation model for grayscale ancient document images[END_REF], can mimic old documents' defects due to document aging and to the old printing/writing processes. Examples of these defects include ink splotches and white specks or streaks (see Fig. 2). Such defects might break the connectivity of strokes or add a connection between separate strokes. For staff line removal algorithms, local noise can lead to many types of challenging degradations. Indeed, it can lead to disconnections of the staff lines or to the addition of dark specks connected to a staff line. In the latter case, for instance, the dark specks might be confused with musical symbols.

Fig. 2: Examples of local noise in real old documents

As detailed in [START_REF] Kieu | A character degradation model for grayscale ancient document images[END_REF], the local noise is generated in three main steps. Firstly, the "seed-points" (i.e. the centres of local noise regions) are selected in the neighborhood of connected components' borders (obtained by binarizing the input grayscale image). Then, we define an arbitrary noise region at each seedpoint (in our case, its shape is an ellipse). Finallly, the grey-level values of the pixels inside the noise regions are modified so as to obtain realistic looking bright and dark specks, and mimic defects due to the age of the document (ink fading, paper degradation...) and writing process (ink drops).

Degraded Database

For comparing the robustness of the staff removal algorithms proposed by the participants to this competition, we degrade the original CVC-MUSCIMA database using the two degradation models presented in Section 2.2. As a result, we obtain a semi-synthetic database that consists of 4000 images in the training set and 2000 images in the test set.

Training Set It consists in 4000 semi-synthetic images generated from 667 out of the 1000 original images. This set is split into the three following subsets:

-TrainingSubset1 (see Fig. 3) contains 1000 images generated using the 3D distortion model and two different meshes. The first mesh consists in a perspective distortion due to the scanning of a thick and bound volume, while the second one contains many small curves, folds and concavities. We wrap the 667 original images on two meshes to produce 2x667=1334 semi-synthetic images. Then, 500 images per mesh are randomly selected so as to obtain a total of 1000 images. -TrainingSubset2 (see Fig. 4) contains 1000 images generated with three levels of local noise (see Sub-section 2.2), as follows (the flattening factor of the elliptic noise region is fixed as g=0.6 for the three levels, whereas the noise region size a 0 increases after each level):

• Low level: 333 images, 500 seed-points, a 0 : 7;

• Medium level: 334 images, 1000 seed-points, a 0 : 8.5;

• High level: 333 images, 1300 seed-points, a 0 : 10. -TrainingSubset3 (see Fig. 5) contains 2000 images generated using both the 3D distortion and the local noise models. We obtain six different levels of degradation (the two meshes used for TrainingSubset1 × the three levels of distortion used for TrainingSubset2).

For each image in the training set, we provide its grey and binary versions. The associated ground-truth are the binary staff-less version (binary images without staff lines), as illustrated in Fig. 6.

Test Set It consists of 2000 semi-synthetic images generated from the 333 original images that differ from the ones for the training set.

-TestSubset1 contains 500 images generated using the 3D distortion (see subsection 2.2). Two meshes -distinct from the ones used in the training set -are applied to the 333 original images and then only 500 images (250 for each mesh) are randomly selected among the 2x333=666 degraded images. -TestSubset2 contains 500 images generated using three levels of local noise, using the same values of the parameters as in TrainingSubset2, under the proportions 1 3 / 1 3 / 1 3 . -TestSubset3 contains 1000 images equally distributed between six different levels of degradation. These six levels of degradation come from the combination of the same two meshes as in TestSubset1 with the same three different levels of local noise as in TrainingSubset2.

For each image in the test set, we provide its grey and binary versions. The test set was provided to the participants 46 days after the training set (containing 4000 degraded images together with their ground-truth). The participants were asked to send to the organizers the outputs of their algorithms as binary staffless images (such images containing only binarized music symbols but no staff lines) 23 days after the test set was provided to them.

Participants Information

In this section we will briefly describe the eight submitted methods of five participants for the ICDAR/GREC2013 competition. Methods 1-3 work on binary images (in that case the participants used the binary versions we provided for the competition), while methods 4-5 can handle both binary and grayscale images.

TAU-bin

This method was submitted by Oleg Dobkin from the Tel-Aviv University, Israel. It is based in the Fujinaga's method [START_REF] Fujinaga | Adaptive optical music recognition[END_REF]. First, the staffline height and staffspace height are estimated using vertical scans. Then, the vertical black runs which are longer than the staffspace height are removed. Afterwards, the music page is globally deskewed, and the staff lines are located using a projection on the y-axis. Finally, the staff lines are removed using masks.

NUS-bin

This method was submitted by Bolan Su (National University of Singapore), Umapada Pal (Indian Statistical Institute, Kolkata, India) and Chew-Lim Tan (National University of Singapore). The method, detailed in [START_REF] Su | An effective staff detection and removal technique for musical documents[END_REF], first estimates the staffline height and staffspace height using the vertical run length histogram. These estimated values are used to predict the lines' direction and fit an approximate staff line curve for each line. Then, the fitted staff line curve is used to identify the exact location of staff lines in the image. Finally, those pixels belonging to these staff lines are removed.

NUASi

Christoph Dalitz and Andreas Kitzig, from the Niederrhein University of Applied Sciences (iPattern Institute), Krefeld, Germany, submitted the following two different methods:

-NUASi-bin-lin: This method is described in Section II of [START_REF] Dalitz | A comparative study of staff removal algorithms[END_REF]. First, the staffs are detected, and the staffline height is estimated as the most frequent black vertical run length. Then, the skeleton of the staff lines is extracted, and all vertical foreground runs shorter than 2* staffline height are removed. The function chordlength(ϕ) (where ϕ is the angle of the chord at the intersection region) is used to filter staff-line pixels belonging to a crossing music symbol. The source code is available at http://music-staves.sourceforge.net/ (class MusicStaves linetracking).

-NUASi-bin-skel: This method, detailed in the Section III.D of [START_REF] Dalitz | A comparative study of staff removal algorithms[END_REF], first splits the skeleton of the staff lines at branching and corner points. Each segment is considered as a staff line segment if it satisfies some heuristic rules. Then, two staff segments are horizontally linked if their extrapolations from the end points with the least square fitted angle are closer than staffline height/2. The staff segment results may contain false positive staff segments (e.g. in the case where a staff line is tangent with the curve of a music symbol or it overlaps with the music symbol at a staff segment). Then, to check for the false positives, non-staff segments which have the same splitting point as a staff segment are extrapolated by a parametric parabola. If the staff segment is tangent with the parabola, it is a non-staff segment. Finally, vertical black runs around the detected staff skeleton are removed when they are shorter than 2* staffline height. The source code is available at http://music-staves.sourceforge.net/(class MusicStaves skeleton).

LRDE

Thierry Géraud, from the EPITA Research and Development Laboratory (LRDE), Paris, France, submitted two methods. For more details, the reader is referred to http://www.lrde.epita.fr/cgi-bin/twiki/view/Olena/Icdar2013Score.

-LRDE-bin: This method relies on mathematical morphological operators. First, a permissive hit-or-miss with a horizontal line pattern as structuring element extracts some horizontal chunks. Second, a horizontal median filter cleans up the result, and a dilation operation is applied using a horizontal neighbourhood in order to enlarge the connected components. A binary mask is obtained thanks to a morphological closing with a rectangular structuring element. Last, a vertical median filter, applied inside the largest components of this mask, removes the staff lines. -LRDE-gray: After removing the image border, Sauvola's binarization and a dilation using a horizontal neighbourhood are applied. The resulting image serves as a mask in which a two-level thresholding with hysteresis of the original image is applied. Finally, some spurious horizontal parts of the stafflines are erased in a post-processing step.

INESC

Ana Rebelo and Jaime S. Cardoso (INESC Porto and Universidade do Porto) submitted the following two methods (more details are given in [START_REF] Santos Cardoso | Staff detection with stable paths[END_REF]) based on graphs of Strong Staff-Pixels (SSP: pixels with a high probability of belonging to a staff line):

-INESC-bin: First, the staffline height and staffspace height are estimated by the method presented in [START_REF] Cardoso | Robust staffline thickness and distance estimation in binary and gray-level music scores[END_REF]. Then, all the pixels of the black runs of staffline height pixels followed or preceded by a white run of staffspace height pixels are set as the SSPs. To decide if a SSP belongs to a staff line, the image grid is considered as a graph with pixels as nodes, and arcs connecting neighbouring pixels. Then, SSPs are classified as staff line pixels acording to some heuristic rules. Then, the groups of 5 staff lines are located among the shortest paths by using a global optimization process on the graph. -INESC-gray: For grayscale images, the weight function is generalized by using a sigmoid function. The parameters of the sigmoid function are chosen to favor the luminance levels of the stafflines. A state-of-the-art binarization technique is used in order to assign the cost for each pixel in the graph (pixels binarized to white have a high cost; pixels binarized to black have a low cost). Once the image is binarized, the previous method is applied.

Results

In this section we compare the participant's output images with the groundtruth (binary staff-less images) of the test set using the measures presented in the next Section. The ground-truth associated to the test set was made public after the competition.

Measures Used for Performance Comparison

The staff removal problem is considered as a two-class classification problem at the pixel level. For each test subset and each level of noise, we compare the output images provided by the participants to their corresponding ground-truth. For this purpose, we compute the number of True Positive pixels (TP, pixels correctly classified as staff lines), True Negative pixels (TN, pixels correctly classified as non-staff lines) False Positive pixels (FP, pixels wrongly classified as staff lines) and False Negative pixels (FN, pixels wrongly classified as non-staff lines). Then, from these measures, we compute the accuracy (also called Classification Rate), precision (also called Positive Predictive Value), recall (also called True Positive Rate or sensitivity), F-measure and specificity (or True Negative Rate) as follows:

accuracy = T P + T N T P + T N + F P + F N (1) precision = T P T P + F P (2) recall = T P T P + F N (3)
F -M easure = 2 × precision × recall precision + recall (4) specif icity = T N T N + F P (5)
Since the first step of a staff removal system is usually the detection of the staff lines, the overall performance highly depends on the accuracy of this preliminary staff detection. Indeed, if the staff line is undetected, it is unable to be removed. For example, a staff removal system may obtain very good results (when the staff is correctly detected) while rejecting many music scores images (if no staff line is detected in the image, it is discarded). Therefore, for each participant's method, we provide the number of rejected pages for each of the three test subsets, and for each level of degradation inside each test subset. Furthermore, we compute the evaluation measures [START_REF] Blostein | Structured Document Image Analysis[END_REF][START_REF] Rebelo | Optical music recognition: state-of-the-art and open issues[END_REF][START_REF] Dalitz | A comparative study of staff removal algorithms[END_REF][START_REF] Santos Cardoso | Staff detection with stable paths[END_REF][START_REF] Fornés | The icdar 2011 music scores competition: Staff removal and writer identification[END_REF] in two ways:

-Without rejection: the five average values of the five measures (1-5) are computed inside each test subset and for each level of degradation, taking into account only the images that the system pre-classified as music score. Thus, the rejected images are not taken into account for these measures. -With rejection: the five average values of the five measures (1-5) are computed inside each test subset and for each level of degradation, taking into account every image in the test set, no matter if it was rejected by the system or not. Thus, for a rejected image, every staff line pixel is considered as a False Negative and every non-staff line pixel is considered as a False Positive.

Baseline

For comparison purposes, we have computed some baseline results using the existing staff removal method proposed by Dutta et al. [START_REF] Dutta | An efficient staff removal approach from printed musical documents[END_REF]. This method is based on the analysis of neighboring components. Basically, it assumes that a staff-line candidate segment is a horizontal linkage of vertical black runs with uniform height. Then, some neighboring properties are used to validate or discard these segments.

Performance comparison

Table 1 and Fig. 7 present staff removal results obtained by the eight participant methods (three out of the five participants presented two different algorithms).

We have also tested the baseline method presented in [START_REF] Dutta | An efficient staff removal approach from printed musical documents[END_REF]. Each of the nine columns indicates the name of the participant and the staff removal method category. All the metrics used for performance comparison (presented in section 4.1) are presented according to the type of degradation (3D distortion, local noise model, or a mixture of the two) and the degree of degradation. Indeed, for each sub-dataset five metrics are presented. The winner's method of each line is highlighted in bold. Since the precision is higher in some methods but with a lower recall, we decided to select the winners according to the accuracy and the F-measure metrics shown in Fig. 7. From Table 1, we can see that the LRDE-bin method is the winner of the 3D distortion set, the INESC-bin is the winner of the Local noise set, and LRDE-bin is the winner of the combination set. Fig. 7 presents averages Accuracy and F-Measures of the nine tested methods on three different sub-datasets (3D distortion, local noise model, a mixture of the two), whereas Table 1: Competition results for each test subset and each degradation level. We give the number # of rejected images, and the values of the measures computed with and without rejection. The measures (M.), showed in %, are: P=Precision, R=Recall, F-M=F-Measure, S=Specificity, A=Accuracy. Much can be learned from the scores presented in Table 1. First, whatever the category (and intensity) of defect integrated in semi-synthetics images, the best results are mainly obtained with binary images (LRDE-bin, NUAS-bin-lin and INESC-bin). Only one method with grey-level image as input (LRDE-gray) gives 11 times the best results. It is also interesting that this method gives always the best recall score. The results showed that the baseline method [START_REF] Dutta | An efficient staff removal approach from printed musical documents[END_REF] performances are almost identical to those obtained by each winning method.

Another interesting way to analyze these results is to compare the scores with the level of degradations of each sub dataset. Concerning 3D distortion, the staff lines images mapped on Mesh 1 (M1) were a bit more difficult to analyze for all the participants (except LRDE-gray and INESC-bin). On average the precision scores drop by 1 point. One method drops its precision by 7 points. We can conclude that these methods seems less robust to perspective deformation defects (Mesh 1) than to the presence of small curves and folds (Mesh 2). The precision scores of each participant decrease when the local noise is getting higher. On average, the precision scores decrease from 13%. In the best case it dropped by Fig. 8: F-measures of the eight participant methods (plus the baseline method) on the 3 Test Subsets and 11 levels of degradation. less than 1%. In the worst case the precision is dropped by 33%. These results clearly show that all these methods are sensitive to the local noise deformation. The tests carried out with images generated by combining local noise and 3D distortions confirm that the results decrease when the level of degradation is important.

Conclusion

The second music scores competition on staff removal held in ICDAR2013 has raised a great interest from the research community, with eight participants' methods in the competition. For this competition, we generated a database of semi-synthetic images using the 1000 images from the CVC-MUSCIMA database and two models of degradation specifically designed to mimic the defects that can be seen in historical documents. This database contains three subsets both for training and testing: one subset containing only 3D distorsions at two different levels; one subset containing thee levels of local noise and one subset with a combination of both sources of noises, with six different levels of degradation. We evaluated the performances of the eight methods proposed by the participants and compared it towards a baseline method based on the analysis of neighbouring connected components.

The eight submitted methods have obtained very satisfying performances, even though the degradations in the proposed images were quite severe. The results of the participants have demonstrated that the performance of most methods significantly decreases when dealing with a higher level of degradation, especially in the presence of both sources of degradation (3D distortion model + local noise model). We hope that our semi-synthetic database, which is now available on the internet and labelled with different types and levels of degradation for both the training set and the test set, will become a benchmark for the research on handwritten music scores in the near future.

Fig. 3 :

 3 Fig. 3: TrainingSubset1 samples. From left to right: original image and two semisynthetic images generated from two different meshes.

Fig. 4 :

 4 Fig. 4: TrainingSubset2 samples. From left to right and top to bottom: original image and semi-synthetic images generated from the original image using the low, middle and high levels of local noise.

Fig. 5 :

 5 Fig. 5: TrainingSubset3 samples. First row, from left to right: images generated using mesh 1 and the low and high levels of local noise. Second row, from left to right: images generated using mesh 2 and the low and high levels of local noise.

Fig. 6 :

 6 Fig. 6: From left to right: an image from TrainingSubset3, its binary version and its binary staff-less version (ground-truth).

Fig. 7 :

 7 Fig. 7: Average music score competition results. The figure presents, for each participant method, the Average Accuracy and F-Measure of their method on the different sets.

Fig. 8

 8 Fig.8gives the values of the F-measures and the number of rejected images for each test subset and each level of degradation.Much can be learned from the scores presented in Table1. First, whatever the category (and intensity) of defect integrated in semi-synthetics images, the best results are mainly obtained with binary images (LRDE-bin, NUAS-bin-lin and INESC-bin). Only one method with grey-level image as input (LRDE-gray) gives 11 times the best results. It is also interesting that this method gives always the best recall score. The results showed that the baseline method[START_REF] Dutta | An efficient staff removal approach from printed musical documents[END_REF] performances are almost identical to those obtained by each winning method.Another interesting way to analyze these results is to compare the scores with the level of degradations of each sub dataset. Concerning 3D distortion, the staff lines images mapped on Mesh 1 (M1) were a bit more difficult to analyze for all the participants (except LRDE-gray and INESC-bin). On average the precision scores drop by 1 point. One method drops its precision by 7 points. We can conclude that these methods seems less robust to perspective deformation defects (Mesh 1) than to the presence of small curves and folds (Mesh 2). The precision scores of each participant decrease when the local noise is getting higher. On average, the precision scores decrease from 13%. In the best case it dropped by

Available at http://www.cvc.uab.es/cvcmuscima/

Acknowledgements

This research was partially funded by the French National Research Agency (ANR) via the DIGIDOC project, and the spanish projects TIN2011-24631 and TIN2012-37475-C02-02.