Nayane Macedo Portela Da Silva

Fabienne Espitalier
email: fabienne.espitalier@mines-albi.fr

Ange Nzihou

Precipitation Process of Calcium Phosphate from Calcium Carbonate Suspension †

Keywords: precipitation, hydroxyapatite, brushite, Raman spectroscopy

The Ca-HA is synthesized by reacting calcium carbonate (CaCO 3 ) and ammonium dihydrogen orthophosphate (NH 4 H 2 PO 4 ) in stoichiometric proportions. In this precipitation process, the ratio of Ca/P by moles is 1.67. The experiments were performed in a batch reactor at 25 °C. From control parameters, the pH and the temperature were measured in line for all experiments. The tests were performed with two liquid-solid mass ratios (H 2 O/ CaCO 3 ) of 3 and 5. For each mass ratio, three stirring rates of 260, 400, and 600 rpm were tested. Samples of the synthesis were collected at different intervals and analysed by laser granulometry and by environmental scanning electron microscopy. For the synthesis conducted at 260 rpm, synthesis monitoring was made using a contact probe in solution coupled to a Raman spectrometer in order to follow the formation of solid phase. This technique is valuable to follow the synthesis of Ca-HA in a concentrated solids suspension (around 20-30 wt%). The results make it possible to propose a mechanism of the precipitation process of Ca-HA. It can be divided into four main stages: (i) dissolution of calcium carbonate (CaCO 3 ), (ii) precipitation of brushite (CaHPO 4 .2H 2 O), (iii) transformation of brushite into Ca-HA (Ca 10 (PO 4 ) 6 (OH 2 )) and (iv) nucleation, growth and agglomeration of Ca-HA.

Introduction

The exponential population growth has led to a similar increase in requirements for resources, energy, food, housing and land as well as an enormous increase in industrialization and urbanization. This rapid industrialization and urbanization have led to great demand for natural resources and created an increase in waste production. A continuing issue facing society today is the disposal of municipal and industrial wastes [START_REF] Wey | Thermal treatment of the y ash from municipal solid waste incinerator with rotary kiln[END_REF]. Waste calcium carbonate (CaCO 3 ) from the paper, cosmetics, pharmaceuticals, catalyst, ceramics, rubber and paint industries often ends up in landfill [START_REF] Wen | Recycle of low chemical potencial substance resources[END_REF]. These CaCO 3 wastes can be used as the calcium compound in the synthesis of hydroxyapatite (Ca-HA). Ca-HA, Ca 10 (PO 4 ) 6 (OH 2 ), is an insoluble calcium phosphate mineral. It is a main constituent of bones and teeth. Due to its excellent biocompatibility, slow biodegradation, good mechanical stability, great sorption properties and heteroge-neous photocatalytic degradation under UV irradiation [START_REF] Wang | Nucleation and growth of apatite on chemically treated titanium alloy: an electrochemical impedance spectroscopy study[END_REF], it has been used extensively for bone repairs, bone implants, bioactive materials and for purification and separation of biological molecules and organic contaminants. It can accept a series of cationic and anionic substitution within its structure [START_REF] Elliott | Structure and Chemistry of the Apatites and Other Calcium Orthophosphates[END_REF]. Similarly, it has been reported to be an efficient adsorbent to treat heavy metals [START_REF] Chen | Effects of ph on heavy metal sorption on mineral apatite[END_REF] and to have a very high capacity for removing divalent heavy metal ions [START_REF] Ma | In situ lead immobilisation by apatite[END_REF]. Consequently, hydroxyapatite is used for the removal of heavy metals from contaminated soils, wastewater and fly ashes [START_REF] Elliott | Structure and Chemistry of the Apatites and Other Calcium Orthophosphates[END_REF]. Several methods have been developed to prepare Ca-HA, including a dry process, precipitation, hydrolyzation of calcium phosphate and hydrothermal synthesis. Spray pyrolysis, freeze drying, gel diffusion, a sol-gel technique and electrochemical deposition have all also been used. Much work has been done on the synthesis of Ca-HA using different methods such as solid-solid reaction [START_REF] Nakamura | Hydroxyapatite nano sol prepared via a mechanochemical route[END_REF][START_REF] Silva | Structural properties of hydroxyapatite obtained by mechanosynthesis[END_REF], ultrasonic spray freeze-drying [START_REF] Itatani | Preparation of various calcium-phosphate powders by ultrasonic spray freeze-drying technique[END_REF] or spray-drying techniques generating a Ca-HA powder of controlled morphology [START_REF] Luo | Preparing hydroxyapatite powders withcontrolled morphology[END_REF], sol-gel technique [START_REF] Anee | A novel technique to synthesize hydroxyapatite at low temperature[END_REF][START_REF] Bezzi | A novel sol-gel technique for hydroxyapatite preparation[END_REF]Salimi et al., 2012), precipitation in aqueous solution [START_REF] Jarcho | Hydroxylapatite synthesis and characterization in dense polycrystalline form[END_REF][START_REF] Rodriguez-Lorenzo | Fabrication of hydroxyapatite bodies by uniaxial pressing from a precipitated powder[END_REF][START_REF] Bailliez | The kinetics of surface area reduction during isothermal sintering of hydroxyapatite adsorbent[END_REF][START_REF] Swain | Synthesis and dispersion of hydroxyapatite nanopowders[END_REF], emulsion route [START_REF] Saha | Synthesis of nanocrystalline hydroxyapatite using surfactant template systems: Role of templates in controlling morphology[END_REF][START_REF] Okada | Solvent-free formation of hydroxyapatite coated biodegradable particles via nanoparticle-stabilized emulsion route[END_REF], microwaveassisted synthesis [START_REF] Ipekoglu | Silver substituted nanosized calcium deficient hydroxyapatite[END_REF] and synthesis under hydrothermal conditions [START_REF] Vasile | Physico-chemical and biocompatible properties of hydroxyapatite based composites prepared by an innovative synthesis route[END_REF][START_REF] Murakami | Hydrothermal synthesis of porous hydroxyapatite ceramics composed of rod-shaped particles and evaluation of their fracture behaviour[END_REF]. Among all these methods, precipitation is characterized by the simplicity of the process, its low cost and its easy application in industrial production [START_REF] Liu | Kinetics of hydroxyapatie precipitation at pH 10 to 11[END_REF]. The main objective of this work is to investigate this synthesis of calcium phosphate powders to be used as adsorbents for pollutants removal. To achieve this goal, the hydroxyapatite is synthesized by a crystallization reaction (precipitation) from solid calcium carbonate in a batch reactor in order to investigate systematically the effects of operating conditions such as the stirring speed and the liquid-solid mass ratios. These parameters determine the crystallization performance. Moreover in this study we show that the solid formations can be followed using Raman spectroscopy.

Experimental section

Materials and methods

In order to synthesize Ca-HA, laboratory-grade calcium carbonate (CaCO 3 ) powder (98 % pure) and ammonium dihydrogen orthophosphate powder (NH 4 H 2 PO 4 ) (94.4 % pure) were used. Both were obtained from Fisher Scientific U.K. and were used as received. The suspension of CaCO 3 was introduced into a 0.5 L glass vessel (diameter 10 cm) with double jacket equipped with a helicoidal stirrer (diameter 4.5 cm). Agitation was powered by a Eurostar IKA labortechnick 2000 motor. The reactor was equipped with four baffles (Fig. 1). The NH 4 H 2 PO 4 powder was dissolved for a minimum time of 30 min at ambient temperature. Then, this solution was gradually pumped into the reactor with a peristaltic pump (Master flex pump) at a flow rate of 11.11 cm 3 /min in order to maintain the pH value higher than 7. The temperature was controlled by a water bath (Julabo VC F30). The pH value was monitored throughout the duration of the experiment using a pH meter (Meterlab PHM210). The reaction continued for 72 h at atmospheric pressure and temperature (25 ± 2 °C). The ratio of Ca/P by moles was 1.67. Two different liquid-solid mass ratios (water/CaCO 3 solid) of 3 and 5 and three stirring rates (260, 400 and 600 rpm) were tested. Addition times were equal to 18.6 min and 18.0 min, respectively, for a mass ratio of 3 and 5.

Analytical methods

The reactor suspension, composed of calcium carbonate and calcium phosphates, was sampled at different intervals in order to characterize the solid and the solution.

Particle-size distributions: The particle size distributions were measured by laser diffraction with a Malvern Master Sizer 2000 instrument covering a size range of 0.1-2000 μm. Suspension samples were diluted and dispersed in water by ultrasonication for 4 min before measurement to disaggregate the solid. Direct sampling of the reactor suspension was used for these measurements.

For the other analysis, the collected samples were filtered. Solids were washed and dried in a vacuum oven (Heraeus Vacutherm-VT-6025) at 60 °C for 24 hours to yield a fine white powder. The dry solid was not very cohesive and it was simply broken up by manual grinding.

Ion chromatography was used to analyse the liquid, and TG-DSC and ESEM for the solids.

Ion phosphate concentration: The phosphate ion concentration was analysed using ion chromatographic separation and it is performed using a Dionex ICS-300 ion chromatograph system consisting of: a single pump (Dionex ICS-3000 SP), an eluent generator (Dionex ICS-3000 EG), a conductivity detector (Dionex Conductivity Detector P/N 061830), a precolumn (Dionex IonPac AG 19, 4 × 50 mm) and a column (Dionex IonPac AS 19 (4 × 250 mm). The eluent was the ultra-pure water with the pH value kept constant by using the KOH eluent generator. The eluent flow was 1 ml/min corresponding to a stable pressure in the column of about 120 bar. The results were analysed by using the software Chromelon.

Habitus: A scanning electron microscope (Philips XL30 ESEM FEG) equipped with EDAX fluorescence analysis following platinum sputtering of the sample, was used to study the morphology of the gold-coated intermediate and final products.

Weight losses: Thermal gravimetric analysis was performed using a Thermogravimetrical Analyser (TGA-DSC 111 Setaram). The experiment was performed with a heating rate of 2 °C/min using nitrogen flow (25 ml/min). The percentage weight loss of the samples (approximately 12 mg) was monitored from 20 to 800 °C.

Raman spectra: For the synthesis conducted at 260 rpm, the synthesis monitoring was conducted using a contact probe in solution coupled to a Raman spectrometer RXN2C785, Mettler-Toledo/Kaiser analysers (Kaiser Optical Systems, Ann Arbor, Michigan, USA). A 400 mW external cavity stabilized by an Invictus NIR diode laser at 785 nm was used for sample illumination. All the spectra were recorded at a resolution of 4 cm -1 .

XR diffraction: For the synthesis conducted at 260 rpm, the phase identification of some samples was carried out using XR diffractometry (Phillips PanAlytical X'Pert Pro MPD diffractometer), which had a Cu Ka radiation (l = 1.54 A) at 45 kV and 40 mA. The reflections were collected in the 2θ ranges from 10 to 75° with a step size of 0.17° and a time of 13.02 s per step. The phase identification was carried out with the JCPDS database.

Results

During the process for the formation of hydroxyapatite, it was possible to form other calcium phosphates. Their solubilities decrease with the pH value [START_REF] Ferreira | The different phases in the precipitation of dicalcium phosphate dihydrate[END_REF]. The comparisons of the solubility of the CaHPO 4 .2H 2 O named brushite and the solubility of hydroxyapatite Ca-HA for a pH value lower than 4, show that brushite is a stable compound and for a pH value higher than 4, the Ca-HA is prominent. Fig. 2 shows the evolution of the pH value during the synthesis for the two liquid-solid mass ratios (3 and 5) at 400 rpm. It can be observed that the pH value readily decreases from 10.2 to 6.9 during the first 15 minutes of reaction. Then it gradually increases to 8.5 and after 40 hours, the pH value stabilizes at about 9. The liquid-solid mass ratio does not influence the pH value.

At 25 °C in the pH range [8-9] (Fig. 2), it was possible to form three solids: the monetite (CaHPO 4 ), the brushite (CaHPO 4 .2H 2 O) and the hydroxyapaptite (Ca 10 (PO 4 ) 6 (OH 2 )).

Their formation follows these three reactions respectively [START_REF] Chkir | Synthèse de gels phosphocalciques issus de déchets industriels carbonates[END_REF][START_REF] Verwilghen | Convenient conversion of calcium carbonate to hydroxyapatite at ambient temperature[END_REF]:

(1) Monetite (metastable phase):

CaCO 3 + NH 4 H 2 PO 4 → CaHPO 4 + CO 2 + H 2 O + NH 3
(2) Brushite:

CaCO 3 + NH 4 H 2 PO 4 + H 2 O → CaHPO 4 .2H 2 O + CO 2 + NH 4 OH
(3) Hydroxyapatite:

10CaCO 3 + 6NH 4 H 2 PO 4 → Ca 10 (PO 4 )6(OH 2 ) + 10CO 2 + 2H 2 O + 6NH 4 OH (4) Transformation of brushite into hydroxyapatite 10CaHPO 4 .2H 2 O + 4NH 4 OH → Ca 10 (PO 4 ) 6 (OH 2 ) + 4NH 4 H 2 PO 4 + 18H 2 O
The analysis did not detect monetite.

When any amount of ammonium dihydrogen phosphate is introduced into the reactor containing the calcium carbonate suspension, gas bubbles with an ammonium odor are observed. This confirms the formation of carbon dioxide and ammonium during the reaction.

As the pH showed low values at the beginning of the synthesis, this may have favored the formation of brushite. The end of synthesis, which sees the stabilization of the pH value at around 9, favors the formation of hydroxyapatite.

Influence of mass ratio

Two syntheses with two liquid-solid mass ratios of 3 and 5, respectively, carried out at a low stirring rate of 260 rpm in the presence of a RAMAN probe to study the influence of this ratio on the synthesis of Ca-HA. [START_REF] Tsuda | Raman spectra of human dental calculus[END_REF] give the Raman bands observed for three pure minerals (brushite, octacalcium phosphate OCP and Ca-HA). The spectral features of solids can be characterized as Raman bands due to phosphate (PO 3 4-) vibrational modes of ν 4 , ν 2 , ν 1 and ν 3 . The intensity of the ν 1 , band is the strongest, and can be used for the identification of minerals.

Figs. 3,4 and 5 show the Raman spectra during the synthesis made with a mass liquid-solid ratio of 5 and a stirring rate of 260 rpm. In Fig. 3, we observe the appearance of a peak at 983 cm -1 after 20 min of synthesis. This peak is relatively wide. Its intensity increases with time. This peak noted on Fig. 4 corresponds to the brushite, a compound observed in previous studies [START_REF] Chkir | Synthèse de gels phosphocalciques issus de déchets industriels carbonates[END_REF].

The presence of brushite is confirmed by XR diffraction analysis (Fig. 6).

After 19 h, a second peak close to the brushite peak appears at 962 cm -1 ; this corresponds to the Ca-HA. While the peak intensity of brushite decreases, the peak inten- sity of Ca-HA increases (Figs. 4 and5). After 30 h, the intensities of all peaks are stable. This assumption has been verified qualitatively by XR diffraction analysis (Fig. 7).

At 44 h, the brushite is no longer visible by this technique. The large peak observed at 30-33° corresponds to Ca-HA. Fig. 8 shows the weight losses of the different samples measured between 20 and 800 °C by TG analysis for dry solids obtained during the synthesis made with a mass ratio of 5 and a stirring rate of 260 rpm. For the pure cal-cium carbonate, two weight losses are observed: one between 350 and 400 °C and another after 600 °C. The first loss is due to the degradation of impurities contained in CaCO 3 . The decomposition of calcium carbonate (CaCO 3 ) in calcium oxide (CaO) and carbon dioxide (CO 2 ) (decarbonation) induces the second loss.

For the other samples, three weight losses are observed. The first is between 180 and 200 °C, the second between 350 and 400 °C and the third after 600 °C. This first is attributed to loss of the hydration water of brushite (detected by RAMAN spectra analysis), the second to impurities and the third to the decarbonation of unreacted CaCO 3 . The weight loss observed between 50 and 120 °C is due to insufficient drying of these samples. From these mass losses, by assuming that only three solids are present in the reactor (CaCO 3 , CaHPO 4 .2H 2 O and Ca 10 (PO4) 6 (OH 2 )), the mass percentage of these three components in the solid can be calculated (Fig. 9). Four steps can be observed. The first step (I) corresponds to the dissolution of calcium carbonate and formation of apatite: a sharp decrease in calcium carbonate (around 40 %) during the first hour, and a sharp increase in brushite (around 40 %) and Ca-HA (around 20 %).

During the second step (II) between 4 and 20 h, the weight percentage of solids seems to be constant. After 20 h, we can observe an increase in the weight percentage of Ca-HA up to 80 wt% and a decrease in the CaCO 3 and brushite weight percentages. This step (III) corresponds to the transformation of brushite into Ca-HA by dissolutioncrystallization or by solid-solid transformation. However, the reaction is not complete and 20 wt% of the solid remaining is composed of calcium carbonate. After 68 h, two solids are in suspension: CaCO 3 and Ca-HA. The mass balance on calcium leads to the total weight as a function of time from mass fractions and initial weight of CaCO 3 : Fig. 10 presents the evolution of weight of each solid during a synthesis for both liquid-solid mass ratios, 3 and 5.

Furthermore, by coupling the results of the loss of mass and the RAMAN spectra, it is possible to calibrate the area of RAMAN peaks observed in 983 and 962 cm -1 , respectively, for the brushite and the Ca-HA according to the mass fractions obtained by TGA. This calibration leads to the continuous curves given in Fig. 10. Both weight ratios lead to final masses that are close to CaCO 3 initial masses. The conversion does not seem to depend on the initial weight ratio. This conversion is not total, but is of the order of 73-74 wt%. We also note that the maximum mass of brushite formed for a liquid-solid mass ratio equal to 3 is 13 times weaker than that formed for a liquid-solid mass ratio of 5 (Fig. 10 (c)).

Beyond 20 h for a liquid-solid ratio of 3 and 40 h a liquid-solid ratio of 5, the brushite seems to have totally disappeared. In parallel, the formation of Ca-HA is much faster for an L/S ratio of 3 within the first hours (77 wt% of the final mass is reached (Fig. 10 (a)). After 10 h of synthesis, a slowing down is observed.

The particles of brushite are formed preferably in the solution (Fig. 11), while the particles of Ca-HA are created on the surface of the brushite particles (Fig. 12) or (Fig. 11). The particles of CaCO 3 are attacked by orthophosphate ions. Therefore, CaCO 3 is used to form monetite. This compound is transformed into brushite which evolves to the hydroxyapatite (calcium phosphate thermodynamically more stable). When the liquid-solid mass ratio decreases, the available surface of CaCO 3 is larger, allowing the reaction (3) to occur on the surface of CaCO 3 . This behavior limits the brushite formation by reaction (2). The mass growth rate of Ca-HA at the initial time can be written as:

Ca-HA j g 0 d d t m K S C t = = ( 2
)
where K is a kinetic constant, ΔC is the absolute supersaturation of Ca-HA, and S is the CaCO 3 disposable surface at the time at which Ca-HA is detectable.

As the initial size distribution of calcium carbonate is identical, the surface (S) is assumed proportional to mass.

The ratio of these initial rates for liquid-solid mass ratios of 3 and 5 leads to a coefficient j of the order of 12.5, thereby showing a strong influence of the available surface of calcium carbonate on the Ca-HA formation. As the Ca-HA layer forms on the surface of CaCO 3 , the diffusion of orthophosphate and calcium ions is limited thus reducing the mass growth formation of Ca-HA observed beyond 10 h for a liquid-solid mass ratio of 3. In the case of a higher ratio, it can be assumed that reactions (2) and (4), which represent the CaCO 3 and brushite formations, respectively, occur in parallel.

Fig. 14 shows the evolution of the mass mean diameter as a function of the time based on both mass ratios (liquid-solid mass ratios of 3 and 5) for a stirring speed of 260 rpm. For a liquid-solid mass ratio of 5, the increase in average diameter (7-22 microns) between 0 and 3 is due to the brushite formation (Fig. 14 (a)). After 3 h, the diameter decreases assumed by brushite dissolution. Beyond 20 h, the mean diameter increases following the same trend as Ca-HA mass (Fig. 14 (b)).

Brushite dissolution allows Ca-HA formation. This compound is thermodynamically more stable than brushite [START_REF] Ferreira | The different phases in the precipitation of dicalcium phosphate dihydrate[END_REF] and it forms on the surface of CaCO 3 causing the increase in diameter. For a liquid-solid mass ratio of 3 (Figs. 15 (a) and (b)), the same behavior is observed, meaning that the mass mean diameter increases, then decreases and finally increases again. However, beyond 20 h, the mean diameter does not change (about 20 μm) because the Ca-HA mass stabilizes. Fig. 16 presents the mean diameters of the particles during synthesis for different stirring speeds (400 and 600 rpm) for both mass ratios (L/S = 3 and 5). The results obtained at 400 and 600 rpm show the same behavior obtained at 260 rpm (increase, decrease, re-increase and stabilization of mean diameter). After 40 h of synthesis, the measured diameters are identical (23 ± 1 μm) for a stirring speed equal to 400 rpm (Fig. 16 (a)). Larger agglomerates are observed at a low stirring speed after 72 h of synthesis. The stabilization of the mean diameter is observed after 40 h for all the syntheses except for the synthesis performed at 600 rpm. In this case after 20 h of synthesis, the mass mean diameter is stable and equals 16 ± 1 μm (Fig. 16 (b)).

Conclusions

In this study, we show that Raman spectroscopy is a valuable technique to follow the synthesis of Ca-HA in a concentrated solids suspension (around 20-30 wt%). This technique makes it possible to follow the formation of Ca-HA and brushite and to deduce from complementary measurements a formation mechanism described as fol- 16 Mean volume diameter during the synthesis, symbols: black circle (liquid-solid mass ratio = 3) and hollow circle (liquid-solid mass ratio = 5).

lows:

• The dissolution of calcium carbonate is observed in the first minutes of synthesis. • The precipitation of brushite and Ca-HA particles is observed in the first four hours of synthesis. • The transformation of brushite into hydroxyapatite begins after one day. It seems to be the limiting step. In terms of morphology, agglomerates of hollow ellipsoidal particles were obtained corresponding to the calcium carbonate, while platelets agglomerated like stars corresponded to the brushite and agglomerated spongy spheres or round stick-like compounds of spongy spheres corresponded to the hydroxyapatite.

Through the mass balance obtained from thermogravimetric analysis and from the phosphate ions, it was also verified that at the end of the synthesis, the calcium carbonate was not completely consumed.

Therefore, the reaction is not complete at the end of the synthesis (72 h).

From the analysis performed by laser granulometry during the synthesis with a mass equal to five and three at 260, 400 and 600 rpm, there was a strong influence of the stirring speed on the process. The two effects (positive and negative) that explain the agglomeration process have been presented. For the synthesis with a mass ratio equal to five, the first effect called the size-enlargement phenomenon predominated. In contrast, for the synthesis with a mass ratio equal to three, a predominance of the second effect called the breakage phenomenon was observed. These phenomena were confirmed by the observations made using the environmental electron microscopy of the synthesized solids.

Fig. 1

 1 Fig. 1 Experimental set-up for precipitation.

Fig. 2

 2 Fig.2Evolution of the pH value during synthesis at 400 rpm for two mass ratios (3 □ and 5 ■).

Fig. 3

 3 Fig. 3 Raman spectra measured every 19 min between 0 and 3 h, (liquid-solid mass ratio = 5 and N = 260 rpm).Times from the bottom to the top.

Fig. 4

 4 Fig. 4 Raman spectra measured every 4 h between 3 h 10 minand 24 h 19 min, (liquid-solid mass ratio = 5 and N = 260 rpm). Times from the bottom to the top.

Fig. 5

 5 Fig. 5 Raman spectra measured every 8 h between 19 h 20 min and 91 h 20 min, (liquid-solid mass ratio = 5 and N = 260 rpm). Times from the bottom to the top.

Fig. 6

 6 Fig.6XR diffraction spectra at 0.75, 2 and 4 hours (liquidsolid = N = 260 rpm). Times from the bottom to the top.

Fig. 7

 7 Fig.7XR diffraction spectra at 44, 50 and 68 hours (liquidsolid mass ratio = 5 and N = 260 rpm). Times from the bottom to the top.

Fig. 8

 8 Fig. 8 Weight losses between 20 and 800 °C at 0, 0.75, 4, 20, 44, 50, 68 and 72 h (liquid-solid mass ratio = 5 and N = 260 rpm).

  weight of CaCO 3 , M j the molar weight and w j the mass fraction of the component j in the solids mixture.

Fig.

  Fig. Mass percentages of brushite, calcium carbonate andCa-HA during the synthesis.

Fig. 11

 11 Fig. 11 Morphology of brushite at 0.75, 4 and 20 h (liquidsolid mass ratio = 5 and N = 260 rpm).

Fig. 12

 12 Fig. 12 Morphology of Ca-HA at 44 h (liquid-solid mass ratio = 5 and N = 260 rpm).

Fig. 13

 13 Fig. 13 Morphology of Ca-HA 50 and 72 h (liquid-solid mass ratio = 5 and N = 260 rpm).

Fig. 14

 14 Fig. 14 CaHPO4.2H2O and Ca-HA masses calculated from RAMAN spectra and mean volume diameter during synthesis (N = 260 rpm and liquid-solid mass ratio = 5), symbols: hollow circle (d43) and --(compound masses).

Fig. 15

 15 Fig. 15 CaHPO4.2H2O and Ca-HA masses calculated from RAMAN spectra and mean volume diameter during synthesis (N = 260 rpm and liquid-solid mass ratio = 3), symbols: black circle (d43) and --(compound masses).

Acknowledgements

The authors would like to thank M. H. Lucas for support in the data analysis of RAMAN spectra. They also thank Ms S. Del Confetto, Ms C. Rolland, Ms N. Lyczko and M.P. Accart, respectively, for TG measurements, microscopy analysis and X-Ray, and particle size analysis.


Adeline Granet

Emmanuel Morin

Harold Mouchère

Solen Quiniou

Christian Viard-Gaudin

Harold Mouch Ère

Transfer Learning for a Letter-Ngrams to Word Decoder in the Context of Historical Handwriting Recognition with Scarce Resources

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

An increasing amount of handwritten historical documents is becoming digitally available, as a mean for preserving this heritage and making it accessible to all. However, digitization is not sufficient to make the documents usable: it is necessary to extract informations in order to index them. Researchers and historians in the humanities and in the social sciences need to be able to query them and find answers rapidly. Therefore, new projects involve the domains of language processing, document recognition, and information retrieval for historical studies.

In the field of Natural Language Processing (NLP) for historical documents, the main challenge is currently to analyse and normalize the spelling of texts [START_REF] Garrette | An unsupervised model of orthographic variation for historical document transcription[END_REF][START_REF] Bollmann | Learning attention for historical text normalization by learning to pronounce[END_REF] whereas challenges of Computer Vision for historical documents are more diverse: they involve segmentation into words, lines, or paragraphs, as well as keyword spotting, or handwriting recognition (HWR). In recent years, the number of competitions regarding historical documents has increased [START_REF] Cloppet | ICFHR2016 Competition on Classification of Medieval Handwritings in Latin Script[END_REF][START_REF] Pratikakis | ICFHR2016 Handwritten Document Image Binarization Contest (H-DIBCO 2016)[END_REF][START_REF] Sanchez | IC-DAR2017 Competition on Handwritten Text Recognition on the READ Dataset[END_REF]. Such systems have to deal with the complexity of the task as well as the document medium, its level of deterioration, or even its written language. All of these can have a strong impact on the system efficiency.

Lately, the trend in HWR is the use of deep neural networks and, more recently, the integration of attention models in the networks [START_REF] Bluche | Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention[END_REF]. The handwriting systems are built with a Multidimensional Long Short-Term Memory (MDLSTM) network or even with a Convolutional Recurrent Neural Network (CRNN) stacked with a Bidirectional Long Short-Term Memory (BLSTM) [START_REF] Granell | Transcription of spanish historical handwritten documents with deep neural networks[END_REF]. The neural network training includes a Connectionist Temporal Classification (CTC) cost function proposed by [START_REF] Graves | Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks[END_REF]. Those approaches enable the use of all the available contexts. To decode sequences, there are several strategies: a dictionary, a language model, or a Weighted Finite State Transducer (WFST) including several dictionaries. Nevertheless, results are constrained by the size of the vocabulary used. When too many words are out-of-vocabulary, the results are degraded. To improve the results, methods can be employed to increase the size of the vocabulary by using Wikipedia or other available resources. Whether improving the results or training the networks, the HWR systems require a lot of data.

We are now working on a new resource (the financial records of the Italian Comedy) with no groundtruth. The changing layout of these documents, the multilingualism (French and Italian), and their special writing make them more complex to study. Moreover, an annotation step would be time-consuming for experts to create a ground-truth. Therefore, transductive transfer learning is an interesting approach when no ground-truth is provided. Indeed, it uses different sources of data to train a system for a specific task by applying various target data [START_REF] Sinno | A survey on transfer learning[END_REF]. It enables us to use available existing resources to annotate unknown data. It is eagerly used to supply the lack of data for greedy systems such as word spotting in historical documents [START_REF] Lladós | On the influence of word representations for handwritten word spotting in historical documents[END_REF] or translation models with multimodal systems such as [START_REF] Nakayama | Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot[END_REF].

The standard of machine translation is an encoder-decoder system based on a recurrent neural network [START_REF] Cho | On the Properties of Neural Machine Translation: Encoder-Decoder Approaches[END_REF]. The first component encodes the source language into a fixed vector and the last one decodes the sequence into the target language. Similarly, [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF] proposed a neural image caption generator that consists of two sub-networks: a pre-trained Convolutional Neural Network (CNN) encoding an image into a fixed size vector, using the last hidden layer from GoogleNet, and a LSTM model generating the corresponding description. We were inspired by this idea to split up our handwriting recognition system into an encoder and a decoder. First, an image encoder takes a word image as input and, thanks to a fully convolutional network (FCN), converts it into a vector of letter-ngrams as in [START_REF] Huang | Learning deep structured semantic models for web search using clickthrough data[END_REF][START_REF] Bengio | Word embeddings for speech recognition[END_REF]. Here, the transfer learning is performed during the training step that uses available labelled image data. Then, the decoder is built with a recurrent layer in order to generate a sequence of characters from the vector of letter-ngrams. Here, a second transfer learning is independently made on the vocabulary. Contrary to the machine translation and description generator works, the sub-networks are trained independently from each other.

In this paper, we investigate the robustness of the decoder part to infer rules on the order of characters from the letter-ngrams vector only. Within this context, we present three contributions:

1. we propose a character-based decoder and evaluate it on the historical vocabulary of the Italian Comedy; 2. we show that historical resources are more appropriate than Wikipedia to complement the vocabulary of historical documents; 3. we also show that the transfer learning across languages and periods of time works based on a bag of letter-ngrams. This paper is organized as follows. We describe our transfer model within the encoder and decoder components in Section 2. Section 3 presents the datasets used and their preprocessing. The experimental setups are presented in Section 4 which is followed by the report of our experiments and results in Section 5.

Transfer Models

In this section, we present the architecture of our system. We aim to build a handwriting recognition system for multilingual historical documents using multiple resources with different languages and different creation periods.

Prior Work

Our first approach was based on a BLSTM-CTC network intergrating the transfer learning (Granet et al., 2018b). Firstly, an image is provided to a fully convolutional neural network to automatically extract features. Then, the extracted features were given to a BLSTM network. The cost was computed by the CTC activation function. The CTC enables the use of an unsegmented input image at a word or a character level and provides a sequence of characters as output. We implemented and evaluated this system with a training and test data from the same resource and, then, used the system on unknown data. The model did not use a dictionary nor a language model during the decoding. The resources were in French, Spanish, and English for the training set and in French, and Italian for the target datatest. The results of this system showed that a language model was learned implicitly to generate the sequence of characters and that the correctly recognized words were words close from one language to the other. Therefore, without a specialized training on a small part of the data to be transcribed, the transfer learning did not work well. The model allowed to reach only 10% of characters which are correctly recognized.

Figure 1: Overview of our encoder-decoder using transfer learning on new historical documents. On the left, the yellow component corresponds to the image encoder from which the features are extracted thanks to a FCN and two connected layers with 1,024 units each and a vector of letter-ngrams is built. On the right, the pink component corresponds to the sequence generation decoder that provides the word, character by character, corresponding to the input vector.

From these results, we concluded that a BLSTM-CTC network was unsuitable for the transfer learning case.

Word classification could have been a solution. Nevertheless, in the field of transfer learning, this strategy quickly reaches its limits when working on target data with a closed vocabulary with many named entities, and a significant number of out-of-vocabulary words. Therefore, classification could not be applied to our data.

Encoder-Decoder for Transfer Learning

Based on the work on image description generation, we define a new system that is divided into two complementary components: an image encoder and a character decoder, as shown in Figure 1. The first one encodes a word image into a vector and the second one decodes the vector into a character sequence or word. The interface vector that links these two parts is a bag of characters or letter n-grams. We estimated the set of letter-ngrams on all resources used for both the training data and the transfer target data. The originality in using a letter-ngrams vector as a pivot is that we encode the input into a non-latent space which is transferable as long as the training data and transfer target data share the same alphabet. In this paper, we focus on the character-based decoder to generate corresponding words. Nonetheless, it is interesting to give informations on the whole system to understand the overall approach.

Image Encoder We define our encoder as shown on the left part of Figure 1: • a fully convolutional network;

• 2 fully connected layers with a ReLU activation function [START_REF] Nair | Rectified Linear Units Improve Restricted Boltzmann Machines[END_REF] and 1,024 hidden units each; • one last fully connected layer with a sigmoid and L+1 units where L is the number of estimated letter-ngrams and the additional unitias a joker if the letter-ngram is unknown. The last layer with the sigmoid activation function allows us to have a probability for each letter-ngram independently of the others.

Character-Based Decoder We turned to neural encoder-decoder architectures in NLP as it enables us to map one sequence to another one without having them to have the same length or word order such as in machine translation [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] and description generation [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF]. We define the architecture for the character-based decoder as shown on the right part of Figure 1: • one fully connected layer with a ReLU activation function;

• one Gated Recurrent Unit layer (GRU) [START_REF] Cho | On the Properties of Neural Machine Translation: Encoder-Decoder Approaches[END_REF];

• one fully connected layer with a softmax activation function. These components were designed to be the most minimalist. Moreover, they do not use an embedding layer since it might interfer with the transfer learning that has to deal with different languages.

For the sequence generation task, we used a recurrent layer to add a temporality followed by a softmax layer that gives one character at the time until the word end symbol ] is given. We chose not to include a bidirectional recurrent layer contrary to encoder-decoder models that are used in machine translation. Since our character-based decoder uses a vector without any order indication on characters from words, all the information and context are included in the vector. Therefore, a bidirectional recurrent layer is useless. Overall the system must infer sequences of characters from the vector content. If we analyse the simple short word "are", it consists of 3 unigrams, 4 bigrams, and 3 trigrams. This decomposition includes the n-grams with the beginning and end symbols of the word. So the decoder needs to deduce the order of characters from the letter-ngrams.

Dataset Collection and Statistics

In this section, we present the resources used in our experiments as well as their representation.

Target Data: Financial Records of the Italian Comedy

The studied documents consist of more than 28,000 pages of financial records of the Italian Comedy from 1716 to 1791. These pages give the list of daily, monthly and annual receipts, with details on the composition of the troupe of actors for each season. We noticed several evolutions over the years:

• the language switches from Italian (with several dialects) to French;

• the structure changes but keeps the same amount of information;

• the writing style is irregular at the beginning of the century but, after that, a cashier was nominated so the style became stable. Figure 2 shows all the information that can be found in a daily record such as the date, the titles of the plays, the revenues and expenses, the actor names, and also some notes. Our study focuses on the title field which contains the list of plays that were performed that day. This list can be extended by some indications as if it was the first performance of the play, if the performance took place at the king's court, or if a new actor started to play. An example of this additional information is shown in Figure 2: the title field explains that it was a performance of "Sophie ou le mariage caché" (lit. "Sophie or the secret marriage") which was a comedy in three acts preceded by "Arlequin toujours Arlequin" (lit. "Arlequin always Arlequin"). Moreover, a title can be written in many ways with the complete title or a shorter title. Titles were mostly constituted by named entities such as "Raton and Rosette" or "Zemire and Azor".

The language and writing style are essential for the decoder part. We explained above that the language can be either Italian or French. Compared to the modern writing of these languages, we note some differences such as special characters like the long form of 's', the evolution of the 'y' spelling to 'i' or using 'i' and 'j' indiscriminately, or some weak and strong abbreviations. We define as weak abbreviation, a symbol that replaces few characters of a word, while a strong abbreviation reduces a long title to only two or three words.

Thanks to recent works and a participative annotation website (Granet et al., 2018a), we collected and validated 971 annotated title lines. We used all of these annotations to test our decoder system: this dataset is thereafter called Italian Comedy Registers (ICR).

Training Datasets

In order to build an efficient decoder, we must carefully select the training datasets. New Dataset (Frinken et al., 2013) built a very large vocabulary gathered from a Google N-grams project and an edition of a manually transcribed book from the 16 th century. We had a similar approach to this work as we built a new dataset from digitized books of the same century of our data, with a similar closed vocabulary. To preserve the cultural heritage for future generations, copies of books have been digitized, automatically OCRized, and distributed via Google Book, for example. We selected 23 books dealing with the Italian Comedy from (and published in) the 18 th century. These books include bilingual scripts of plays (French and Italian), collections of plays, and anecdotal stories of the Italian theater. The extracted texts were cleaned to remove the noise such as the structure of the texts and special characters. This new resource, called Google Italian Comedy (GIC), has the advantage of having the same closed vocabulary with specific writings as in the records of the Italian Comedy.

Existing Datasets For our transfer learning approach, we carried out experiments with four available datasets of images, sharing at least one common feature with our target data, as well as Wikipedia to compare the effect of a modern and general resource on the Italian Comedy data:

• RIMES (RM) stands for Recognition and Indexing of Handwritten Documents and Faxes) and is a French database developed to evaluate automatic systems that recognizes and indexes handwritten letters [START_REF] Grosicki | ICDAR 2011 -French Handwriting Recognition Competition[END_REF]); • George Washington (GW) is an English database created from the George Washington Papers at the Library of Congress [START_REF] Fischer | Lexicon-free handwritten word spotting using character HMMs[END_REF]); • Los Esposalles (ESP) is a Spanish database compiled from a marriage licence book collection from the 15 t h and 17 t h centuries [START_REF] Fischer | Lexicon-free handwritten word spotting using character HMMs[END_REF]); • Wikipedia Fr data (Wiki), as used and distributed by [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF], provides all words whose frequency is greater than 5 in Wikipedia. From this dataset, we randomly selected 30 000 words. To our knowledge, no other experiments dedicated to the construction of a decoder use the resources RM and ESP which are initially image resources. However, we found it interesting to observe the impact of trained models because ESP is mainly composed of named entities, and RM is a contemporary base with an administrative vocabulary. Therefore, they are far from our Italian Comedy data.

Representation

Preprocessing of the Training Datasets In all the datasets, we replaced the diacritical marks by their simple form, such as {é, è} are e characters, and the typical long form of s from the 18 th as a short form for the decoder input and also the target data. All words are case sensitive. For GIC only, text lines were split on whitespaces and punctuation marks. In contrast, ICR title lines were only split on whitespaces and we kept with the punctuations in the data.

Letter-Ngrams As suggested by [START_REF] Bengio | Word embeddings for speech recognition[END_REF][START_REF] Vania | From Characters to Words to in Between: Do We Capture Morphology[END_REF], we use letterngrams as a pivot in our multimodal encoder-decoder. Initially, the authors selected the 50 000 most popular letter-ngrams to represent words. Adding [ and ] to symbolize the beginning and end of a word, respectively, we computed all the possible letter-ngrams with a maximum length of 3 on all the datasets. Therefore, we selected around 12,500 letter-ngrams. The large difference in the number of ngrams comes from the choice of keeping only letter-ngrams appearing in at least 2 different datasets: a joker was created to replace the non-selected letter-ngrams. With our example in Figure 1, the name Sophie is thus decomposed as {S,o,p,[S,So,op,[So ,. . . ,ie],e]}.

Regarding the choice of having at most 3-grams, this allows to give the system an idea on the order in which the characters are arranged in the word. In the case where there are only unigrams, the learning process is longer and tends towards over-learning to be able to have sequences of characters forming words. Including 4-grams into the vector would dramatically increase its length and would make it more difficult to select the ngrams recognized in a word by the optical model.

Decoder Input and Output

The input of the decoder input consists of the normalized frequencies of the n-grams in the input word. The frequencies are normalized by the length of the word to provide a vector with values between 0 and 1, corresponding to probabilities for each n-gram. Using frequencies enables us to hold information about the word length and to compensate the lost temporality. For the sequence generation, we have 79 output units for all the upper and lower characters, the digits, the punctuation symbols including the space, the start and the end of word. The last added symbol is a blank label that allows the system to provide an out-of-character answer after the end of a word.

Experimental Setup

This section presents the experimental setup of our decoder as well as the evaluation metrics we used.

Decoder Training

The size of the datasets varies from 660k to 25k words. For this reason, we trained the system with either one dataset or several datasets, except for GW. Its size is too small so we always combined it with other datasets.

To avoid overfitting, we used the classical technique of early stopping. The training process was stopped after five epochs if the validation loss did not decrease anymore. For the sequence generation, we used the loss defined by Equation (1) that computes the multi-classification with a normalization through all the output units.

L i = - j t i,j log(p i,j )
(1)

Decoder Parameters

We defined the structure of the decoder part with 1,024 units in the fully connected layer to extract features, 500 hidden units in the GRU layer, and 79 units with the Softmax activation function on the last layer. To be sure to stay in the same configuration, we used the Adam function that controls the learning rate which is initialized at 0.0001 for the stochastic optimization. As we have less data than the others experiments of the state-of-the-art, our rate is lower. In the scope of sequence generation, we padded the length of the sequence with blank labels up to 50 characters. Therefore, the network is free to define any sequence without any constraints.

Evaluation Metrics

To evaluate the performance of our system, we used the recognition rate at the character level (CRR) and at the word level (WRR). The Character Recognition Rate is defined as:

CRR = N -(Ins + Subs + Dels) N
with N the number of characters of the reference words, Subs the number of character substitutions, Dels the number of character deletions, and Ins the number of character insertions. The Word Recognition Rate corresponds to the Word Accuracy and is defined as the number of correctly recognized words divided by the total number of words to recognize. The WRR is computed with and without using the Levenshtein edit distance to correct the output sequence with a multilingual dictionary. The dictionary was built from the training and the validation parts of the vocabulary of all the datasets, except Wikipedia. The dictionary contains 39,051 words. We also computed the lexical coverage of a training set with respect to the test set. It is defined as the number of words shared between the training set and the test set, normalized by the size of the test vocabulary. This metric defines a high boundary for the word recognition using a dictionary. Since the dictionary is built on the training and the validation sets, out-of-vocabulary words of the test set could not be correctly recognized.

Baseline Approach

As a baseline approach we used a vector of unigrams to represent each word in the decoder input. We added the symbols for the beginning and the end of words so each word is represented by 2 bigrams and several unigrams.

Results and Analysis

In this section, we present the results of our experiments for the sequence generation decoder, in Table 2 which is divided into 3 parts following the test resources. We performed three types of experiments corresponding to the sub-parts of the The experiments are done with the artificial true n-gram vectors so the input is not noisy. Overall, the results in Table 2 prove that using letter-ngrams is a better choice than only using unigrams and the use of a dictionary degrades the results. In all the experiments, the models provide a CRR greater than 75% on the validation sets and greater that 85% on the test sets, even if the lexical coverage is less than 25%. Furthermore, WRR also exceed the lexical coverage. To decode the ICR vocabulary, the model must be trained with at least GIC as a resource. All following analyses are presented on the validation sets to evaluate the best ressources to train the model. The last analysis relates to the test sets.

Baseline Whether experimenting with RM on RM (validation and test set) or with GIC on GIC (validation set) and ICR (test set), we found that the results obtained with letter-ngrams are better than with unigrams. On GIC, letter-ngrams dramatically outperformed unigrams with a 70% relative increase of WRR without the dictionary. It should also be noted that only 5% of the characters were incorrectly recognized with letter-ngrams. These results corroborate other studies using trigrams such as [START_REF] Vania | From Characters to Words to in Between: Do We Capture Morphology[END_REF]. Use of a Dictionary Using the dictionary drops performances except for Expe. Id 6, 7, 8, and 20. We notice that when the lexicon coverage is greater than 20%, WRR always remains lower. We notice that a dictionary misleads the decoder when it generates a correct sequence but there is only another form (e.g., plural) of this word, in the dictionary. However, the dictionary sometimes helps to get closer to the original word even if this word does not exist in the dictionary and if the training and the test data are from different periods or different languages. Nevertheless, these cases are too rare to improve CRR with the dictionary.

Results on ICR (our Target Data) We focus on the central part of the ICR experiments, where GIC is combined with other resources. The CRR and WRR results are very similar to those of the training step with only GIC (Expe. Id 2 vs. Expe. Id 3, 4, and 5). Therefore, the increase in the number of resources does not always have a great impact on the performance. Nonetheless, training on GIC and ESP achieved better than GIC and RM despite the fact that languages are mixed.

For the transfer learning process with an out-of-domain vocabulary (Expe. Id 6, 7, and 8), the lexical coverage is extremely low (around 15%). Nevertheless, the system reaches a maximum of 19.49 % WRR using only RM and 41.18 % WRR using Wikipedia, which are the two modern French resources.

Results on Rimes (RM) These results are impressive because Rimes is the only modern image base we used. For transductive transfer learning, without RM during the training step, CRR and WRR reach the same results obtained with only RM as a training data. Moreover, they exceed the lexical coverage of 15%. When we work with historical training data applied to modern data but with the same language, the lexical coverage is greater than using a modern training data and testing on ICR. It seems that the historical spelling is easier to extend to the modern spelling for the decoder. The results obtained with Wikipedia show a CRR 6.89% lower and a WRR 9.49% lower than with GIC, despite the fact that the vocabulary used in RM and Wikipedia is modern French.

Results on Los Esposalles (ESP)

The vocabulary of Los Esposalles is built primarily with named entities as it is extracted from 18 th century Spanish wedding registers. This explains the low lexical coverage for the transductive transfer learning. Nevertheless, CRR is greater than 90% except with RM (Expe. Id 19) and WRR outperforms the lexical coverage. We did not experiment the decoding with unigrams on this resource because there are too few words. Expe. Id 21 using GIC, GW, and RM shows that this decoder enables to learn a representation from one language (mainly in French) to another.

Analysis on Test Sets Finally, creating and using a new dataset in the field of the Italian Comedy is an interesting approach to decode ICR. This enables a better word reconstruction since the lexical coverage is increased by more than 20%. Among the unknown but well-recognized words can be found light abbreviations such as "arleq." instead of "Arlequin". Contrary to the validation results, the best results are obtained thanks to the combination with RM dataset (Expe. Id 3 versus 4). This can be explain by the difference of the vocabulary, the validation set is composed of GIC while the test set is composed of ICR. For the transductive transfer learning process, the results with Wikipedia are similar (+1.20%) to using mixed resources with less data (Expe. Id 7 versus Expe. Id 8) contrary to the results on the validation set, where there is a significant difference around 15% on WRR with and without dictionary.

Error Analysis Table 3 shows some mistakes made by the decoder. For words with a repeated character such as "cavalcade" and "clemence", it is common for the system to fail to generate the characters inserted between each repetition. Common mistakes are a permutation between two consecutive characters such as "suitte" and one doubled character to replace another one. In the last example, "Soldat", the decoder predicts two start symbols: the second one replaces the first letter of the word. This happens when small resources are used to train the system.

Table 4 shows the distribution of the different types of errors presented in Table 3 for 3 selected experimentations (Expe. Id 2, 5, and 8). The most common mistake is a simple switch between two characters: this represents 65.14% to 88.32% of the errors, depending on the considered experiments. For the experiments including GIC in the training step, the redundant character errors and wrong starting character are similar. They represent only 10% and 20% of all the identified errors. Expe. Id 8 has a high rate of wrong starting characters that can be explained by the lack of lexical coverage between Wiki and ICR. Despite the equivalent quantity of words used for the training step with Wiki than with GIC, the system has difficulties finding out the start of words. Indeed, Wiki has been preprocessed and normalized by removing capitalization so the information seems to be important for the system.

To conclude, we can see that, in Expe. Id 2, there is on average 1.14 character error by word while, in Expe. Id 8, there is on average 1.73 character error by word: that can be explained by the 45.8% drop of WRR between these two experiments.

Conclusion and Future Work

In this paper, we presented a transfer learning approach dealing with the lack of ground truth of our Italian Comedy data, for sequence generation. We selected some available resources with the same domain and from the same time period to achieve the transfer learning. We chose an approach that did not use an indication of the order of characters in words rather than usual sequence-to-sequence approaches to facilitate the transfer learning.

Our results show that a character-based decoder can infer a word from a letter-ngrams vector. Moreover, the lexical coverage between the training and test data is reached by the word recognition rate and mostly exceeded without the dictionary. The dictionary misleads the decoder rather than helping it because only some forms of a word are included in the dictionary. We demonstrated that transfer learning is more efficient from historical spelling to modern vocabulary than the opposite. The difference in languages implies a low coverage of the words but it still enables a correct decoding of the words as long as the resources come from the same time period. Therefore, the letter-ngrams models are efficient across languages sharing the same alphabet. Nevertheless, the decoder encounters some difficulties with specific words. We might be wondering whether the letter-ngram size limit of 3 is large enough, or whether a higher amount of training data could solve this issue. Despite this, our simple decoder built with only 4 layers obtained good results. This strengthens our idea of looking for new untapped resources instead of relying on resources traditionally used such as Wikipedia.

Currently, we are carrying out experiments with this approach at the word-level only and without punctuation marks. The next step for the decoder would be to extend it to the sentence-level, more specifically to deal with the space character. Initially, we aim at implementing a new handwriting recognition system with an encoder-decoder model. We consider that the decoder presented in this article is operational. Our future work is going to look at the encoder components independently. The main difficulty could be to handle the errors by encoding them in the vector of letter-ngrams. Indeed, unlike the experiments carried out in this paper, the input data of the decoder will be noisy.

Figure 2 :

 2 Figure 2: Example of a financial daily record of the Italian Comedy with fields identified.

Table 1 :

 1 Table 1 presents the vocabulary size for each dataset that are used to train the decoder. We choose to create a new dataset from available digitized historical books, as well as to use three existing image resources (RIMES, George Washington, and Los Esposalles) and the French version of Wikipedia since it is a widely used resource in NLP tasks. Vocabulary size of the training, validation, and test sets of each dataset.

		Google IC RIMES	Los Esposalles	Georges Washington	Wikipedia Fr IC Registers
	Train	26,573	4,477	2,565	660	24,456	0
	Validation	2,953	1,578	629	521	3,843	0
	Test	0	1,627	629	431	1.928	1,431

Table 2 :

 2 Sequence generation results. CRR and WRR with/without the dictionary on the Italian Comedy Records (ICR), Rimes (RM), and Los Esposalles (ESP) validation and test datasets. Several experiments are presented: evaluation of the baseline, combination of the resources, and transductive transfer learning with all the resources. The lexical coverage is computed between the training and the test datasets.

	Test Train	Expe. Id	Letter ngrams	Lexical Coverage (%)	Validation CRR WRR	WRR dict.	Test CRR WRR	WRR dict.
		GIC GIC	1 2	1 1,2,3	65.57	69.11 95.85	13.05 77.83	14.65 69.27 66.47 97.10	14.54 86.22	10.83 39.30
		GIC+RM	3	1,2,3	67.53	95.97	78.78	66.17 97.27	86.57	39.23
	ICR	GIC+ESP	4	1,2,3	65.83	96.49	85.87	67.22 96.96	81.46	39.09
		GIC+ESP+GW+RM	5	1,2,3	67.65	95.99	77.86	66.37 95.85	79.65	38.25
		RM	6	1,2,3	14.52	74.69	19.49	23.32 79.70	30.42	17.27
		RM+ESP+GW	7	1,2,3	23.39	78.11	25.36	28.44 83.68	40.21	23.99
		Wiki	8	1,2,3	0.0	83.67	41.18	42.75 87.32	41.40	25.24
		RM RM	9 10	1 1,2,3	75.09	82.66 93.95	37.83 79.93	27.58 83.97 37.45 94.72	43.07 79.50	28.49 37.78
		GIC+RM	11	1,2,3	83.83	97.32	89.04	39.94 98.25	92.0	40.49
	RM	GIC+ESP+GW+RM	12	1,2,3	83.95	96.8	86.18	39.43 96.22	80.73	39.14
		GIC	13	1,2,3	58.55	93.93	75.48	36.56 95.51	81.53	38.58
		GIC+ESP	14	1,2,3	59.04	93.89	74.33	36.05 95.46	80.61	38.15
		Wiki	15	1,2,3	0.0	87.04	65.99	33.31 90.36	67.57	35.20
		GIC+ESP	16	1,2,3	85.94	98.25	91.61	62.92 98.57	91.11	56.51
		GIC+ESP+GW+RM	17	1,2,3	86.10	98.14	90.48	61.94 98.40	90.79	57.62
	ESP	GIC	18	1,2,3	15.96	88.18	54.35	45.48 91.68	65.87	44.76
		RM	19	1,2,3	7.27	67.82	14.03	8.87 72.83	18.25	12.70
		GIC+RM	20	1,2,3	17.37	87.4	41.77	51.61 92.05	64.13	46.51
		GIC+RM+GW	21	1,2,3	17.69	88.77	57.1	44.35 91.68	64.60	44.60
		Wiki	22	1,2,3	0.0	78.89	27.26	24.52 84.52	34.28	32.38

Table 3 :

 3 Examples of mistakes made by our decoder.

	Expe. Id	% Error Redundant Character	% Error Switch	% Error Starting Char.
	2	9.13	81.22	9.65
	5	4.81	88.32	6.87
	8	8.44	65.14	26.42

Table 4 :

 4 Quantitative analysis of the different error types on 3 ICR results.