Mostafa Bendahmane
email: mostafabendahmane@yahoo.fr

Thomas Lepoutre
email: thomas.lepoutre@inria.fr

Americo Marrocco
email: americo.marrocco@inria.fr

Benoît Perthame
email: benoit.perthame@upmc.fr

Conservative cross diffusions and pattern formation through relaxation

This paper is aimed at studying the formation of patches in a cross-diffusion system without reaction terms when the diffusion matrix can be negative but with positive self-diffusion. We prove existence results for small data and global a priori bounds in space-time Lebesgue spaces for a large class of 'diffusion' matrices. This result indicates that blow-up should occur on the gradient. One can tackle this issue using a relaxation system with global solutions and prove uniform a priori estimates. Our proofs are based on a duality argument à la M. Pierre which we extend to treat degeneracy and growth of the diffusion matrix.

We also analyze the linearized instability of the relaxation system and a Turing type mechanism can occur. This gives the range of parameters and data for which instability may occur. Numerical simulations show that patterns arise indeed in this range and the solutions tend to exhibit patches with stiff gradients on bounded solutions, in accordance with the theory.

Introduction

The dynamics of interacting population with cross-diffusion have been widely investigated by several researchers. The concept of this phenomena was studied by Levin [START_REF] Levin | A more functional response to predator-prey stability[END_REF], Levin and Segel, [START_REF] Levin | Hypothesis for origin of planktonic patchiness[END_REF], Okubo [START_REF] Okubo | Diffusion and ecological problems: mathematical models[END_REF], Mimura and Murray [START_REF] Mimura | On a diffusive prey-predator model which exhibits patchiness[END_REF], Mimura and Kawasaki [START_REF] Mimura | Spatial segregation in competitive interaction-diffusion equations[END_REF], Mimura and Yamaguti [START_REF] Mimura | Pattern formation in interacting and diffusing systems in population biology[END_REF], and many other authors. All these papers base the pattern formation on a reaction term as prey-predator interactions.

Spatial patterns can however emerge from pure diffusions without reaction terms nor oriented drift at the individual level. This is the case of N populations described microscopically by a brownian process which intensity depends upon the macroscopic density U = (U 1 , ..., U N ) of the populations

dX k (t) = σ k U (X, t) dW k (t), 1 ≤ k ≤ N.
When set on a bounded domain with reflexion on the boundary, the corresponding models for the population density are cross-diffusions

∂ ∂t U -∆A(U ) = 0, in Ω, ∂ ∂n A(U ) = 0 on ∂Ω, (1)
where U = U (x, t) ∈ R N , x ∈ Ω a smooth bounded domain of R d , n denotes the outward normal to Ω. Finally A : R N → R N is a nonlinearity related to the intensity of the interactions by the relation

A k (U ) = U k a k (U ), (2)
and a k (U ) = 1 2 σ k (U ).σ t k (U ). We also complete the system with an initial data

U (t = 0) = U 0 = (U 0 1 , ..., U 0 N ) with U 0 k ≥ 0.
The properties, and pattern formation capacity of such systems are better described by introducing the more general form

∂ ∂t U k - N l=1 div[D kl (U )∇U l ] = 0, (3)
where D kl (U ) are the components of a N × N matrix, the derivative of A in the case [START_REF] Amann | Dynamic theory of quasilinear parabolic equations. I. Abstract evolution equations[END_REF]. Boundary conditions have to be imposed and we consider here the case of Neumann conditions D(U ).∇U.n = 0, on ∂Ω.

For such boundary conditions, mass conservation yields naturally

U (t) = U 0 , ∀t ≥ 0,
where U 0 denotes the average

U = 1 |Ω| Ω U (x)dx.
The Lotka-Volterra competition with cross-diffusion has recently received great attention. They are many established results concerning the global existence of classical solutions (see [START_REF] Wang | The global existence of solutions for a cross-diffusion system[END_REF][START_REF] Li | Global existence of solutions to a cross-diffusion system in higher dimensional domains[END_REF] and the references therein) where most of the proofs rely on Amann's theorem [START_REF] Amann | Dynamic theory of quasilinear parabolic equations. I. Abstract evolution equations[END_REF][START_REF]Dynamic theory of quasilinear parabolic equations. II. Reaction-diffusion systems[END_REF]. We point out that standard parabolic theory is not directly applicable to our model due to the presence of cross-diffusion terms.

In opposition with pattern formations, an important issue has been widely studied which is to know in which circumstances the solutions exist globally and behave like in the case a single heat equation, i.e., relax to a constant state as t → ∞. Typically three kinds of special methods have been helpful in this scope. The first method is to rely on the maximum principle. It can occur on certain combinations of the U i as in [START_REF] Amann | Dynamic theory of quasilinear parabolic equations. I. Abstract evolution equations[END_REF][START_REF]Dynamic theory of quasilinear parabolic equations. II. Reaction-diffusion systems[END_REF][START_REF] Erratum | Dynamic theory of quasilinear parabolic systems[END_REF]. Entropy methods also applies to particular systems and has also been a useful tool because of the related symmetrization of the system following [START_REF] Kawashima | On the normal form of the symmetric hyperbolic-parabolic systems associated with the conservation laws[END_REF][START_REF] Degond | Symmetrization and entropy inequality for general diffusion equations[END_REF]. It provides a natural method both for existence and relaxation to steady state as in the recent studies in [START_REF] Chen | Analysis of a parabolic cross-diffusion population model without selfdiffusion[END_REF][START_REF]Analysis of a parabolic cross-diffusion semiconductor model with electron-hole scattering[END_REF] of the Shigezada-Kawasaki prey-predator system [START_REF] Shigesada | Spatial segregation of interacting species[END_REF], or for tumous models [START_REF] Hadjadj | An existence result to a strongly coupled degenerated system arising in tumor modeling[END_REF]. This method typically applies in the special case of the square entropy when D is definite positive meaning that there is

ν > 0 such that N k,l=1 ξ k D kl (U )ξ l ≥ ν|ξ| 2 , ∀ξ ∈ R N . (4)
This strong positivity property gives the energy inequality 1 2

d dt Ω |U (t)| 2 dx = N k,l=1 ∇U k D kl (U )∇U l ≤ -ν Ω |∇U (t)| 2 dx.
The third method, by duality, has been used in [START_REF] Boudiba | Global existence for coupled reaction-diffusion systems[END_REF] on a particular upper-diagonal diffusion with Dirichlet boundary condition; we show here that the method can be extended to very general systems with Neuman conditions. Concerning instabilities, the interplay between diffusion and reaction terms has raised surprising results in the spirit of the Turing instability mechanism [START_REF] Turing | The chemical basis of morphogenesis[END_REF]. The question to know if cross-diffusion or self-diffusion gives an advantage to competing species is studied in [START_REF] Lou | Evolution of cross-diffusion and self-diffusion[END_REF][START_REF] Lou | Diffusion, self-diffusion and cross-diffusion[END_REF].

Our interest in this paper concerns instability mechanisms that may appear only from the diffusion intensity and the lost of positivity in the second order matrix. We study in which circumstances the increase of this intensity with higher density of the other species can lead to a segregation phenomena. Of course such instability is incompatible with any entropy inequality, and thus (4) cannot hold. This rises several mathematical questions which seem to be new in the domain of cross-diffusions. Can it still be that small solutions exist globally even though the maximum principle does not hold in general? For large data, what kind of regularity or 'blow-up' can we expect? Finally, how do regularized systems behave in the 'instability' regime. We will study these questions with a model problem in mind related to Shigezada-Kawasaki's system and that represents two species with stronger interactions

∂ ∂t U 1 -∆[U 1 (1 + a 11 U p 1 + a 12 U p 2 )] = 0, x ∈ Ω, ∂ ∂t U 2 -∆[U 2 (1 + a 21 U p 1 + a 22 U p 2 )] = 0, (5)
still with Neumann boundary conditions. One can check that as soon as p > 1, the matrix D kl is negative for U large.

We approach these questions both theoretically and numerically. In particular we prove existence for small initial data (section 2) and we give a priori bounds in L p t,x for possible solutions to [START_REF] Boudiba | Global existence for coupled reaction-diffusion systems[END_REF] thus showing that the break-down should come from the blow-up of gradient estimates rather than usual L p norms (section 3). Our main tool here is a general estimate due to M. Pierre [START_REF] Pierre | Blowup in reaction-diffusion systems with dissipation of mass[END_REF][START_REF] Pierre | Weak solutions and supersolutions in L 1 for reaction-diffusion systems[END_REF] in the context of semilinear parabolic systems (arising in population dynamics or more generally reaction-diffusion systems) (see also [START_REF] Desvillettes | Global existence for quadratic systems of reaction-diffusion[END_REF]). In section 4, these bounds are extended to a relaxation system that takes into account a local measurement of densities; we show that the method is well adapted to general (even not parabolic) cross-diffusions and prove global existence for the relaxation system. For non-parabolic cases, we show that Turing instability occurs in a certain range of data and for small relaxation parameters. Numerical simulations of this relaxation system are performed in section 5. They show that the oscillatory initial regime reorganizes to create patches where one species density dominates the other and interfaces are generated which width is related to the relaxation length. The technical and general extension of M. Pierre's estimate to bounded domains for Neumann boundary conditions is kept for an Appendix as well as another remarkable energy estimate which holds for particular cross-diffusion coefficients in (5).

Global solutions for small data

The lack of maximum principle for general diffusion systems is a major difficulty that arises for systems as [START_REF] Erratum | Dynamic theory of quasilinear parabolic systems[END_REF]. Using stronger H 1 estimates, we can show in one dimension that for small initial data there is global existence. Such solutions decay to the constant state for large time and this is incompatible with the patterns formation we are interested in. This indicates that large initial data are necessary for pattern formation as expected in general.

We consider the system under the form (1)-( 2) and assume that

a k ∈ C 1 (R N ), (6)
a k (0) ≥ ν > 0. ( 7
)
Theorem 2.1 (Global small solutions in 1 dimension) In one dimension, under assumptions ( 6), [START_REF]Analysis of a parabolic cross-diffusion semiconductor model with electron-hole scattering[END_REF], and for an initial data satisfying, with α small enough,

U 0 L ∞ (Ω) + ∇U 0 L 2 (Ω) ≤ α,
there is a global solution U (t, x) to the cross-diffusion system (1)- [START_REF]Dynamic theory of quasilinear parabolic equations. II. Reaction-diffusion systems[END_REF]. It satisfies for all t > 0

U (t) L ∞ (Ω) ≤ Cα, ∇U (t) L 2 (Ω) ≤ α, with C independent of t and U (t) -→ t→∞ U 0 .
Proof. Firstly, since U (t) is a priori conserved, we notice that

U (t) L ∞ (Ω) ≤ U 0 L ∞ (Ω) + |Ω| ∇U L 2 (Ω) ≤ α + |Ω| ∇U L 2 (Ω) .
Secondly, we multiply (1) by the vector A (U ) and differentiate. We obtain

∂ t ∇A(U ) = ∇ A (U )∂ t U = ∇ A (U )∆A(U ) ,
we multiply by ∇A(U ) and integrate

d dt Ω |∇A(U )| 2 2 = Ω ∇A(U )∇ A (U )∆A(U ) = - Ω ∆A(U ) A (U ) ∆A(U ). (8)
But from ( 2) and ( 7), we have A (0) kl = a kk (0)δ kl , and thus for U L ∞ (Ω) ≤ ε small enough, we have

XA (U )X ≥ ν 2 X 2 , ∀X ∈ R N , (9)
|A (U )X| 2 ≥ ν 2 2 X 2 , ∀X ∈ R N . ( 10
)
We now consider T * defined by

T * := sup{t ≥ 0, U (t) ≤ ε}.
For α < ε, then T * > 0. Suppose T * < ∞, then for 0 ≤ t ≤ T * , we have, from ( 8) and [START_REF] Desvillettes | Global existence for quadratic systems of reaction-diffusion[END_REF],

Ω |∇A(U (t))| 2 dx ≤ Ω |∇A(U 0 )| 2 dx.
Therefore,

Ω |A U (t) ∇U (t)| 2 ≤ Ω |∇A(U 0 )| 2 dx.
Using [START_REF] Evans | A survey of entropy methods for partial differential equations[END_REF], this leads to

Ω |∇U (t)| 2 dx ≤ 2 ν 2 Ω |∇A(U 0 )| 2 dx ≤ 2C ν 2 Ω |∇U 0 | 2 dx. Now, we choose α such that α ≤ ε 3 and 2C|Ω| ν α ≤ ε 3 , this ensures, U (t) L ∞ (Ω) ≤ 2ε 3 and thus, T * is not maximal. Therefore T * = ∞.
Finally, the existence of a solution for small times follows from standard parabolic theory and the a priori bound above shows that these are global solutions.

We now prove the time convergence to

U 0 . Because Ω (U -U 0 ) 2 = Ω U 2 -U 2 , we compute d dt Ω (U -U ) 2 = d dt Ω U 2 = Ω ∆A(U )U = - Ω A (U )∇U.∇U.
But as the solution stays in the domain

U (t) L ∞ (Ω) ≤ ε, we have for some constant C 0 , d dt Ω (U -U 0 ) 2 ≤ - ν 2 Ω |∇U | 2 ≤ -2C 0 Ω (U -U 0 ) 2 ,
thanks to the Poincaré Wirtinger inequality. We conclude from Gronwall lemma that

U -U 0 L 2 (Ω) ≤ e -C 0 t U 0 -U 0 L 2 (Ω) .

A priori bounds for large data

For large initial data and when the condition (4) does not hold, we cannot expect in general the existence of solutions for the cross-diffusion system (3)- [START_REF]Dynamic theory of quasilinear parabolic equations. II. Reaction-diffusion systems[END_REF]. For a single equation, the corresponding situation is when A (u) can be negative on some interval I ⊂ R + .

∂ ∂t u -∆A(u) = 0.
The situation is analyzed in [START_REF] Plotnikov | Passage to the limit with respect to viscosity in an equation with a variable direction of parabolicity[END_REF][START_REF] Novick-Cohen | Stable patterns in a viscous diffusion equation[END_REF][START_REF] Evans | Irreversibility and hysteresis for a forward-backward diffusion equation[END_REF] (see also the survey in [START_REF] Evans | A survey of entropy methods for partial differential equations[END_REF]) and it is better analyzed in term of relaxation systems, an approach which we will follow later. We expect that oscillations or jumps occur at positive times, but a first issue is a priori control in L ∞ for possible solutions. This follows from the maximum principle for a single equation (and possibly from entropy constructions for relaxation systems, see [START_REF] Plotnikov | Passage to the limit with respect to viscosity in an equation with a variable direction of parabolicity[END_REF]). For systems this is an open question and we give here a first a priori bound Theorem 3.1 Smooth solutions to (3)-( 2) satisfy the a priori bounds

T 0 Ω N k=1 A k (U ) N k=1 U k dxdt 1 2 ≤ C 1 (Ω) U 0 L 2 (Ω) + C 2 Ω, N k=1 U 0 k √ T . ( 11
)
Therefore if we assume in (2) that

a k (U ) ≥ ν > 0 ∀k = 1, ..., N, (12)
then we also have, with the notation

Q T = Ω × [0, T ], ν U L 2 (Q T ) ≤ C 1 (Ω) U 0 L 2 (Ω) + C 2 (Ω, U 0 ) √ T . (13)
In the particular case of model ( 5), we observe that the larger is p, the best is the bound in [START_REF] Evans | Irreversibility and hysteresis for a forward-backward diffusion equation[END_REF]. In particular, A(U ) is always integrable. The L 2 estimate in ( 13) is much weaker.

Proof. Our proof is based on a variant of a general duality argument due to [START_REF] Pierre | Weak solutions and supersolutions in L 1 for reaction-diffusion systems[END_REF][START_REF] Desvillettes | Global existence for quadratic systems of reaction-diffusion[END_REF], that is presented in Appendix A. We denote w = N k=1 U k . We sum up the equations and we find

∂ t w -∆ N k=1 a k (U )U k = 0,
which we write

∂ t w -∆α(t, x)w = 0, α(t, x) := N k=1 a k (U (t, x))U k (t, x) w(t, x) = α(U (t, x)).
We can use now [START_REF] Okubo | Diffusion and ecological problems: mathematical models[END_REF] in Appendix A and obtain,

√ α w L 2 (Q T ) ≤ C(Ω) w 0 L 2 (Ω) + 2 w 0 √ α L 2 (Q T ) . (14)
In order to control the right hand side by

√ α w L 2 (Q T )
, we use a truncation method. Since the coefficients a k (U ) are continuous, and U k are nonnegative, we may define for any R > 0, sup w≤R α(U ) := M (R) < +∞. Furthermore, we may truncate w away from values less than R, a parameter to be fixed later on, with the indicator function 1 I {w≥R} and rewrite [START_REF] Izuhara | Reaction-diffusion system approximation to the cross-diffusion competition system[END_REF] as

√ α w 1 I {w≥R} L 2 (Q T ) ≤ C(Ω) w 0 L 2 (Ω) + 2 w 0 √ α1 I {w≥R} L 2 (Q T ) + 2 w 0 √ α 1 I {w≤R} L 2 (Q T ) , √ α w 1 I {w≥R} L 2 (Q T ) ≤ C(Ω) w 0 L 2 (Ω) + 2 w 0 √ α w R 1 I {w≥R} L 2 (Q T ) + 2 w 0 M (R) L 2 (Q T ) .
We choose R = 4 w 0 and obtain

√ α w 1 I {w≥R} L 2 (Q T ) ≤ 2C(Ω) w 0 L 2 (Ω) + 4 w 0 |Ω| T M (R).
Since we also know that

√ αw1 I {w≤R} L 2 (Q T ) ≤ R M (R)|Ω|T = 4 w 0 M (R)|Ω|T , we conclude √ α w L 2 (Q T ) ≤ 2C(Ω) w 0 L 2 (Ω) + C 2 ( w 0 ) |Ω| T ,
with C 2 ( w 0 ) = 8 w 0 M (4 w 0 ). This is exactly the a priori estimate [START_REF] Evans | Irreversibility and hysteresis for a forward-backward diffusion equation[END_REF].

The other statement is a simple and direct consequence.

A relaxation system

If we assume that the intensity of the brownian motion depends on the density of the populations measured with a space scale δ > 0 and not at the exact location x, then the system (5) can be replaced by a cross-diffusion relaxation system

∂ ∂t u k -∆[a k (ũ)u k ] = 0, x ∈ Ω, k = 1, ..., N, -δ 2 ∆ũ k + ũk = u k , (15)
together with Neumann boundary conditions both on u k and ũk . Relaxation procedures are usual and several other examples for cross-diffusions can be found in [START_REF] Izuhara | Reaction-diffusion system approximation to the cross-diffusion competition system[END_REF][START_REF] Anguige | A one-dimensional moel of cell diffusion and aggregation, incorporating volume filling and cell-to-cell adhesion[END_REF] and for phase transitions see [START_REF] Iida | Diffusion, cross-diffusion and competitive interaction[END_REF][START_REF] Evans | A survey of entropy methods for partial differential equations[END_REF][START_REF] Plotnikov | Passage to the limit with respect to viscosity in an equation with a variable direction of parabolicity[END_REF]. In terms of the ecological interpretation, it is also more realistic than the initial system (5), because individuals are unlikely to be able to access a pointwise density, but might estimate their environment from sensing at a smaller scale. We can expect that the system ( 15) is well-posed, and we first study this question. Then we prove uniform bounds independent of δ which indicates that instability should arise from the blow-up of gradients. To tackle the question of instabilities, we show that the system exhibits Turing patterns for δ small, this is our second goal in this section.

We keep in mind the example [START_REF] Boudiba | Global existence for coupled reaction-diffusion systems[END_REF] and assume that for some p > 0 one has

0 < ν ≤ a k (U ) ≤ C 0 (1 + |U | p ), ∀k ∈ {1, ..., N }. (16)
For later purpose, we also introduce the assumption that for some constant K > 0 and some η > 0, we have

∇a k (U ) a η k (U ) ≤ K|∇U |. (17)
We have in mind coefficients of the form (1 + Ũ p j ) and then we can take η = p-1 p .

Uniform estimates for p < 2

We first extend the a priori estimate of section 3 to this relaxation system. The coupling induces a limitation on the possible growth of the nonlinearities a k (U ) and we have the Theorem 4.1 Assume that ( 16) holds for some 0 < p < 2, then, the a priori bound holds for a constant

C independent of δ u L 2 (Q T ) ≤ C( u 0 L 2 (Ω) , T ), ∀T > 0.
This is weaker than the a priori bound in [START_REF] Evans | Irreversibility and hysteresis for a forward-backward diffusion equation[END_REF]. The difficulty in the case at hand comes from the dependency of a k (ũ) which we cannot lower bound from u itself.

Proof. We denote by ãk the quantity a k (ũ). The estimate [START_REF] Okubo | Diffusion and ecological problems: mathematical models[END_REF] of Appendix A gives, for all k ∈ {1, ..., N } ãk

u k L 2 (Q T ) ≤ C(Ω) u 0 k L 2 (Ω) + 2 u 0 k ãk L 2 (Q T ) .
The last term may be estimated as

ãk 2 L 2 (Q T ) = T 0 Ω ãk ≤ C 0 T 0 Ω (1 + l ũp l ).
Thanks to Holder inequality and direct estimate on the solution to the elliptic equation on ũl , we have,

T 0 Ω ũp l ≤ (|Ω| T ) 2-p 2 ũl p 2 L 2 (Q T ) ≤ C(Ω)T 2-p 2 u l p L 2 (Q T ) .
Finally, back to the original inequality we arrive at

√ ν u k L 2 (Q T ) ≤ C(Ω) u 0 k L 2 (Ω) + 2 C 0 T |Ω| u 0 k + 2 u 0 k C 0 (|Ω|T ) 2-p 4 u p 2 L 2 (Q T ) , which leads to √ ν u L 2 (Q T ) ≤ C(Ω, u 0 L 2 (Ω) , T ) + C(Ω, u 0 )T 2-p 2 u p 2 L 2 (Q T ) . As p/2 < 1, this proves that u L 2 (Q T )
is a priori bounded as by a constant depending only on Ω, T , u 0 L 2 (Ω) and the two constants in (16).

Existence of solutions

We now show stronger estimates from which strong compactness of solutions follows. They use fundamentally the regularity on ũ in [START_REF] Kawashima | On the normal form of the symmetric hyperbolic-parabolic systems associated with the conservation laws[END_REF] by elliptic regularizing effects. Existence of global solutions follow and the details are carried out in [START_REF] Lepoutre | Partial differential equations arising in biology[END_REF].

The main result of this section is the following Proposition 4.2 Assume that (16) holds with p > 1, and p < 2d d-2 when d > 2. Then, the a priori estimate holds

ãi u i L 2 (Q T ) ≤ C(δ, u 0 L 1 ∩L 2 (Ω) , T ). ( 18
)
Furthermore, if we assume (17) in dimension 1 with any η > 0, and in dimension 2 with 0 < η < 1, then we have for all 1 ≤ q < ∞,

u(t) L q (Ω) ≤ C(q, δ, u 0 L 1 ∩L q (Ω) , T ), 0 ≤ t ≤ T, (19)
T 0 Ω |∇u q/2 | 2 dxdt ≤ C(q, δ, u 0 L 1 ∩L q (Ω) , T ). ( 20
)
Proof. We begin with the proof of (18) which improves that of the theorem 4.1. We use again the estimate (27) applied to u i which yields

√ ν u i L 2 (Q T ) ≤ ãi u i L 2 (Q T ) ≤ C(Ω) u 0 i L 2 (Ω) + 2 u 0 i ãi L 2 (Q T ) . (21)
We use the hypothesis [START_REF] Lepoutre | Partial differential equations arising in biology[END_REF] to get

ãi L 2 (Q T ) ≤ C 0 |Ω| T + C 0 T 0 Ω |ũ| p ≤ C (Ω) √ T + T 0 Ω |ũ| p = C 0 | Ω| T + T 0 ũ p p .
Thanks to elliptic regularity we also have

ũ p ≤ C(δ, r) u r ,
for any r > 1 satisfying also 1 p ≥ 1 r -2 d (particularly it is true for any r if d = 1, 2). Then, using interpolation inequality and choosing r < 2, we find successively

ũ p ≤ C(δ, r) u 1-θ(r) 1 u θ(r) L 2 (Ω) = C(δ, r) u 0 1-θ(r) 1 u θ(r) L 2 (Ω) θ(r) = 1 -1 r 1 -1 2 = 2(1 - 1 r ), ãi L 2 (Q T ) ≤ C 0 √ Ω T + T 0 C(δ, r) p u 0 (1-θ(r))p 1 u pθ(r) L 2 (Ω) , ãi L 2 (Q T ) ≤ C 0 √ Ω T + C(δ, r) p/2 u 0 (1-θ(r))p/2 1 T 0 u(t) pθ(r) L 2 (Ω) dt . (22)
Now, if we may choose r such that θ(r)p < 2, we get, thanks to Jensen's inequality

T 0 u(t) pθ(r) L 2 (Ω) dt = T 0 u(t) 2(pθ(r)/2) L 2 (Ω) dt ≤ T 1-pθ(r)/2 T 0 u(t) 2 L 2 (Ω) dt pθ(r)/2
, which we rewrite as

T 0 u(t) pθ(r) L 2 (Ω) dt ≤ T 1-pθ(r)/2 u pθ(r) L 2 (Q T ) .
Replacing in [START_REF] Marrocco | Numerical simulation of chemotactic bacteria aggregation via mixed finite elements[END_REF], we have

ãi L 2 (Q T ) ≤ C 0 √ Ω T + C(δ, r, p, u 0 1 )T 2-pθ(r) 4 u pθ(r) 2 L 2 (Q T )
And replacing in [START_REF] Lou | Diffusion, self-diffusion and cross-diffusion[END_REF], we obtain

ν u L 2 (Q T ) ≤ C(Ω) u 0 L 2 (Ω) + C (Ω) √ T + C(δ, r, p, u 0 1 )T 2-pθ(r) 4 u pθ(r) 2 L 2 (Q T ) (23)
This concludes the first inequality when pθ(r)/2 < 1, and it remains to find the range of p in order to fulfill the constraints. These can be obtained choosing r close enough to 1 for d = 1, 2. For d > 2, we need the conditions

1 p ≥ 1 r -2 d , 1 < r < 2, pθ(r) 2 = p(1 -1 r ) < 1.
We choose to satisfy the second line 1 r > p-1 p , but close to equality (which gives 1 < r < 2 as we check it a posteriori). This leads, in the first line, to the condition p < 2d d-2 , but close to equality (which imposes r < 2d d+2 ). The bound on u L 2 (Q T ) gives then a bound on √ ãi u i L 2 (Q T ) thus concluding the proof of [START_REF] Levin | A more functional response to predator-prey stability[END_REF]. This estimate leads to the stronger a priori bounds [START_REF] Li | Global existence of solutions to a cross-diffusion system in higher dimensional domains[END_REF] that we prove now. We go back to the equation and write d dt u q + C q ã|∇u q/2 | 2 = -C 1 q u q/2 ∇u q/2 ∇ã.

From this equality, and writing ã|∇u q/2 | = |∇(ã 1/2 u q/2 ) -u q/2 ∇ã 1/2 |, we derive directly the inequality

d dt u q + Cq 3 ã|∇u q/2 | 2 + Cq 3 |∇(ã 1/2 u q/2 )| 2 ≤ C 2 q u q |∇ã| 2 ã ≤ C q u q |∇ũ| 2 ã2η-1 . ( 24
)
We show separately how in dimensions 1 and 2 this allows us to control any L q norm for q < +∞.

The case d = 1. The proof is easier for d = 1 because there exists a constant C (depending only on δ and Ω) such that

ũ L ∞ (Ω) + ∇ũ L ∞ (Ω) ≤ C |u| ≤ C( u 0 1 ).
Therefore from [START_REF] Mimura | On a diffusive prey-predator model which exhibits patchiness[END_REF] we deduce d dt u q ≤ C.C q u q .

We conclude thanks to Gronwall lemma.

The case d = 2. In dimension 2, we divide the proof into two steps.

Step 1, 1 < q < 2. We first focus on small values of the exponent q namely 1 < q < 2 and 2η < 1 + q 2 (the limitation η < 1 comes from choosing q close to 2). From ( 24) and using successively [START_REF] Levin | Hypothesis for origin of planktonic patchiness[END_REF] and Hölder inequality, we obtain

d dt u q ≤ C 3 q u q ãq/2 |∇ũ| 2 . ≤ C 3 q u √ ã q qr ∇ũ 2 2r ≤ C(q, δ, r) u √ ã q qr u 2 m
thanks to elliptic regularity, with

1 m = 1 2r + 1 2 = 1 - 1 2r .
Choosing r = 2 q then 1 m = 1 -q 4 and we arrive at

d dt u q ≤ C(δ, r) u √ ã q 2 u 2 m ≤ C(δ, r) u √ ã q 2 u 2θ q u 2(1-θ) L 2 (Ω) ≤ C(δ, r) u √ ã q 2 u q q u 2-q? L 2 (Ω)
because, as we have 1 2 < 1 m < 1 q , we may interpolate m between q and 2 with

θ = 1 m -1 2 1 q -1 2 = 2-q 4 2-q 2q = q 2 .
We finally obtain by Young's inequality,

d dt u q ≤ C(q, δ, r) u √ ã 2 2 + u 2 L 2 (Ω) u q q ,
and we may then conclude with Gronwall lemma using the estimates on

T 0 u √ ã 2 2 and
T 0 u 2 2 in [START_REF] Levin | A more functional response to predator-prey stability[END_REF]. By interpolation, this also gives a priori bound for any L q norm for q ∈ [1, 2[. This ends step 1.

Step 2 We now focus on L q norms for q ≥ 2. We notice that now, controlling any L q norm for q < 2, we control by elliptic regularity any L q norm of ∇ũ except the L ∞ norm. We also control the L ∞ norm of ũ and therefore of ã. We go back to [START_REF] Mimura | On a diffusive prey-predator model which exhibits patchiness[END_REF] and conclude

d dt u q + C |∇u q/2 | 2 ≤ C u q |∇ã| 2 ã ≤ C(δ, q, T ) u q |∇ũ| 2 ≤ C(δ, q, T ) u q qr ∇ũ 2 2r ≤ C(δ, q, r, T ) u q qr .
We use now interpolation: for any s > r > 1, we have

u qr ≤ u 1-θ q u θ qs θ = 1 qr -1 qs 1 q -1 qs = 1 r -1 s 1 -1 s .
Using Young's inequality, we obtain for ε small (to be chosen later)

d dt u q + C |∇u q/2 | 2 ≤ C(δ, q, r, T, ε) u q q + ε u q qs .
We have by Poincaré Wirtinger inequality

u q qs = u q/2 2 2s ≤ 2 u q/2 -u q/2 2 2s + 2 u q/2 2 2s ≤ C(Ω, s, ν) Ω |∇u q/2 | 2 + C(Ω, s) u q q/2 ≤ C(Ω, s, ν) Ω |∇u q/2 | 2 + C(Ω, s) u q 1 + u q q ,
from interpolation and Young's inequality. We fix s as above, choose ε small enough and we obtain

d dt u q ≤ C(Ω, q) u q 1 + u q q .
And we conclude ( 19) by Gronwall lemma. The last estimate (20) also follows from [START_REF] Mimura | On a diffusive prey-predator model which exhibits patchiness[END_REF].

Turing patterns

In order to go further and study the instability occurring in the regularized model, we consider the following particular system:

               ∂ t u -∆ u(1 + ṽ2 ) = 0, ∂ t v -∆ v(1 + ũ2 ) = 0, -δ 2 ∆ũ + ũ = u, -δ 2 ∆ṽ + ṽ = v, (25)
still with Neumann boundary conditions and initial data u 0 , v 0 .

It is rather intuitive that for δ large, diffusion is dominant; this is also the case for small initial data thanks to the argument in section 2. Therefore, the appearance of patterns depends upon a relation between the average densities of populations u and v and the parameter δ. In order to study this in details, we begin with some notations

• the only possible constant steady state of the system is given by u = ũ = u 0 and v = ṽ = v 0 ,

• we denote by (λ > 0, w) the non-zero solutions to the Neumann eigenproblem

-∆w = λw,
∂ n w = 0 on ∂Ω we also denote by λ 1 the first eigenvalue for the Laplacian.

In order to investigate when the (in)stability of the constant steady state occurs, we study the linearized system:

               ∂ t u -(1 + v 0 2 )∆u -2 u 0 v 0 ∆ṽ = 0, ∂ t v -(1 + u 0 2 )∆v -2 u 0 v 0 ∆ũ = 0, -δ 2 ∆ũ + ũ = u, -δ 2 ∆ṽ + ṽ = v.
As usual, we look for solutions of type e µt (a, b, c, d)w. Such solutions should satisfy

         c = a 1+δ 2 λ , d = b 1+δ 2 λ , µa + λa(1 + v 0 2 ) + λ2 u 0 v 0 b 1+δ 2 λ = 0, µb + λb(1 + u 0 2 ) + λ2 u 0 v 0 a
1+δ 2 λ = 0, which may be written under the matrix form

  1 + v 0 2 2 u 0 v 0 1+δ 2 λ 2 u 0 v 0 1+δ 2 λ 1 + u 0 2   a b = - µ λ a b .
We denote by M = M ( u 0 , v 0 , δ, λ) this symmetric matrix.

The question of the stability of the constant steady state can now be formulated in terms of eigenvalues of the matrix M . It is unstable if µ > 0 and thus if M has negative eigenvalues. In this case, local behavior around the equilibrium should lead to segregation since the associated eigenvector to -µ/λ should satisfy a.b < 0. We have the following Lemma 4.3 If the initial populations u 0 and v 0 are large enough and the relaxation parameter δ is small enough, then the constant steady state is linearly unstable. More precisely, it occurs under the conditions γ := 4 u 0 2 v 0 2 -1 + u 0 2 1 + v 0 2 > 0,

δ 2 < 2 u 0 v 0 - 1 + u 0 2 1 + v 0 2 λ 1 1 + u 0 2 1 + v 0 2 . ( 26
)
The domain influences instability only through the smallness condition on δ 2 λ 1 (Ω) when the initial data are such that γ > 0.

Notice that the first condition ensures that the limiting system (δ = 0) has a negative 'diffusion' matrix D in the setting (3). Proof. As mentioned earlier, the constant steady state is unstable if the symmetric matrix M admits negative eigenvalues, i.e., if det(M ) < 0. We calculate

det(M ) = - 4 u 0 2 v 0 2 (1 + δ 2 λ) 2 + (1 + u 0 2 )(1 + v 0 2 ) ≥ -γ.
As det(M ) is a non-decreasing function of δ, with limit γ as δ → 0 we first need γ > 0 that is our first condition. The second condition gives the upper bound on δ to satisfy this inequality.

Numerical results

The theoretical results indicate that solutions of the relaxation system (15) remain bounded in L 2 . Therefore, we expect that the instability obtained for large initial data or a small δ (through Turing mechanism) should lead to stiff gradients.

We present several numerical tests for the particular cubic system [START_REF] Mimura | Pattern formation in interacting and diffusing systems in population biology[END_REF]. They aim at showing that (i) the conditions of Lemma 4.3 are accurate and describe the numerical transition to instability, (ii) stationary patterns are indeed obtained in this range of data with stiff gradients. These numerical results also show the variety of possible steady state, an interesting phenomena widely studied theoretically ( [START_REF] Lou | Diffusion, self-diffusion and cross-diffusion[END_REF] and the references therein). We have performed both 1D and 2D simulations in the following domains

• In interval Ω = [0, 1] (1D simulation)
• In rectangle Ω = [0, 2] × [0, 0.5] (in both cases |Ω| = 1)

• In unit square Ω =]0, 1[ 2
In 2D, the computations use an unstructured grid and a mixed finite element method for space and backward Euler scheme for time. The method is already presented in [START_REF] Marrocco | Numerical simulation of chemotactic bacteria aggregation via mixed finite elements[END_REF].

We recall the eigenvectors of Laplace operator with Neumann boundary condition: for Ω = [0, 1], e n (x) = cos(nπx) and particularly, the first nonzero eigenvalue is π 2 , associated to the eigenvector cos(π * x). For Ω = [0, 2] × [0, 0.5], the eigenvectors are given by e n,m (x, y) = cos( nπx 2 ) cos(2mπy), the first nonzero eigenvalue is π 2 /4.

We compare the theoretical formula of Lemma 4.3 and the numerical stability of the steady state. In all simulations we take u 0 = 2, v 0 = 1. In this case, instability might occur, since γ = 16 -10 = 6 > 0 and the limiting values of δ 0 are given in the table 1. Therefore, in a first series of numerical tests, we choose the parameters δ 2 = 0.26 < δ 2 0 and δ 2 = 0.27 > δ 2 0 in 1D and for δ 2 = 0.1 < δ 2 0 , and δ 2 = 0.11 > δ 2 0 in 2D. In both cases, we have obtained relaxation to constant equilibrium when δ is taken larger than the critical value (for all the initial data we have tested), and instability of the constant equilibrium when δ is smaller than the critical value. We illustrate the instability case with steady states in figure 3 for 1D simulations and in figure 1 for 2D simulations. For the 1D simulation, we took v 0 ≡ 1 and u 0 = 1.9 + 0.21 I {]0.1,0.6[} .

Next we study the singularity that occurs on the transients for small relaxation parameter δ. Numerical solutions show that strong oscillations occur. In figure 2 we depict, for the same initial data, the effect of δ on the solution at a given time.

A Appendix: Michel Pierre's estimate

Consider the problem

∂ t u -∆[a(t, x)u] = 0, u(t = 0) = u 0 ,
together with Neumann boundary condition in a bounded domain Ω. We denote Q T = (0, T ) × Ω). We assume that a(t, x) > 0 is smooth and u is a weak solution. We can also assume withour lack of generality that u 0 ≥ 0. Then we have the a priori estimate Lemma A.1 For any T > 0, we have

√ a u L 2 (Q T ) ≤ C(Ω) u 0 L 2 (Ω) + 2 u 0 √ a L 2 (Q T ) , (27)
where C(Ω) is the constant of Poincaré Wirtinger's inequality. Proof. Consider smooth functions F (t, x) and the solutions to the adjoint problem ∂ t v + a(t, x)∆v = F (t, x), v(t = T ) = 0, [START_REF] Pierre | Weak solutions and supersolutions in L 1 for reaction-diffusion systems[END_REF] still with Neumann conditions. We have

d dt Ω uv = Ω F u,
and thanks to the final condition for the adjoint problem,

- Ω u 0 v 0 = T 0 Ω F u. (29)
Multiplying ( 28) by ∆v, we get

Ω ∂ t v∆v + Ω a|∆v| 2 = Ω F ∆v,
integrating by parts on Ω , we obtain,

- d dt Ω |∇v| 2 2 + Ω a|∆v| 2 ≤ Ω ( F 2 2a + a 2 |∆v| 2 ),
which gives after integration in time, using again v(T ) = 0,

Ω |∇v 0 | 2 + T 0 Ω a|∆v| 2 ≤ T 0 Ω F 2 a ,
and by consequence,

∇v 0 L 2 (Ω) ≤ F √ a L 2 (Q T ) , (30)
√ a∆v L 2 (Q T ) ≤ F √ a L 2 (Q T ) . (31)
We need additionally a bound on v 0 that we derive as follows. We use again [START_REF] Pierre | Weak solutions and supersolutions in L 1 for reaction-diffusion systems[END_REF] to find

| Ω v 0 | = | T 0 Ω a∆v -F | ≤ T 0 Ω √ a √ a|∆v| + F √ a ,
which gives, thanks to the Cauchy Schwarz inequality and [START_REF] Shigesada | Spatial segregation of interacting species[END_REF],

Ω v 0 ≤ 2 √ a L 2 (Q T ) F √ a L 2 (Q T ) . (32)
Finally, we get using Poincaré-Wirtinger ineqality, [START_REF] Turing | The chemical basis of morphogenesis[END_REF] and then [START_REF] Plotnikov | Passage to the limit with respect to viscosity in an equation with a variable direction of parabolicity[END_REF],

Ω u 0 v 0 ≤ Ω u 0 (v 0 -v 0 ) + | Ω u 0 v 0 | ≤ C(Ω) u 0 L 2 (Ω) ∇v 0 L 2 (Ω) + 2 u 0 √ a L 2 (Q T ) F √ a L 2 (Q T ) ≤ C(Ω) u 0 L 2 (Ω) F √ a L 2 (Q T ) + 2 u 0 √ a L 2 (Q T ) F √ a L 2 (Q T ) .
Back to [START_REF] Pierre | Blowup in reaction-diffusion systems with dissipation of mass[END_REF], we conclude that

T 0 Ω F u = T 0 Ω F √ a √ au ≤ C(Ω) u 0 L 2 (Ω) + 2 u 0 √ a L 2 (Q T ) F √ a L 2 (Q T ) ,
which is equivalent to [START_REF] Okubo | Diffusion and ecological problems: mathematical models[END_REF].

B Energy for a particular cross-diffusion system

A particular choice of cross-diffusion terms in (5) permits for an energy inequality even for negative second order matrices. This is the case of the system

     ∂ t U -∆(U (1 + V 2 )) = 0, ∂ t V -∆(V (1 + U 2 )) = 0, (33)
still with Neumann boundary conditions and initial data U 0 , V 0 . For this system, the energy is given by E(x, t) := (1 + U 2 )(1 + V 2 ).

One can easily check that it holds ∂ ∂t E(x, t) = 2U (1 + V 2 )∆(U (1 + V 2 )) + 2(V (1 + U 2 ))∆(V (1 + U 2 )), which leads immediately to

d dt Ω (1 + U 2 )(1 + V 2 ) = -2 |∇(U (1 + V 2 ))| 2 -2 |∇(V (1 + U 2 ))| 2 ≤ 0.
It follows an a priori estimate in the space L ∞ t (L 2 x ) that completes the L p tx bound proved in section 3. The system [START_REF] Wang | The global existence of solutions for a cross-diffusion system[END_REF] is not always elliptic. This is related to the non-convexity of this energy (still for large data), an important difference with the Shigezada-Kawasaki prey-predator system which comes with a convex entropy functional ( [START_REF] Chen | Analysis of a parabolic cross-diffusion population model without selfdiffusion[END_REF]).

Table 1 :

 1 Critical value of the parameter δ 0 for Turing instability, computed from formula[START_REF] Novick-Cohen | Stable patterns in a viscous diffusion equation[END_REF].

Figure 1 :

 1 Figure 1: Initial condition (left) and steady state (right) in 2D simulations. The relaxation parameter δ 2 = .1 is small enough to fulfill condition (26). The scales for the solutions are not the same in the two figures.

(a) δ 2 = 2 . 10 - 3 (b) δ 2 = 2 . 10 - 4 (c) δ 2

 22103221042 = 5. 10 -5 (d) δ 2 = 2. 10 -5

Figure 2 :

 2 Figure2: Cuts, at a given time, in the y direction and in the middle of the domain Ω = (0, 1) 2 in 2D. The piecewise initial condition is also represented in dashed line. As expected strong oscillations occur with species seggregation. These oscillations are stronger when δ is smaller.

(a) time t = 0 (b) time t = 2 (

 02 c) time t = 5 (d) time t = 9

Figure 3 :

 3 Figure 3: Time evolution for a 1D simulation for δ 2 = 0.25 < δ 20 . This figure shows how a small perturbation is amplified and creates a steady pattern. Because δ is large (close to δ 0 ) there are not strong oscillations as in the case of smaller values.

Acknowledgment This research has been supported by INRIA-CONICYT scientific cooperation program.


Killian Barrere

Yann Soullard

Aurélie Lemaitre

Bertrand Coüasnon

A Light Transformer-Based Architecture for Handwritten Text Recognition

Keywords: Light Network, Hybrid Loss, Transformer, Handwritten Text Recognition, Neural Networks

Transformer models have been showing ground-breaking results in the domain of natural language processing. More recently, they started to gain interest in many others fields as in computer vision. Traditional Transformer models typically require a significant amount of training data to achieve satisfactory results. However, in the domain of handwritten text recognition, annotated data acquisition remains costly resulting in small datasets compared to those commonly used to train a Transformer-based model. Hence, training Transformer models able to transcribe handwritten text from images remains challenging. We propose a light encoder-decoder Transformer-based architecture for handwriting text recognition, containing a small number of parameters compared to traditional Transformer architectures. We trained our architecture using a hybrid loss, combining the well-known connectionist temporal classification with the cross-entropy. Experiments are conducted on the well-known IAM dataset with and without the use of additional synthetic data. We show that our network reaches state-of-the-art results in both cases, compared with other larger Transformer-based models.

Introduction

Handwritten Text Recognition (HTR) refers to the process of automatically recognizing the text written inside an image of a text-line, a paragraph or even whole pages, after a first step of document layout analysis. This task is valuable nowadays for a growing number of people as it enables the text to be available in a digitized format and resist in time. There is still a tremendous amount of document collections waiting to be processed, and HTR models have shown acceptable error rates on specific documents [START_REF] Strauß | Icfhr2018 competition on automated text recognition on a read dataset[END_REF]. This is mostly thanks to the recent advances and growing interest in deep learning approaches. However, HTR remains challenging for a variety of reasons. The variability of writing styles, degraded documents, the need for data matching documents and the lack of annotated records limit the abilities of current deep learning approaches.

Popular network architectures are based on both convolutional layers and long short-term memory layers, trained using the Connectionist Temporal Classification (CTC) loss function [START_REF] Graves | A novel connectionist system for unconstrained handwriting recognition[END_REF]. These layers result in an optical model, typically followed by a language model which aims to correct recognition errors. Fully Convolutional Networks have been investigated recently with the goal to significantly reduce training time by removing recurrent layers. Lately, few works based on Transformers [START_REF] Vaswani | Attention is all you need[END_REF] have been proposed for HTR [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF][START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF][START_REF] Wick | Transformer for handwritten text recognition using bidirectional post-decoding[END_REF]. Based on multi-head attention layers, a Transformer network proves to be an efficient alternative to recurrent layers. It enables a sequential analysis of the sequence thanks to positional encoding and efficient parallelism. Transformer-based models offer a great potential, but those architectures generally require a tremendous amount of annotated training data to be robust. This limits the efficiency of such networks in HTR tasks where annotated data are expensive.

In this paper, we propose a light encoder-decoder Transformer-based architecture for handwriting recognition. The network presented in this work is significantly smaller than traditional Transformer networks, which makes it lighter in the number of weights and easier to train on small datasets. Our architecture is trained using a hybrid loss combining both the CTC loss and the Cross-Entropy (CE) loss, which seems crucial.

The article is organized as follows. After reviewing related works in Section 2, our network architecture is presented with details in Section 3. Afterward, we present results obtained on the popular IAM dataset. As opposed to traditional Transformer approaches, we show that our approach reaches state-of-art results without any additional data nor transfer learning. Nevertheless, it is capable to outperform the other types of network architectures by using synthetic data.

Related Works

Standard approaches for HTR

Popular network architectures used for Handwriting Text Recognition combine both convolutional layers and recurrent layers. A number of convolutional layers are stacked at the beginning of the network to extract local features from textline images. Then, recurrent layers, and more specifically Bi-directional Long Short-Term Memory (BLSTM) layers are stacked to process the features sequentially and output character probabilities based on contextual dependencies. Such an architecture is frequently called a Convolutional Recurrent Neural Networks (CRNN) [START_REF] Bluche | Gated convolutional recurrent neural networks for multilingual handwriting recognition[END_REF][START_REF] Puigcerver | Are multidimensional recurrent layers really necessary for handwritten text recognition?[END_REF][START_REF] Soullard | Improving text recognition using optical and language model writer adaptation[END_REF][START_REF] Strauß | Icfhr2018 competition on automated text recognition on a read dataset[END_REF]. Models are generally trained using the well-known Connectionist Temporal Classification (CTC) loss [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF]. It enables to deal with label sequences of shortest length than predicted sequences, without any knowledge about character segmentation.

Encoder-decoder based architectures have also been investigated for HTR. They typically rely on an attention mechanism and a decoder based on LSTM to sequentially predict the characters [START_REF] Bluche | Scan, attend and read: End-to-end handwritten paragraph recognition with mdlstm attention[END_REF][START_REF] Doetsch | Bidirectional decoder networks for attention-based end-to-end offline handwriting recognition[END_REF][START_REF] Michael | Evaluating sequence-to-sequence models for handwritten text recognition[END_REF]. Michael et al. [12] propose to use a hybrid loss combining both a CTC loss applied to the encoder and a cross-entropy loss for the decoder. Such models can obtain low error rates on common datasets. However, they suffer from the lack of computation parallelization inherently due to recurrent layers, which impacts both training and inference time.

Recently, Fully Convolutional Networks (FCN) have been proposed for HTR. They refer to deep architectures composed of many convolutional layers and no recurrent layer. They often include the most recent innovations in deep learning, like gating mechanism or residual connections to obtain state of the art results [START_REF] Coquenet | End-to-end handwritten paragraph text recognition using a vertical attention network[END_REF][START_REF] Coquenet | Span: a simple predict & align network for handwritten paragraph recognition[END_REF][START_REF] Ingle | A scalable handwritten text recognition system[END_REF][START_REF] Yousef | Accurate, data-efficient, unconstrained text recognition with convolutional neural networks[END_REF]. These architectures benefit from the computational parallelism offered by convolutional layers, and hence can be trained faster than traditional architectures based on recurrent layers. However, they may require tremendous work to be optimized well and require data augmentation to attain state-of-theart results. By removing recurrent layers, Fully Convolutional Networks might struggle to learn long-range contextual dependencies, which can be useful in HTR.

CRNN architectures dominate in the field of HTR, thanks to their ability to learn local and long-range features. However, they highly suffer from the notable training time of recurrent layers. The last few years, Fully Convolutional Networks obtained state-of-the-art results but they might experience difficulties related to long-range contextual dependencies.

Transformer-Based Architectures

In natural language processing, multi-head attention have been proposed by Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF] inside the so-called Transformer model. They propose an effective alternative to recurrent layers, capable of handling broad context in a constant amount of operation while enabling efficient parallelism. Transformer models also started to gain interest in the field of HTR. Kang et al. [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF] propose an end-to-end Transformer that aims for both recognizing handwritten text and modeling the language. They obtain very low error rates by implementing a big network architecture that requires synthetic data to be trained efficiently. Singh et al. [START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF] propose a Transformer-based architecture to address the problem of full-page handwriting recognition. They obtain promising results on full-page recognition thanks to the use of synthetic data again. Wick et al. [START_REF] Wick | Transformer for handwritten text recognition using bidirectional post-decoding[END_REF] use a bidirectional Transformer architecture coupled with a voting mechanism and show that their architecture outperforms a standard Transformer-based architecture.

Transformer-based layers propose an efficient trade-off between CRNN and Fully Convolutional Networks. Multi-head attention layers offer indeed both parallelism and the ability to learn long-range contextual dependencies. However, to perform well, a Transformer-based architecture, methods from the state of the art rely on synthetic data. Such additional data require to be designed as a complement to the training data which might remain a challenging task. Handwritten datasets typically contains too few examples for a Transformerbased model to perform well [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF]. Synthetic data seem to be an efficient solution but designing valuable synthetic data might prove to be a complex task. While it takes efforts to design synthetic data, they may result in deteriorated performance, especially for difficult datasets like historical documents. In this work, by contrast, we aim for a Transformer-based architecture capable of obtaining state-of-the-art results without the need of additional data, while still benefiting from the usage of synthetic data. We propose a light encoder-decoder Transformer-based architecture that can be trained efficiently on datasets of limited size, without the need for additional data 1 . This section provides details about our network architecture.

Summary of the architecture

We propose an end-to-end trainable Transformer-based architecture for HTR. Our network architecture is illustrated in Fig. 1. Our Transformer-based architecture remains low in the number of parameters, with 6.9M parameters, compared with traditional Transformer networks that might use up to 100M parameters. The architecture follows the principle of encoder-decoder models as it is composed of two key parts: an encoder and a decoder.

The encoder takes as input text-line images and aims to extract and process visual features. Our encoder is principally based on convolutional layers and Transformer encoder layers (Fig. 2). More details about the encoder are disclosed in Section 3.2.

The decoder subsequently uses the output of the encoder to sequentially predict the character sequence written inside the image (character by character).

In addition, it also has access to the sequence of previously predicted characters. Thus the decoder might act as a language model at character level together with optical features from the input image. Section 3.3 provides more specific explanations about the decoder.

We train our models with a hybrid loss combining both the Connectionist Temporal Classification (CTC) loss [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF] and the Cross-Entropy (CE) loss. This is discussed further in Section 3.4.

Network Encoder

The encoder is inspired by traditional CRNN and by Transformer layers [START_REF] Vaswani | Attention is all you need[END_REF]. We propose replacing the recurrent layers from the CRNN architecture by Transformer encoder layers. These layers refer to the encoder block of the Transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF]. They are based on multi-head attention followed by a positionwise feed forward layer. Transformer encoder layers have the advantage of being more parallelizable than recurrent layers on GPU while being able to handle long-range contextual dependencies in a constant number of operations. Our convolutional Transformer encoder is illustrated in Fig. 2.

The first part of the encoder is composed of 5 convolutional blocks used to extract visual features from the image. Except the last one, each convolutional block is composed of a 2D convolutional layer with a kernel of size 3×3, a stride of 1 and no padding. The last convolutional block uses a kernel size of 4×2 to better match the shape of a character [START_REF] Bluche | Gated convolutional recurrent neural networks for multilingual handwriting recognition[END_REF][START_REF] Puigcerver | Are multidimensional recurrent layers really necessary for handwritten text recognition?[END_REF]. The number of filters in the convolutional layers are respectively equal to 8, 16, 32, 64 and 128. Each convolution layer is then followed by a LeakyReLU activation function. Following the activation function, we apply a layer normalization to ease the network training capabilities and increase the regularization capacities of the network. A 2 × 2 max pooling is used inside the first three convolutional blocks to decrease the size of intermediate feature maps. It also focuses the training process on the most impacting features and reduces the number of network parameters. Lastly, a dropout is applied with a probability of 0.2 at the end of each block.

Following the last convolutional block, a collapse layer is used to flatten the vertical dimension of the feature maps, therefore enabling us to easily work with Transformer layers. It is composed of a convolution layer with a kernel size of width 1 and height similar to the height of the input feature maps. We subsequently apply a LeakyReLU activation function followed by a layer normalization.

Following that layer, we use a dense layer to increase the hidden size from 128 to 256. Before the Transformer encoder layers, sinusoidal positional encoding [START_REF] Vaswani | Attention is all you need[END_REF] is added to the output of the dense layer. We then use 4 stacked Transformer encoder layers. Each Transformer layer uses a hidden dimension of 256 and is composed of self-attention layers based on multi-head attention with 4 heads. They are then followed by a position-wise feed-forward layer with an intermediate feature size of 1024. Residual connections are used between each sub-layer. A dropout with a probability of 0.2 is applied to each sub-layer output.

The encoder of our architecture remains light in its number thanks to a small convolutional backbone, while additionally using a few numbers of parameters inside of Transformer layers. Compared to other Transformer-based architectures that use heavier convolutional neural networks like ResNets [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF][START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF], our architecture only uses 5 convolutional layers. It results in a total of only 237k parameters for the convolutional backbone, which is lower than most of the convolutional backbone used in other Transformer-based architectures [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF][START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF]. In addition, we maintain the number of intermediate neurons small inside Transformer layers, with a hidden size of 256. Heavier Transformer-based architectures might use up to 1,024 neurons inside Transformer layers.

Transformer Decoder

Our encoder-decoder Transformer-based model then processes the output from the encoder along with the sequence of previously predicted characters (or the target sequence shifted right when training with teacher forcing). An illustration of the decoder is shown in Fig. 1.

Our Transformer-based decoder uses both the output from the encoder part for mutual (or encoder-decoder) attention and the sequence of predicted characters. Sinusoidal positional encoding [START_REF] Vaswani | Attention is all you need[END_REF] is added to the output of the encoder. Despite the fact that our encoder already includes a positional encoding inside, we nevertheless find it beneficial to add a second positional encoding to the output of the encoder. Similarly to including an additional loss in the middle of a dense network, we believe that re-adding the information of the position in the middle of the architecture might help our architecture to converge. In a similar way, we apply positional encoding to the sequence of predicted characters, after applying a character-level embedding and before feeding it to the decoder part.

The Decoder is composed of a stack of 4 Transformer decoder layers [START_REF] Vaswani | Attention is all you need[END_REF]. Each layer takes as input the output from the previous layer or the embedded character sequence for the first layer. It is composed of a self-attention layer, using multi-head attention. Following the first sub-layer, a multi-headed encoderdecoder attention is applied. It uses the sequence coming from the encoder and apply weighted attention based on the encoder output and the output from the previous decoder sub-layer. Each attention sub-layer uses a hidden size of 256 and 4 heads. Lastly, we apply a position-wise feed forward layer composed of two dense layers with an intermediate feature size of 1,024. As used in the encoder, a residual connection is applied to each sub-layer and dropout of probability 0.2 is used.

Hybrid Loss

To train the entire model, we use a hybrid loss combining both the Connectionist Temporal Classification (CTC) loss function [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF] and a Cross-Entropy (CE) loss function. Such a hybrid loss has been introduced fist in the domain of HTR by Michael et al. [START_REF] Michael | Evaluating sequence-to-sequence models for handwritten text recognition[END_REF] to train an encoder-decoder model. In this work, both losses are linearly combined as follow:

L = λ • L CT C + (1 -λ) • L CE (1)
where λ balances between the CTC loss and CE loss. For a given input observation x = (x 1 , ..., x T ) of length T with label y = (y 1 , ..., y L ) of length L, we have:

L CT C = -ln π∈B -1 (y) p(π|x) and L CE = - L i=1 y i • log(ŷ dec i ) (2)
with ŷdec i the probability to predict y i in output of the decoder and π ∈ B -1 (y) a path from the encoder output that produces the label sequence y by applying the function B that maps the output sequence2 of length T to a sequence of length L with T > L. Note that the CTC loss requires the probability of each character per frame in input with an additional blank label that refers to predict no character. The function B removes all repeated labels in the path and then removes blank labels (see [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF] for more details).

Thus, the CE loss is applied using the output from the decoder, whereas the CTC is used with the output of the encoder. The decoder is therefore only trained using gradients coming from the CE loss, while the network encoder is trained using gradients coming from both the CTC and CE losses after backpropagation through the decoder.

Both losses require the probabilities for each class which are the characters and the CTC Blank label for the CTC loss and the characters and an endof-sequence token for CE loss. Therefore, we apply a dense layer followed by a softmax at the end of the encoder and the decoder. Unlike the architecture proposed by Michael et al. [START_REF] Michael | Evaluating sequence-to-sequence models for handwritten text recognition[END_REF], the character probabilities in output of the encoder are unused in input of the decoder. As illustrated in Fig. 1, the character probabilities are only used for the CTC loss and the output of the last hidden layer of the encoder is given in input of the decoder.

Experiments and results

Handwritten datasets are generally too small to get traditional Transformerbased architectures that perform well. As stated before, additional data are generally used to improve the performance of such architectures. By contrast, we aimed for a relatively light Transformer-based architecture to perform well both with and without additional data.

In this section, we demonstrate that, compared to other Transformer-based architectures, our light Transformer architecture reaches state-of-the-art results without requiring any additional data. To prove the efficiency of our network, we conduct experiments on the well-known IAM dataset (1) without using additional data; (2) using synthetic data as often done in other works.

We start by introducing experimental settings as well as the process used to generate synthetic data. We investigate the impact on performance of the main components from our network. Later, we compare the results obtained with a larger version of our light architecture and show that a big architecture is unrequired for a task of HTR. We then show the interest of using a hybrid loss that include a CTC loss at the end of the encoder part. We conclude the section by comparing the performance of our architecture with the ones of other methods from state of the art.

Handwritten Text-line Data

IAM The IAM offline handwriting database [START_REF] Marti | The iam-database: an english sentence database for offline handwriting recognition[END_REF] is probably the most popular dataset in the domain. It is composed of text-line images of modern English handwriting, produced by several writers that have been asked to write a specific text. It has been extensively used in the literature.

To conduct our ablation study and compare our network with state of the art methods, we use the aachen split. While this is not the original split, the aachen split, is commonly used in the field to compare results. This split provides 6,482 text-line images in training, 976 for validation and 2,915 images to evaluate the performance of the model.

Synthetic Data

To further investigate our model, we use generated synthetic data in addition to the training dataset. Recent Transformer architectures in the fielduse synthetic data, and report significant gains by using such data [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF][START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF].

We use textual data extracted from a collection of English Wikipedia articles. We then generate text-line images by using handwritten fonts 3 . Generated textline images could be more realistic thanks to the recent advances on generative models [START_REF] Bhunia | Handwriting transformers[END_REF]. However, this approach would also require training data, while we aim at performing well with limited amount of real training data. We generate various handwritings by setting different stroke widths and slant angles. Lastly, to simulate real handwritten variations, we apply some image transformations close to a usual data augmentation pipeline. We apply elastic distortions, vary the perspective of the image and add some noise to the generated image. Some examples of generated synthetic data are displayed in Fig. 3. Synthetic text-line images are generated on the fly and are combined with the training dataset. In our experiments using synthetic data, we use 10,000 synthetic text-line images for each training epoch in addition to every real example from the IAM training dataset.

Experimental Settings

In this section, we describe the experimental settings we use for each of the proposed experiments, unless specified otherwise.

We train our model using teacher forcing with a probability of 1 by feeding the target sequence shifted right to the decoder. We employ masking inside the layers of self-attention in the decoder, therefore ensuring that each prediction of a character only depends on the sequence of characters before it. It follows the identical principle used by Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF]. In addition to the usual gain in convergence speed, teacher forcing also allows our model to be trained in a parallel fashion, using a single decoder pass. Hence, teacher forcing helps in reducing training times. At test time, characters are predicted step by step, by applying one decoder pass for each character to be produced. The sequence of characters that have been previously predicted is provided in input of the decoder to predict the following character. We decode until an end-of-sequence token is predicted or up to 128 characters. Meanwhile, at training time, we use the length of the target sequence to reduce training times.

We use a custom learning rate policy as proposed by Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF] with a first linear ramp for a warm-up phase of 4,000 steps. We then decay the learning rate following an invert square root function. When training with a hybrid loss, we use a value of λ = 0.5 to train our models as the two losses are of equivalent orders of magnitude.

Input images are resized to a fixed height of 128 pixels while keeping the aspect ratio. Following that, we apply data augmentation as usually done in the field to virtually increase the number of training examples. We then randomly apply elastic transform, random erosion or dilatation, random perspective and random padding both on the left and right size of the image. Images are then standardized and we add a gaussian noise. Each transformation is applied with a probability of 0.2. Images are then padded according to the largest image in each batch, and a similar process is applied to the target sequence.

To compare results with others, we measure our models performance with both Character Error Rate (CER) and Word Error Rate (WER). It is computed using the Levenshtein distance (also called edit distance), by measuring the number of inserted, replaced and deleted characters (respectively words) between the predicted text and the ground-truth. That distance is then normalized by the length of the ground-truth.

Ablation Study of the Main Components of our Network

In this section, we investigate the impact on error rates of the main components of our architecture (the impact of the hybrid loss will be discussed later). We compare the following networks; each trained and evaluated on the IAM dataset with and without synthetic data:

-Light Transformer corresponds to our proposed architecture as detailed in Section 3. -CTNN is our network encoder, trained and performing as a standalone. It is trained with CTC loss. -CRNN is a common and popular network (it refers to a baseline architecture) composed of convolutional and recurrent layers. It uses the same convolutional layers than our Light Transformer (presented in Section 3.2) followed by a stack of four BLSTMs layers with a hidden size of 256. It is similar to CTNN, but uses recurrent layers instead of Transformer layers and is also trained using CTC loss. -CRNN + Decoder combines both the CRNN and our network decoder.

Compared to Light Transformer, we replaced transformer layers in the encoder by recurrent layers.

Results are available in Table 1. First, we discuss results without synthetic data. Our Light Transformer architecture is able to obtain better results than a baseline CRNN architecture on the IAM dataset. Compared to this baseline, our architecture is able to obtain a CER 22% better relatively without using any additional data, but with more parameters. When we remove the decoder from our Light Transformer, we observe that the resulting architecture (CTNN) performs worse. However, it is nevertheless able to obtain a better CER and WER than the baseline architecture. Besides, using a decoder in addition to a CRNN architecture seems to bring worse CER, even if the WER is improved. We believe that when using a Transformer-based decoder, the context from an architecture based on recurrent layers might not be good enough. Lastly, when we compare recurrent layers and Transformer layers in the encoder (CRNN + Decoder versus Light Transformer and CRNN versus CTNN), we observe a gain in the network performance.

Using synthetic data with our Light Transformer architecture, we observe a major improvement of the CER and WER. This gain seems to be far more beneficial for the decoder part, while it might even be unbeneficial for the encoder part performing as a standalone. On the one hand, synthetic data may not fit the real data well and this may affect more specifically the encoder part for which the images are given in input. On the other hand, the decoder trained to predict one character at a time might be able to act as a language model, benefiting from the amount of synthetic training data to learn the language. In addition, we also observe that the WER highly benefits from the usage of a decoder with, and without synthetic data.

As expected regarding a Transformer architecture, synthetic data bring a major improvement for our proposed architecture. Nevertheless, our architecture is able to perform well without any additional data, and we highlight the relevance of the different parts of our network.

Benefits of Using a Light Architecture

Traditional Transformer-based architecture are relatively heavier (up to 100M parameters) than our network (6,9M of parameters) and they generally perform poorly without additional data. To highlight the advantages of our light architecture, we propose a variant of it, resulting in a similar architecture with more parameters. Our scaled variant of the light architecture is mostly based on the same parameters described in Section 3. We use twice as many neurons inside Transformer layers for both the encoder and the decoder. While our light architecture uses a hidden size of 256, our scaled version uses 512 neurons instead. We also increase the number of attention heads to 8, therefore resulting in the identical number of neurons per head as in our light architecture. Lastly, intermediate feature size of position-wise feed-forward layers is also doubled, with 2,048 features instead of 1,024. It results in a scaled version of our architecture using 28M parameters instead of 6.9M parameters. We refer to this scaled version as Large Transformer. Table 2 shows the results obtained by both our light and large Transformer architectures.

We obtain slightly better results with our Light Transformer architecture, therefore indicating the heavy architecture might not prevail over our light architecture. This might be explained by the fact that the number of parameters is lower. Our light architecture might be easier to train on a scenario in which

Interest of the Hybrid Loss

We evaluate how important it is to train our model with a hybrid loss composed of both the Connectionist Temporal Classification (CTC) loss and a Cross-Entropy (CE) loss. To do so, we compare the results obtained with our light architecture trained with CE only (without the CTC loss at the end of the encoder) and with a hybrid loss. Results are available in Table 3.

Using a hybrid loss seems to be crucial in our architecture, as our light Transformer model trained with a hybrid loss is able to attain lower error rates when compared to a training achieved with CE loss only. Despite the fact that our Transformer architecture relies on residual connections (that help gradients to flow), this may not be enough to efficiently train our architecture. Using a hybrid loss improves this. This is especially accurate when the model is only trained with the given original dataset, as the amount of data is too small for traditional Transformer-based models. However, even by using additional synthetic data, we find it beneficial to include a hybrid loss inside our training procedure. No matter the amount of training data, we believe a hybrid loss is essential four our light architecture to converge quickly and efficiently. Adding an intermediate loss at the end of the encoder seems to assist the model in efficiently training the first layers in a deep architecture. In addition, the CTC loss is dedicated to recognize characters from input sequence of longer length which seems to be useful in our application. From our point of view, using a hybrid loss might help training an encoder-decoder network and combining the CTC and CE losses seems to be efficient for HTR.

Comparison with the State of the Art

The performance of our architecture is compared with the main results from the state of the art on the IAM dataset (Table 4). To allow a proper comparison, we do not include methods that use an explicit Language Model to correct outputs from the optical model. This is a general tendency, as most results from the state of the art use neither language model nor lexicon. We only report the CER as most of the works do not present WER results. However, we have shown WER results obtained with our light Transformer in the previous tables.

Without using synthetic data, we obtain a CER on the IAM test set of 5.70%. Compared to models based on FCN proposed by Yousef et al. [START_REF] Yousef | Accurate, data-efficient, unconstrained text recognition with convolutional neural networks[END_REF] and Coquenet et al. [START_REF] Coquenet | End-to-end handwritten paragraph text recognition using a vertical attention network[END_REF], our architecture obtains worse results without synthetic data, as other transformer-based models. To the best of our knowledge, there are no published results on FCN with synthetic data included to which we can compare fairly. An FCN model might obtain better results than our approach when dealing with limited real samples while benefiting from the addition of synthetic data. However, we believe a transformer-based model will benefit more than an FCN from the addition of synthetic data due to the fact that it includes a decoder, capable of learning to model the language to some extent.

Compared to other Transformer-based networks, our architecture is able to obtain state-of-the-art results. As opposed to the Transformer model proposed Table 4. Results on the IAM dataset (aachen split). We compare our architecture to methods based on CRNN [START_REF] Michael | Evaluating sequence-to-sequence models for handwritten text recognition[END_REF], FCN [START_REF] Yousef | Accurate, data-efficient, unconstrained text recognition with convolutional neural networks[END_REF][START_REF] Coquenet | End-to-end handwritten paragraph text recognition using a vertical attention network[END_REF] and Transformer [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF][START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF][START_REF] Wick | Transformer for handwritten text recognition using bidirectional post-decoding[END_REF].

Model

# params. IAM IAM + Synth. Data CER (%) CER (%) CRNN + LSTM [START_REF] Michael | Evaluating sequence-to-sequence models for handwritten text recognition[END_REF] 5.24 FCN [START_REF] Yousef | Accurate, data-efficient, unconstrained text recognition with convolutional neural networks[END_REF] 3.4M 4.9 VAN (line level) [START_REF] Coquenet | End-to-end handwritten paragraph text recognition using a vertical attention network[END_REF] 1.7M by Kang et al. [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF], our light architecture performs well even without additional data, while Singh et al. [START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF] do not present result without synthetic data. Still, by including synthetic data to the training set, our model obtains even lower error rates, reaching a CER of 4.76%. Our proposed architecture reaches results close to the best performing architecture, while using 14 times less parameters. In a general manner, our light architecture remains low in its number of parameters. Our model only uses 6.9M parameters which is far lower than the Transformer architectures from the state of the art [START_REF] Kang | Pay attention to what you read: Non-recurrent handwritten text-line recognition[END_REF][START_REF] Singh | Full page handwriting recognition via image to sequence extraction[END_REF][START_REF] Wick | Transformer for handwritten text recognition using bidirectional post-decoding[END_REF]. Hence, we expect our model to be trained much faster in comparison, while requiring less resource.

Wick et al. [START_REF] Wick | Transformer for handwritten text recognition using bidirectional post-decoding[END_REF] propose a medium-sized Transformer (Forward Transformer) which has 13M parameters. In their work, they propose to duplicate the architecture to perform both a forward and backward scan of the text-line image, and reduce the error rates. The resulting bidirectional Transformer attains similar CER to the one obtained by our architecture. Furthermore, our contribution is compatible with their work by adding a backward Transformer which might improve the results obtained by our network.

To conclude, compared to other Transformer-based networks, our light Transformer architecture reaches state of the art results, no matter the amount of training data with significantly less parameters.

Conclusion and Future Works

We present a light encoder-decoder network based on Transformer-like architecture for handwritten text recognition. We highlight the relevance to use a hybrid loss combining linearly the connectionist temporal classification loss and the cross-entropy loss, which seems crucial for encoder-decoder architecture to be trained efficiently. Compared to other Transformer-based models, our architecture remains light in the number of parameters and does not require any additional data to be trained efficiently. Our network architecture reaches results at the level of state-of-the-art Transformer-based models, with a 5.70% CER on the IAM test set. Using synthetic data, our architecture is able to attain a 4.76% CER, close to the best performing network.

As future works, we would like to apply our architecture to historical documents in which the number of annotated data is even more critical. Using a light architecture might be beneficial considering the moderate amount of training data, while a Transformer-like architecture based on multi-head attention might be useful for even more difficult writings compared to modern texts. Synthetic data prove to be an efficient solution to the limited number of training data, but the context of historical documents will require a thorough design of the generation process of synthetic data.

3Fig. 1 .

 1 Fig. 1. Our encoder-decoder Transformer-based architecture. Our architecture is composed of an encoder combining convolutional layers and Transformer layers (Section 3.2), and of a Transformer-based decoder (Section 3.3).

  resized to fixed height and padded to fixed width

Fig. 2 .

 2 Fig. 2. Our Convolutional Transformer Encoder. This encoder is composed of a first stack of convolutional layers followed by Transformer Encoder layers.

Fig. 3 .

 3 Fig. 3. Examples of generated English synthetic data, trying to match the IAM dataset.

Table 1 .

 1 Comparison between the results obtained by our Transformer-based architecture and a CRNN architecture on the test set of the IAM dataset (aachen split), with and without synthetic data added.

	Architecture	# params.		IAM	IAM + Synth. Data
			CER (%) WER (%) CER (%) WER (%)
	CRNN (Baseline)	1.7M	6.14	23.26	5.66	21.62
	CRNN + Decoder	5.5M	6.92	21.16	5.36	18.01
	CTNN (Encoder)	3.2M	5.93	22.82	6.15	24.02
	Light Transformer	6.9M	5.70	18.86	4.76	16.31

Table 2 .

 2 Comparison of the results obtained by our light Transformer-based architecture (Light Transformer) and a heavy version of it (Large Transformer) on the test set of the IAM dataset (aachen split). Even so, by adding synthetic data, we find that our light Transformer-based architecture performs better than Large Transformer version. Hence, we believe a big Transformer architecture is unnecessary to obtain good results on a specific type of documents.In addition, a light architecture would require less resource to be trained efficiently. The training cost could then be reduced both in training time and in the number of training examples.

	Architecture	# params.		IAM	IAM + Synth. Data
		CER (%) WER (%) CER (%) WER (%)
	Light Transformer	6.9M	5.70	18.86	4.76	16.31
	Large Transformer	28M	5.79	19.67	4.87	17.67
	we have a relatively low number of annotated training data for training tra-
	ditional Transformer-based models.

Table 3 .

 3 Comparison of the results obtained with our model trained with or without a hybrid loss on the test set of the aachen split of the IAM dataset. We present results with and without synthetic data.

	Loss		IAM	IAM + Synth. Data
	Function(s)	CER (%) WER (%) CER (%) WER (%)
	CE only	10.29	26.36	6.76	19.62
	Hybrid (CTC + CE) 5.70	18.86	4.76	16.31

By additional data, we mean data from another dataset than the one which is studied and synthetic data. We nevertheless use data augmentation techniques to improve our models.

Note that T is equal to T if no reduction of the input sequence length is applied inside the network.

The fonts are available on https://fonts.google.com and https://www.dafont.com.

Acknowledgments

This work was granted access to the HPC resources of IDRIS under the allocation 2021-AD011012550 made by GENCI.