A Decelle

C Furtlehner

Exact Training of Restricted Boltzmann Machines on Intrinsically Low Dimensional Data

The restricted Boltzmann machine is a basic machine learning tool able, in principle, to model the distribution of some arbitrary dataset. Its standard training procedure appears however delicate and obscure in many respects. We bring some new insights to it by considering the situation where the data have low intrinsic dimension, offering the possibility of an exact treatment and revealing a fundamental failure of the standard training procedure. The reasons for this failure -like the occurrence of first-order phase transitions during training -are clarified thanks to a Coulomb interactions reformulation of the model. In addition a convex relaxation of the original optimization problem is formulated thereby resulting in a unique solution, obtained in precise numerical form on d = 1, 2 study cases, while a constrained linear regression solution can be conjectured on the basis of an information theory argument.

Recent advances in machine learning (ML) pervade now many other scientific domains including physics by providing new powerful data analysis tools in addition to traditional statistical ones. The restricted Boltzmann machine (RBM) could be considered as one of these when already a large spectrum of possible uses has been proposed in physics [START_REF] Torlai | Learning thermodynamics with Boltzmann machines[END_REF][START_REF] Carleo | Solving the quantum manybody problem with artificial neural networks[END_REF][START_REF] Nomura | Restricted Boltzmann machine learning for solving strongly correlated quantum systems[END_REF][START_REF] Melko | Restricted Boltzmann machines in quantum physics[END_REF][START_REF] Tubiana | Learning protein constitutive motifs from sequence data[END_REF]. Introduced more than three decades ago [START_REF] Smolensky | chapter 6: Information Processing in Dynamical Systems: Foundations of Harmony Theory[END_REF], the RBM played an important role in early developments of deep learning [START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF]. It is a special case of generative models [START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Salakhutdinov | Deep Boltzmann machines[END_REF] that remains very popu- lar thanks to its simplicity and effectiveness when applied to moderately high dimensional data [START_REF] Hjelm | Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks[END_REF][START_REF] Hu | Latent source mining in fmri via restricted Boltzmann machine[END_REF][START_REF] Yelmen | Creating artificial human genomes using generative neural networks[END_REF]. It is a twolayers undirected neural network that represents the data in the form of a Gibbs distribution of visible and latent variables (see Fig. 1):

s 1 s n v σ 1 σ n h s i σ j W ij
p(s, σ) = 1 Z[Θ] exp i,j s i W ij σ j - Nv i=1 η i s i - N h j=1 θ j σ j .
(1) The former noted s = {s i , i = 1 . . . N v } correspond to explicit representations of the data while the latter noted σ = {σ j , j = 1 . . . N h } are there to build arbitrary dependencies among the visible units. They play the role of an interacting field among visible nodes. While many different types of variables can be considered, we take here spin variables s i , σ j ∈ {-1, 1} for definiteness. Θ = (W, η, θ) are the parameters, W being the weight matrix, η and θ are local field vectors called respectively visible and hidden biases. Each weight vector associated with a given hidden unit and its corresponding bias defines an hyperplane partitioning the visible space into two regions corresponding to the hidden unit being activated or not (see Fig. 1). Z[Θ] is the partition function of the system. The joint distribution between visible variables is then obtained by summing over hidden ones. Learning the RBM amounts to find Θ such that generated data obtained by sampling this distribution should be statistically similar to the training data. The standard method to infer the parameters is to maximize the log-likelihood (LL) of the model [START_REF] Carleo | Solving the quantum manybody problem with artificial neural networks[END_REF] with Data denoting the average over training data. This is a nontrivial optimization problem in two respects: it is nonconvex and the loss function -L[Θ] is difficult to estimate because log Z[Θ] is not tractable. Nevertheless, the gradient ∇ Θ L[Θ] can be written in terms of simple response functions of the RBM. These can be estimated approximately via Monte Carlo methods, leading to various algorithms called contrastive divergence [START_REF] Hinton | Training products of experts by minimizing contrastive divergence[END_REF] with possible refinements [START_REF] Tieleman | Training restricted Boltzmann machines using approximations to the likelihood gradient[END_REF][START_REF] Fischer | Training restricted Boltzmann machines: An introduction[END_REF].

L[Θ] = j log cosh i W ij s i -θ j Data - i η i s i Data -log Z[Θ] ,
The similarity of the RBM with disordered spin systems has raised a lot of interest in statistical physics. Mean-field-based training algorithms and analyses have been proposed [START_REF] Gabrié | Training restricted Boltzmann machine via the TAP free energy[END_REF][START_REF] Huang | Advanced mean-field theory of the restricted Boltzmann machine[END_REF][START_REF] Takahashi | Mean-field inference in Gaussian restricted Boltzmann machine[END_REF][START_REF] Mézard | Mean-field message-passing equations in the Hopfield model and its generalizations[END_REF], a mapping with the Hopfield model has been found in [START_REF] Barra | On the equivalence of Hopfield networks and Boltzmann machines[END_REF], retrieval capacity has been characterized in [START_REF] Barra | Phase diagram of restricted Boltzmann machines and generalized Hopfield networks with arbitrary priors[END_REF][START_REF] Barra | Phase transitions in restricted Boltzmann machines with generic priors[END_REF] and compositional mechanisms analyzed in [START_REF] Agliari | Multitasking associative networks[END_REF][START_REF] Monasson | Emergence of compositional representations in restricted Boltzmann machines[END_REF] (see more recent references, e.g. in [START_REF] Decelle | Restricted Boltzmann machine, recent advances and mean-field theory[END_REF]).

In previous works [START_REF] Decelle | Spectral dynamics of learning in restricted Boltzmann machines[END_REF][START_REF] Decelle | Thermodynamics of restricted Boltzmann machines and related learning dynamics[END_REF] we studied to what extent the learning process of the RBM is reflected in the spectral dynamics of the weight matrix: a certain number of modes, corresponding to principal modes of the data, emerge from a Marchenko-Pastur bulk at initialization and condense to build up a structured ferromagnetic phase. Here we focus on the latter and most difficult stage and show that the two main difficulties (nontractability and nonconvexity) of the training can be addressed in the special case, where a flat intrinsic space of low dimension has been identified in the first stage.

Effective theory in the ferromagnetic phase. -Let us first disentangle the contribution of the collective modes corresponding to the information stored from the data (the ferromagnetic and difficult part) from the other degrees of freedom corresponding to the noise (the paramagnetic and easy part). After summing over the hidden variables in (1) the visible distribution reads

P [s|Θ] = 1 Z[Θ] exp N h j=1 log cosh Nv i=1 W ij s i -θ j - i η i s i (3)
As in [START_REF] Decelle | Thermodynamics of restricted Boltzmann machines and related learning dynamics[END_REF] the weight matrix is expressed via its singular value decomposition (SVD)

W ij = min(Nv,N h ) α=1 w α u α i v α j ,
with w α , u α and v α representing, respectively, the singular values and left and right singular vectors. Assume that some modes α ∈ {1, . . . d} have condensed along a magnetization vector denoted m = (m 1 , . . . , m d ), i.e. that s α = m α = O(1), with by definition

s α def = 1 √ N v Nv i=1 s i u α i .
For a RBM trained on some data, d would represent their intrinsic dimension at least locally. The corresponding modes u α i can, in principle, be obtained directly from the SVD of the data or emerge naturally from the linear regime of the learning process described in [START_REF] Decelle | Thermodynamics of restricted Boltzmann machines and related learning dynamics[END_REF]. These magnetization constraints define a canonical statistical ensemble. We look for a change of variables s -→ (m, s ⊥ ), where the original spin variables are replaced by a set of d continuous variables and N [m] transverse weakly interacting spin variables. N [m] is related to the configurational entropy per spin

S[m] = N [m]
Nv log(2) under these constraints. Thanks to a large deviation argument S[m] is the Legendre transform of (see SM, Ap-pendix A)

Φ[µ] = 1 N v i log cosh N v d α=1 u α i µ α ,
with µ[m] given implicitly by the constraints [29]

m α = 1 √ N v Nv i=1 u α i tanh N v d β=1 u β i µ β , α = 1, . . . d. (4
Z[Θ] = D⊂[-1,1] d d d m e -NvF [m|Θ] , (5)
where the canonical free energy

F[m|Θ] = F [m|Θ] + F ⊥ [m|Θ]
is decomposed into two contributions coming respectively from the condensed modes and the transverse fluctuations (See SM, Appendix B):

F [m|Θ] = -S[m] + d α=1 η α m α -V [m|Θ], (6)
F ⊥ [m|Θ] = - 1 N v log 1 2 N [m] s ⊥ e -H eff [s ⊥ |m,Θ] , (7)
(η α def = 1 √ Nv i η i u α i
) which are respectively associated with a potential function for the magnetizations

V [m|Θ] = 1 N v N h j=1 log cosh N v d α=1 w α m α v α j -θ j , (8)
and an effective Hamiltonian H eff for the transverse degrees of freedom given in the form of a disordered Ising model of N [m] spins with paramagnetic-like state of order defined for each m (see SM, Appendix C). The default entropy (N [m] log(2)) of the transverse variables is assigned by convenience to F so that F ⊥ vanishes when H eff = 0. In the following we focus on the dominant aspects of the training process resulting from the expression F .We leave aside specific training problems associated with the transverse fluctuations, like e.g. the emergence of spurious modes, which will be analyzed elsewhere in detail thanks to this effective Hamiltonian formalism.

Coulomb formulation and linear regression. -The potential term in F , which acts on the magnetization m representing here the position of a particle in a ddimensional space, can be re-written as (See SM Appendix D)

V [m|Θ] = dndz q(n, z)|n T m -z|, (9)
after introducing in the space O(d) × R, the density [START_REF] Salakhutdinov | Deep Boltzmann machines[END_REF] of latent features, δ ν (x) = ν 2 1tanh 2 (νx) being a "smoothed" delta function of width ν -1 , with

q(n, z) = 2 N v N h j=1 ν j δ νj z - θ j ν j δ(n -n j ) ≥ 0,
ν j = Ã N v d α=1 w 2 α v α j 2 (11)
n α j = √ N v ν j w α v α j ( 12
)
The kernel |n T j m -z| represents the Coulomb potential exerted by a uniformly charged hyperplane, defined by its normal vector n and its distance z to the origin, on a charge located at m. As a result, each feature j corresponds also to a charged hyperplane of normal vector n j , offset z j = θ j /ν j and finite thickness ν -1 j . At this point let us remark that the w α control through [START_REF] Hjelm | Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks[END_REF] both the strength of the Coulomb interaction via (9,10) and the charged hyperplanes thickness; the right singular vectors projections v α j control on their side the orientation of these hyperplanes in the intrinsic space through [START_REF] Hu | Latent source mining in fmri via restricted Boltzmann machine[END_REF]. Note that the visible bias vector η is equivalent to some surface charge placed at the edge of the domain of m and can be incorporated into q(n, z). The log-likelihood of the RBM has then three terms

L[Θ] = -E p V [m|Θ] + F ⊥ [m|Θ] -log Z[Θ] ,
where log Z[Θ] is a complex self-interaction of the charged hyperplanes among each other; E p F ⊥ [m|Θ] is in principle small, especially if there is no transverse bias; finally, [START_REF] Yelmen | Creating artificial human genomes using generative neural networks[END_REF] takes the form of a repulsive Coulomb interaction between training data points represented by the empirical distribution p(m), and positively charged hyperplanes. It corresponds to a slight extension of the RBM model in terms of more general activation function (encompassing RELU [START_REF] Nair | Rectified linear units improve restricted Boltzmann machines[END_REF] for instance and similar to [START_REF] Ping | Learning infinite RBMs with Frank-Wolfe[END_REF]), where each feature contribution in (3) comes with a non-negative weight q j to be optimized, while the features themselves defined by the pairs (n j , θ j ) are predefined. This formulation introduced here at first in a theoretical perspective to understand the RBM, can also be used in practice when the intrinsic space is identified in advance. Then letting w β = 0 for β > d results in F ⊥ independent of Θ and the optimization of L[Θ] (w.r.t. the features weights q(n, z)) becomes convex, this "Coulomb" formulation being in the exponential family. As a result the optimal solution can be obtained with good numerical precision thanks to a natural gradient ascent [START_REF] Amari | Natural gradient works efficiently in learning[END_REF] following the geodesics of the Fisher metric (See SM, Appendix G), the complexity being

E p V [m|Θ] = dmdndz p(m)|n T m -z|q(n, z),
O(N 3 f + N 2 f × N d p )
in the number of predefined features N f and of points N d p needed to compute Z[Θ] (and its derivatives) through [START_REF] Tubiana | Learning protein constitutive motifs from sequence data[END_REF]. Typically this remains tractable for d ≤ 3 and N f ≤ O(10 3 ) simply using a regular discretization of the feature space (n, z) ⊂ [-1, 1] d as shown in the next section. Additionally, an even more tractable approach bypassing the computation of Z[Θ], based on a linear regression seems plausible according to the following observations. In terms of the Coulomb charges density

ρ(m|Θ) = dndz q(n, z)δ n T m -z , (14)
resulting from a distribution q(n, z) of uniformly charged hyperplanes the marginal distribution of m reads

P (m|Θ) = 1 Z[Θ] e -NvF [m|Θ] , = e Nv S(m)+ dm ρ(m |Θ)K d (|m-m |)-F ⊥ [m|Θ] Z[Θ] with K d (|m-m |) the inverse of the d-dimensional Lapla- cian ∇ 2 d operator (See SM, Appendix D).
Assuming for the moment that ρ is not restricted to be of the specific RBM form [START_REF] Hinton | Training products of experts by minimizing contrastive divergence[END_REF], this relation can be explicitly inverted to match any smoothed version p (m) of the empirical distribution p(m):

ρ(m|Θ) = ∇ 2 d 1 N v log p (m) -S[m] + F ⊥ [m|Θ] ( 15
)
up to surface terms, provided that F ⊥ is independent of ρ. Doing that leads to overfit the data with a den- sity of Coulomb charges concentrated on the faces of the Voronoi cells enclosing the data points (see SM, Appendix E). To be meaningful this solution has to be projected on the "RBM" space, i.e. a density ρ of the form [START_REF] Hinton | Training products of experts by minimizing contrastive divergence[END_REF] corresponding to a finite number of features. The fact that any distribution ρ can be approximated to arbitrary precision by such a superposition of charged hyperplanes relates to the property that the RBM is a universal approximator [START_REF] Roux | Representational power of restricted Boltzmann machines and deep belief networks[END_REF]. The appropriate metric to perform such a projection is the Fisher metric [START_REF] Amari | Natural gradient works efficiently in learning[END_REF] and this ends up being equivalent to minimizing the Kullback-Leibler divergence (D KL ) between p(m) and P (m|Θ) i.e to maximizing the LL. Nonetheless, if we expect the optimal solution to be very close to p, we may use directly the Fisher metric estimated at the empirical point p thereby turning the problem into the following linear regression

Training distribution RBM distribution F [m] ρ(m) -1 -0.5 0 0.5 1 m - -1 -0.
Θ = argmin q E p F ⊥ [m] -S[m] - N h j=1 q j V j [m] 2 (16) of F ⊥ [m] -S[m] on the score variables V j [m] def = ∂ log(P (m|Θ)) ∂qj
conjugate to q j (see SM, Appendix F). Study cases. -To illustrate these statements first consider a dataset supported by a 1-d subspace given by the vector u i = 1/ √ N v with unbiased fluctuations along other directions. A rank one W = w 1 u 1 v 1T is assumed since we expect transverse modes to vanish from the linear stability analysis of the training given in [START_REF] Decelle | Thermodynamics of restricted Boltzmann machines and related learning dynamics[END_REF].

The relation (4) reduces then to the magnetization m = tanh(µ) along u leading in the Coulomb formulation to

F[m|q] = h(m) - N h j=0 q j |m -z j |, (q j ≥ 0) with h(m) = 1 2 (1 ± m) log(1 ± m).
The natural gradient ascent of the LL yields optimal solution as the one shown on Fig. 2. As is manifest on Fig. 2 the result is the linear regression ( 16) of

F ⊥ [m] -S[m] = h(m)
in terms of a piecewise linear function, where the break points correspond to the locations z j of the relevant features and q j the corresponding break of slope at these points. This involves, however, an implicit regularization which will be studied elsewhere, in order to maintain the regions free of data below h(m) in order to stay away from first-order transitions where the local Fisher metric would cease to be a meaningful approximation to the D KL . As a 2-d example we consider data concentrated in the subspace spanned by the vectors

u 1 i = 1/ √ N v and u 2 i = (-1) i / √ N v with irrelevant transverse fluctuations, hence assuming now W = w 1 u 1 v 1T + w 2 u 2 v 2T .
We have then a finite magnetization (m 1 , m 2 ) along each direction and the free energy considered in the Coulomb formulation reads

F[m|Θ] = 1 2 [h(m + ) + h(m -)] - N h j=1 q j |m 1 cos(ω j ) + m 2 sin(ω j ) -z j |,
where Discussion. -The physical picture of the RBM emerging here, in addition to identifying and disentangling via Eqs. [START_REF] Hjelm | Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks[END_REF][START_REF] Hu | Latent source mining in fmri via restricted Boltzmann machine[END_REF] the role played by some key factors, underlines the importance of two distinct aspects of learning a high dimensional distribution : the ordered part corresponding to global statistical patterns and the fluctuations around these patterns encoding possibly short range correlations or corresponding to noise. Under a flat intrinsic space hypothesis our formalism decouples them and gives indications of how to learn them separately in order to obtain high quality models that are needed in scientific applications, when default RBM algorithms are thwarted by low dimensional global patterns as we see in our experiments. Among many possible developments we foresee that the "Coulomb" convex relaxation could be used to fine-tune some otherwise poorly trained RBM, and opens the intriguing possibility of tackling unsupervised learning via regularized linear regressions.

m ± = m 1 ± m 2 ∈ [-1,
Appendix A: Canonical ensemble with magnetization constraints

The decomposition of the vector of visible variables s on the left singular basis (A2) represents the density of states (normalized to one) associated with the magnetization constraints m, S[m] the configuration entropy associated with these magnetizations and 2) representing the remaining number of degrees of freedom s ⊥ taken out of the N v initial ones. Note that there is a formal difficulty here because the size of the transverse variables vector s ⊥ depends explicitly on m. This however is not really a problem if consider s ⊥ to be a vector of size N v where the last N v -N (m) bits are frozen arbitrarily to 1, which is done in practice by defining the prior distribution

s α def = 1 √ N v Nv i=1 s i u α i , (
P prior [s ⊥ |m] def = 1 2 N [m] , with N [m] = N v S[m]/ log(
P prior [s ⊥ |m] def = 1 2 N [m] Nv =N [m]+1 δ(s ⊥ -1).
To avoid additional burden on the notations we keep this as implicit and s ⊥ always refers to the set of non-frozen variables. We want here to determine S[m] from (A2).

We have

E s∼U [s α ] = 0 and E s∼U [s α s β ] = 1 N v δ αβ ,
the second relation resulting from the orthogonality of the u α vectors. As a result, for large N v we have

P prior [m] = 1 (2π) d/2 exp - N v 2 d α=1 m 2 α . (A3)
This is valid as long as the magnetization are not too large (m

α = O(1/ √ N v )).
To study the regime where modes condense, i.e. when m α = O(1), we have to resort to large deviations estimations [START_REF] Touchette | The large deviation approach to statistical mechanics[END_REF]. With d assumed to be O(1), as N v → ∞ we expect in this regime a behaviour of the form

P prior [m] e -NvI[m] ,
where I[m] called the rate function, has 0 as minimum value and can be determined in the present situation thanks to the Gärtner-Ellis theorem from the moment generating function of P prior [m]. Denoting by µ = O(1) a conjugate d-dimensional vector and assuming that we can make sense of the following limit

Φ[µ] def = lim Nv→∞ 1 N v log E s∼U e Nv d α=1 mα(s)µα , = lim Nv→∞ 1 N v i log cosh N v d α=1 u α i µ α , I[m]
is then simply given by the Legendre-Fenchel transform of Φ:

I[m] = mµ[m] T -Φ µ[m] ,
with µ[m] implicitly given by (in principle when N v → ∞)

m α = lim Nv→∞ 1 √ N v Nv i=1 u α i tanh N v d β=1 u β i µ β , = E u∼pu u α tanh d β=1 u β µ β (A4)
where we assume in the last equality some limit p u of the joint empirical distribution of

u α = √ N v u α i when N v → ∞.
From the small m behaviour given in (A3) we finally have determined the configuration entropy as

S[m] = -I[m] + log(2) + d 2N v log(2π), = Φ[µ[m]] -m T µ[m] + log(2) + O 1 N v .
Note that practically speaking we use finite N v estimates of Φ and m α so that the preceding relation is in fact valid up to some O(1/ √ N v ) corrections w.r.t. limit defined by some hypothetical p u when N v → ∞.

Appendix B: Longitudinal and transverse free energy

In order to disentangle the contributions of the collective modes materialized by some magnetization m along some directions u α , α = 1, . . . d from the noise corresponding to the fluctuations of the transverse variables s ⊥ we assume first to be able to rewrite the Hamiltonian corresponding to the visible distribution (3)

P [s|Θ] = e -H[s|Θ] Z[Θ] ,
in terms of the new degrees of freedoms

H[s|Θ] = H[m, s ⊥ |Θ],
such that the joint distribution takes the form

P [m, s ⊥ |Θ] = e -H[m,s ⊥ |Θ] Z[Θ] .
This in turn can be written

P [m, s ⊥ |Θ] = P [s ⊥ |m, Θ]P [m|Θ], (B1)
with

P [m|Θ] = s P [s] d α=1 δ(s α -m α ) def = e -NvF [m|Θ] Z[Θ] ,
after introducing the canonical free energy

H[s|Θ] = Nv i=1 η i s i - N h j=1 log cosh Nv i=1 W ij s i -θ j = N v Nv α=1 η α s α - N h j=1 log cosh N v Nv α=1 w α s α v α j -θ j ,
this leads to the definition

H 0 [m|Θ] = N v d α=1 η α m α -V (m|Θ) , (B3)
with

V (m|Θ) def = 1 N v N h j=1 log cosh N v d α=1 w α m α v α j -θ j .
In terms of H 0 [m|Θ] the free energy reads

F[m|Θ] = 1 N v H 0 [m|Θ] -log s ⊥ e -H[s ⊥ |m,Θ] , = 1 N v H 0 [m|Θ] -S[m] -log s ⊥ e -H[s ⊥ |m,Θ] 2 N [m] = F [m|Θ] + F ⊥ [m|Θ]
where we have introduced respectively the longitudinal and transverse free energy:

F [m|Θ] def = 1 N v (H 0 [m|Θ] -S[m]), F ⊥ [m|Θ] def = - 1 N v log 1 2 N [m] s ⊥ e -H[s ⊥ |m,Θ] .
The longitudinal free energy of the system is the free energy of the system for a given magnetization m when the transverse magnetization and interactions among the s ⊥ are neglected. These indeed are expected to be small by definition of the intrinsic space. Vanishing interactions corresponds to H s ⊥ |m, Θ = 0, in which case F [m|Θ] coincides with F[m|Θ]. Non-vanishing interactions are accounted for by the transverse free energy.

Appendix C: Effective Hamiltonian

To enter further into the description of transverse fluctuations we need to specify the transverse degrees of freedom s ⊥ and the way they interact through H[s ⊥ |m, Θ] in the form of an effective Hamiltonian. For β > d the components s β given in (A1) are stochastic variables and we denote them by s β [s ⊥ |m], i.e. a mapping to be defined of transverse variables to transverse projections given some magnetization m. This mapping cannot be determined exactly but since we look for an effective theory we consider a linear map i.e. of the form

s β [s ⊥ |m] = m β [µ] + N [m] =1 c β s ⊥ , β = d + 1, . . . N v , (
C1) where the prior of these new variables is to be iid with s ⊥ ∼ U {-1,1} . Under the magnetization constraints only we have (see Section A)

E[s i |m] = m i [µ] Cov(s i , s j |m) = (1 -m 2 i [µ])δ ij + O 1 N v with m i [µ] = tanh N v d α=1 µ α u α i .
As a result m β [µ] (for β > d) represents the prior bias resulting from the magnetizations constraints (A4)

m β [µ] = 1 √ N v Nv i=1 u β i tanh N v d α=1 µ α u α i , ∀β > d (C2
) as a function of {µ α , α = 1, . . . d} solution to equation (A4) and are O(1/ √ N v ). The second set of constraints comes from the set of prior covariances between the s β [s ⊥ |m] for β > d that have to be maintained to properly account for the transverse degrees of freedom. These are diagonal at leading order:

Cov(s β [s ⊥ |m], s γ [s ⊥ |m]) = 1 N v Nv i=1 u β i 2 (1 -m 2 i [µ])δ βγ + O 1 N 2 v . ( C3
)
A simple way to maintain at best these covariances in the new representation is to associate a binary variable s ⊥ with the first N [m] principal axes of the previous covariance matrix neglecting the remainder. Since the covariance resulting from (C1) reads

Cov(s β [s ⊥ |m], s γ [s ⊥ |m]) = N [m] =1 c β c γ ,
the vector c can be chosen as a principal axes of the covariance matrix (C3) normalized to the standard deviation along the same axes which are O 1/ √ N v so that c β coefficients are O 1/N v . Now that the transverse variables are unambiguously defined we can obtained their effective Hamiltonian by expanding H[s ⊥ |m] defined in (B2) at second order in s ⊥ . From the definition (B2,B3) there is a zero order term

F ⊥ 0 [m|Θ] = O(1/ √ N v )
with convoluted expression which we give only the first terms, contributing to the transverse free energy and the first and second order terms corresponds to a conventional Hamiltonian of a disordered Ising model:

H[s ⊥ |m, Θ] ≈ H eff s ⊥ |m, Θ = N v F ⊥ 0 [m|Θ] + N [m] =1 η ⊥ [m, Θ]s ⊥ + N [m] , =1 W ⊥ [m, Θ]s ⊥ s ⊥ .
with

F ⊥ 0 [m|Θ] = Nv β=d+1 (η β + w β mβ )m β [µ] η ⊥ [m, Θ] = N v Nv β=d+1 c β (η β -w β mβ ) W ⊥ , [m, Θ] = -N v N h j=1 (1 -m2 j ) Nv β,γ=d+1 w β w γ c β c γ v β j v γ j ,
after introducing the notations:

mj = tanh N v d α=1 w α m α v α j + Nv β=d+1 w β m β [µ]v β j -θ j , mβ = 1 √ N v N h j=1 mj v β j . η ⊥ [m, Θ] is potentially O(1) while W ⊥ , [m, Θ] is O 1 √ Nv .
To make connection with data, i.e. given a configuration s with magnetization m α , α = 1, . . . d, from which are extracted the transverse magnetization {m β [µ], β = d + 1, . . . N v by solving the equations (A4), the s ⊥ are constructed as follows. First let for each = 1, . . . ,

N [m] m ⊥ [s] = Nv β=d+1 (s β -m β [µ])u ⊥ β (thresholded to 1 [resp.
-1] when bigger than 1 [resp. lower than -1]) the magnetization of the configuration s along this mode. This allows us to define the probability

p [s] = 1 + m ⊥ [s] 2 .
Then

s ⊥ = 2τ -1, ∀ = 1, . . . N [m]
with τ a Bernoulli variable of parameter p [s], gives us a set of spin variables fulfilling our needs.

Appendix D: Coulomb interaction picture

Let us consider the green function for the d-

dimensional Laplacian ∇ 2 d K d (|m -m |) =                1 2 |m -m |, (d = 1) 1 2π log |m -m |, (d = 2) - Γ d 2 -1 4π d/2 |m -m | d-2 , (d > 2)
which by definition is solution of

∇ 2 d K d (|m -m |) = δ(m -m ).
We can use it to rewrite V [m|Θ] up to an irrelevant constant term as

V [m|Θ] = d d m ρ(m |Θ)K d (|m -m |) (D1)
where ρ is the source term of the Poisson equation

∇ 2 d V [m|Θ] = ρ(m)
hence giving a density of Coulomb charges

ρ(m|Θ) = N h ,d j=1,α=1 w 2 α v α j 2 1 -tanh 2 N v d β=1 w β m β v β j -θ j .
To make sense of this quantity first remark that the function

δ ν (x) def = ν 2 1 -tanh 2 (νx) -→ ν→∞ δ(x)
represents a normalized 1-d narrow density of width ν -1 such that ρ can be expressed as

ρ(m|Θ) = 2 N v N h j=1 ν j δ νj n T j m -z j (D2)
with ν j and n j given by equation [START_REF] Hjelm | Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks[END_REF][START_REF] Hu | Latent source mining in fmri via restricted Boltzmann machine[END_REF]. In this form, ρ is readily a superposition of N h uniformly charged hyperplanes of finite width. Each hyperplane j being defined by a normal vector n j , an offset z j = θ j /ν j from the origin, a finite width ν -1 j and a (hyper)surface charge density 2ν j /N v . Furthermore this can be decomposed into more elementary charged hyperplanes of zero width. Equation (D1) now rewrites

V [m|Θ] = 2 N v N h j=1 ν j d d m δ νj n T j m -z j K d (|m-m |)
For each term j, writing m = (n T j m )n j + m ⊥ = zn j + m ⊥ , the transverse integration of m ⊥ yields

dm K d |m -m | = dzdm ⊥ K d » (n T j m -z) 2 + (m ⊥ -m ⊥ ) 2 = dz|n T m -z|.
up to an ill defined constant term after properly regularizing at large distances the integral over m ⊥ . As a result the one particle potential takes the form

V [m|Θ] = 2 N h N h j=1 ν j dzδ νj z -z j |n T j m -z|.
= dndz q(n, z)|n T m -z| with q(n, z) given by equation ( 10)

Appendix E: Exact Coulomb charges interpolation
In order to interpolate exactly the empirical distribution p with a generalized Coulomb charges RBM based distribution it is needed to regularize log(p(m)). This can be done in many different ways. Consider for instance

δ (m) def = exp -|m| 2 2 (2π ) d/2
with infinitesimal to approximate our point-like distribution as

p(m) = 1 M M k=1 δ (m -m k ),
and let

q (k|m) = δ (m -m k ) l δ (m -m l )
.

These probability weights realize a smooth partition of the space at finite with Voronoi cells R k centered at each data point, q (k|m) representing the probability that m belongs to kth cell. Equipped with this notation we have

∇ m δ (m -m k ) = - m -m k δ (m -m k ) ∇ m q (k|m) = - m -m k q (k|m) + M =1 m -m q (k|m)q ( |m).
As a result we get

∇ 2 d log p(m) = - 1 + 1 2 M k=1 Var k∼q (k|m) [m k ].
When becomes small compared to nearest neighbour distances this quantity becomes constant (= -1/ ) except on the intersections between Voronoi cells, in particular on common faces R k ∩ R between two cells R k and R it is

∇ 2 d log p(m) ∼ →0 - 1 + |m k -m | 2 δ(m ∈ R k ∩ R ).
onto the set of centered random variables (the score variables associated with q)

V j (m)

def = dm δ n T j m -z j K d (|m -m |) -E m∼ p(m) dm δ n T j m -z j K d (|m -m |) = |n T j m -z j | -E m∼ p(m) |n T j m -z j | .
E m∼ p(m) and Cov m∼ p(m) denote respectively empirical expectation and covariance, according to our assumption that the solution is close to p. Indeed, from elementary linear algebra, the orthogonal projection V of a given vector V , onto a subspace spanned by a set of independent vectors V k is given by

V = k,l G -1 ] kl (V l , V )V k (F3)
with G kl = (V k , V l ) the Gram matrix of the set of vector V k for some given inner product (•, •). Specified to our problem, the vectors are the densities ρ, or equivalently the random variables V (m|ρ) with inner product (F1,F2) resulting in

V (m|ρ), V (m|ρ ) = Cov m∼ p(m) V (m|ρ), V (m|ρ ) .
The projection of V [m] = V (m|ρ) is then given by V in (F3) with G the empirical covariance matrix of {V 1 , . . . V N h } and (V k , V ) the empirical covariance between V k and V (if the set V k is not independent the pseudo-inverse of G is taken instead of G -1 ). At this point this regression seems intractable since V involves a very complicated density of charge ρ. This is not the case because by construction we have

dm ρ(m )K d |m -m| = 1 N v log p(m)-S[m]+F ⊥ [m],
and log p(m) = log 1 M when evaluated on the data. This means that the solution to our projection problem is obtained by performing the previous linear regression with

V (m) = F ⊥ [m] -S[m] -E m∼ p(m) F ⊥ [m] -S[m] ,
so that the RBM distribution will be finally of the form

P RBM (m) = 1 Z e -NvF (m|Θ)
with

F(m|Θ) = F ⊥ [m] -S[m] - N h j=1 q j |n T j m -z j |, def = V (m) -V RBM (m),
i.e. a difference between 2 convex potential whenever F ⊥ [m] is convex or negligible. The interpolation point corresponding to the situation where there is a sufficient amount of Coulomb features to model exactly the empirical distribution is shown on Fig. 4. In this appealing picture there is however a shortcoming. The fact that we impose the features weights to be non-negative insures the regression curve to be convex but do not prevent it to pass above the fitted potential in empty regions of data. The reason for this, while the information theory argument provides us with a strong guarantee at first sight, is that the empirical Fisher metric is not relevant everywhere on the embedding functional space defined by the RBM features used to approximate V (m), but only on regions supported with data. Other directions are represented by random variables which are decorrelated from the data, so the Fisher metric is not covered by (F1) along these directions. This requires the linear regression to be complemented with some additional regularization which as shown on Fig. 4 should involve also the distance between the derivatives of the two profiles measured at the sample points. tion concerns only z j = -1 + 2j/N h , j = 1, . . . N h . The features j = 0 and j = N h correspond to the visible bias η. In the 2-d study case we have n j = cos(pπ/ √ N h ), sin(pπ/ √ N h ) and z j = -1 + 2r/ √ N h with the entire decomposition of j = p √ N h + r. The continuous dynamics of the parameters corresponding to an ordinary gradient reads q = γ∇ q L[q], with the learning rate γ. The scores associated with these parameters are centered random variables classically defined as q * (m) = ∇ q log p(m|q), which in our case read:

q * j (m) = |n T j m -z j | -E m∼p(m|q) |n T j m -z j | .
In terms of these variables the gradient of the log likelihood simply reads

∇ q L[q] = E m∼ p q * (m) ,
p being the empirical data distribution. In a continuous limit of the training process indexed by the training time t, let q * t the time dependent expectation of q t (m) w.r.t the empirical distribution p. We have q * j,t = dm p(m)p(m|q t ) |n T j mz j |.

The evolution of q * t with time is given by

d dt q * t = -γCov m∼p(m|q) q * t (m), q * T t (m) q * t ,
where Cov m∼p(m|q) denotes the covariance under p(m|q) and corresponds to the Fisher metric of the q parameter space. As we see as long as the covariance matrix is strictly positive definite the dynamics is contractant. When using the natural gradient [START_REF] Amari | Natural gradient works efficiently in learning[END_REF] defined here as ∇q = Cov m∼p(m|qt) q * t (m), q * T t (m) -1 ∇ q , the dynamics simplifies to

d dt q * t = -γ q * t .
The norm of q * t can be monitored during learning allowing for an adaptive strategy for the learning rate used in practice in these experiments to control the convergence as well as the stopping criterion. Thanks to the choice made for u α in these experiments the number of configurations m is equal to N +1 and (N +1) 2 /4 respectively in the 1 and 2-d case, so the LL of the resulting "Coulomb" RBM which are obtained can be evaluated exactly in both cases as well as the corresponding standard RBM. The latter is obtained through the following asymptotic mapping assuming that » i W 2 ij is large for all j: with

log cosh Nv i=1 W ij s i -θ j ≈ N v i W 2 ij s T √ N v n j -z j
n ij = W ij » i W 2 ij and z j = θ j » N v i W 2 ij
leading to

q j = 1 N v i W 2 ij .
This results into the following correspondence W ij = N v q j n ij , θ j = N v q j z j .

In both experiments the number of visible variables is N v = 10 3 , and the LL is an average value estimated on an independent test set of 10 3 samples. The number of points N d pts used to estimate the integrals over m needed to compute the natural gradient give a contribution N f N d pts to the complexity in a naive setting. The indicated values in the Table I correspond to the point where the results become insensitive to N d pts . The values LL Coulomb and LL RBM measured respectively for the "Coulomb" machine which is optimized with the natural gradient and its corresponding RBM using the previous mapping are reported on Table I are compared with the reference value LL Ref of the hidden mixture model used to generate the data. Note that the mapping gives a poor model when many weak features are used as in the d = 2 case with N h = 900. Note also that using the ordinary gradient instead of the natural one seems to keep the last bits of LL out of reach in a reasonable time.

Finally Fig. 6 illustrates the reasons for the failure of the standard RBM training. First the Gibbs sampling procedure is plagued by the presence of 1st order phase transitions which is well understood when considering the "Coulomb" RBM. Indeed, in that case changing the temperature corresponds to multiplying all the feature weights q j by a common factor β representing for instance an annealing inverse temperature. The learning procedure is supposed to tune precisely the difference F β [m|q] = βV (m|q) -S[m] at β = 1, in order to obtain many coexisting states corresponding to different values of condensed magnetization m. Then as in the example of Fig. 5, changing slightly β has the effect of concentrating the probability distribution on the state with highest or lowest value of V (m|q) depending on whether β is smaller or greater than one. The second source of failure is, as expected from the electrostatic picture, that the hidden bias given in rescaled form by z j = θ j /ν j along with [START_REF] Hjelm | Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks[END_REF] get trapped, around zero in the example of Fig. 6 which prevent the machine to form more than two ferromagnetic states in 1d.

FIG. 1 .

 1 FIG.1. Bipartite structure of the RBM (left). Hyperplanes defined by the weight vectors and bias associated with each hidden variable can delimit fixed density regions in input space (right).

  ) Given a condensed magnetization vector m, there remains N [m] interacting degrees of freedom represented by spin variables denoted {s ⊥ 1 , . . . , s ⊥ N [m] }. With help of this new set of visible variables the partition function takes the form of a d-dimensional integral

FIG. 2

 2 FIG. 2. 1-d intrinsic data (Nv = 10 3 ) with 5 clusters solved with N h = 20 predefined features thanks to a natural gradient ascent of the LL. Dotted lines indicate location of features with nonvanishing weights qj. The feature contributions F(m) -h(m) to the free energy are seen to regress h(m) on the data. The resulting distribution is shown (red) on the inset with the empirical training distribution (blue) and the failed result of a standard RBM training (green).

8 FIG. 3

 83 FIG. 3. 2-d intrinsic dataset (Nv = 10 3 ) with 6 point-like clusters and a circular one (upper left) and corresponding RBM density (upper right) found with N h = 900 predefined features, along with its free energy landscape (bottom left) and Coulomb charges distribution (bottom right)

  1] and ω j ∈ [0, π[ are the angles made by the charged lines with the m 2 axis. The result of the natural gradient ascent of the LL is shown on Fig.3. Here a large number of features (ω j , z j ) ∈ [0, π]×[-1, 1] have been predefined on a regular lattice in order to obtain a continuous charge distribution and a smooth free energy landscape (see more details in SM, Appendix G). Finally, in both study cases the standard RBM training fails for two distinct reasons unveiled by the Coulomb picture (see SM, Appendix G): (i) the Gibbs sampling is plagued by the presence of first-order phase transitions with respect to an annealing temperature; (ii) the charged hyperplanes get easily trapped by Coulomb barriers formed by the clusters of data , a pitfall bypassed by the convex "Coulomb" relaxation.

2

 2 A1) coincides with m α for α = 1, . . . d by definition of the magnetization constraints. We look for a change of variables s -→ (m, s ⊥ ) where the original spin variables are replaced by the set of m = {m α , α = 1, . . . d} and N [m] transverse spin variables. Let us denote by E s∼U the expectation taken when the original spin variables are iid, s i ∼ U {-1,1} . The change of measure is made by looking at the prior distribution over the original spin variables: P prior [s] = 1 Nv = P prior [s ⊥ |m]P prior [m], where P prior [m] = E s∼U d α=1 δ(s αm α ) = e Nv(S[m]-log 2)

FIG. 4 .

 4 FIG. 4. Picture of the VRBM(m) (in red) at the interpolation threshold in 1-d.

FIG. 5 .

 5 FIG.5. First order phase transition mechanism of the "Coulomb" RBM illustrated on the 1-d case. β represents an annealing inverse temperature, which effect is to multiply the energy part V (m) in the free energy F β (m) = βV (m) -h(m). β = 1 is a reference RBM where many states are present with comparable probabilistic weights; for β > 1 [resp. β < 1] the state with highest [resp. lowest] V (m) is favored.

FIG. 6 .

 6 FIG.6. Illustration of the failure of the standard RBM training due the occurrence of a first order transition signaled here by a divergence between the Gibbs sampling evaluation of the magnetization along the intrinsic axis with the exact one (main plot), and due to the trapping of the zj's close to zero (inset).

  √ N v residual transverse magnetizations m β for β > d resulting from the constraints (see next Section). Rewriting the Hamiltonian in terms of the SVD components s α and η α respectively of the visible variables and biases

F[m|Θ]. Let us denote by H[s ⊥ |m, Θ] def = H[m, s ⊥ |Θ] -H 0 [m|Θ], (B2) the "conditional" Hamiltonian, where H 0 [m|Θ] is at this point an arbitrary function of m independent of s ⊥ . We chose it as to contain only contributions from the longitudinal magnetization, i.e. coincides with the constant part of H[m, s ⊥ |Θ] w.r.t. s ⊥ when neglecting the singular values w β of W and the O 1/

TABLE I .

 I LL obtained for the RBM in the study cases

		LL Ref	LL Coulomb	LLRBM	N features	N epochs	N d pts	γ
	d = 1	-471.45	-479.85	-479.82	5	5000	100	0.0001
		-471.45	-471.66	-471.69	10	7000	100	0.0001
		-471.45	-471.48	-471.48	20	10000	100	0.001
		-471.45	-	-535.00	20	2500	-	0.001
	d = 2	-621.90	-623.74	-623.61	49	113000	900	0.0001
		-621.90	-622.16	-622.24	169	155000	900	0.0001
		-621.90	-621.97	-631.79	900	410000	1600	0.0001

Acknowledgments A.D. was supported by the Comunidad de Madrid and the Complutense University of Madrid (Spain) through the Atracción de Talento program (Ref. 2019-T1/TIC-13298).

Indeed, let

For δm = mθ k small compared to ∆ k we have

Since ν 2 1tanh 2 (νx) tends to δ(x) when ν → ∞ we arrive at the statement. As a result the distribution of charges is composed of a constant background + surface distribution on Voronoi cells intersections:

The Voronoi cells intersecting with the boundary of the m domain induce additional surface charges which can be directly taken care of with visible bias. Let us show how this works in 1-d. Let us call

which when regularized reads

From what precedes, this potential can be exactly decomposed onto a set of features as

while from the limit behaviour V (1) and V (-1) of V (m) we get

Appendix F: RBM optimization seen as a linear regression

The projection of the empirical distribution onto the space of RBM with finite number of features is classically done by minimizing the Kullback Liebler divergence (D KL ). If however our RBM space is chosen with a high number of relevant features, we may expect the solution to be close enough to the empirical distribution so that a Fisher metric, i.e. the infinitesimal counterpart of the D KL , evaluated from the solution or from the empirical distribution should coincide. In that case it might be pertinent to use it instead of the D KL . Let us formalize more precisely this projection problem. On one hand we have the empirical measure approximated by a Coulomb based RBM model of the form

with

where ρ(m) is, as seen in the previous Section, the charge density concentrated on the Voronoi cells faces coming from the empirical part log p(m) (including surface terms at the edge of the domain of m). On the other hand we have an RBM with a pointwise distribution of features q(n, z) yielding a free energy of the form

Our goal is to find the (positive) weights {q j , j = 1, . . . N h } such that the following distance

between ρ and ρ is minimized. Here the relevant metric J is the Fisher metric defined as

approximated at the empirical point in last equation. As we shall see this projection turns out to be a linear regression of the centered random variable

The transverse free energy being absent and the entropy term independent of Θ ≡ q, the loss optimized in the experiments is given by

e -NvF [m|q] with

In the 1-d study case, we have n j = 1 and the discretiza-


Catherine Jami

Keywords:

In recent decades research in the social sciences, including in the history of science, has shown that women scientists continue to be depicted as exceptions to the rule that a normal scientist is a man. The underlying message is that being an outstanding scientist is incompatible with being an ordinary woman. From women scientists' reported experiences, we learn that family responsibilities as well as sexism in their working environment are two major hindrances to their careers. This experience is now backed by statistical analysis, so that what used to be regarded as an individual problem for each woman of science can now be identified as a multi-layered social phenomenon, to be analysed and remedied as such. Over the last five years, international scientific unions have come together to address these issues, first through the Gender Gap in Science Project, and recently through the setting up of a Standing Committee for Gender Equality in Science (SCGES) whose task is to foster measures to reduce the barriers that women scientists have to surmount in their working lives.

A Measure of the Gender Gap in Science

In 2017, I joined the Executive Committee of the Gender Gap in Science Project. This was an international project led by the International Union of Pure and Applied Chemistry (IUPAC) and by the International Mathematical Union (IMU), and supported by the International Science Council (ISC), within which I represented the international Union of History and Philosophy of Science and Technology (IUHPST) [1]. My enthusiasm for the project stemmed from several motivations. First, there is a significant and unaddressed gender gap in the discipline to which I belong, and which I represent in the Gender Gap in Science Project. Secondly, historical studies on women and gender in science shed interesting light on why the gender gap in science persists and how historical and scientific discourse and practice may contribute to perpetuating it. Last but not least, the Gender Gap in Science Project is an important "first", in that it was set up and conducted by women and men of science rather than by science policy makers, in other words, by actors rather than observers.

A survey of scientists, covering all regions and development levels, and several disciplines, was carried out as part of the project. It shows that women's experiences in science are consistently less positive than men's, in both educational and employment settings [2, p. 19]. For example, more than 25% of women scientists who answered the survey reported having encountered sexual harassment during their studies or at work (compared to less than 4% of men). Moreover, women are 3.3 times more likely to report slower rate of promotion after having children than men [2, pp. 35, 48-52]. These quantitative data are crucial, because they shed a different light on women's individual experiences in the scientific profession: these experiences are no longer the personal problem of each of them, to be surmounted without "unnecessary fuss", but a feature of the profession itself, that needs to be dealt with by the scientific communities at various levels.

These and other figures in the results of the survey confirm the fact that the gender gap in science will not simply be reduced as the result of a spontaneous evolution. Progress towards equality in every part of our societies does not just happen "naturally." In order to devise and implement strategies to actively promote gender equality in science, we need to better understand the processes that have resulted in the present situation, and that contribute to the perpetuation of the gap. In what follows, I share some reflections based on social studies of science, mainly but not exclusively historical. I discussed some of these ideas with participants in the final conference on the Gender Gap in Science Project [START_REF]A Global Approach to the Gender Gap in Mathematics[END_REF], from whom I received very valuable feedback. Discussions during this conference also brought out the need to create permanent links between concerned scientists in order to work towards gender equality in science. I will conclude this article by setting out how we have undertaken to do that.

Picturing Scientists: Children's Drawings and Nobel Prizes

Since the 1950s, specialists in science education have conducted experiments with students in Anglophone countries, asking them to "draw a scientist". Unlike in Romance languages, in English this sentence does not point to a gender, as the indefinite article "a" is neuter. Nevertheless, an overwhelming majority of both boys and girls represent the scientist as male. The figure of the male chemist in his laboratory seems to have long predominated in their drawings [START_REF] Chambers | Stereotypic Images of the Scientist: The Draw a Scientist Test[END_REF][START_REF] Finson | Drawing a Scientist: What We Do and Do Not Know After Fifty Years of Drawings[END_REF]. Although the proportion of women among scientists has increased since the experiment was first conducted, a study published in 2018 reported that a majority of children continued to depict a scientist as male; moreover, the older the children are, the more likely they are to draw male scientists [START_REF] Miller | The Development of Children's Gender-Science Stereotypes: A Meta-analysis of 5 Decades of U.S. Draw-A-Scientist Studies[END_REF]. The title of a Washington Post article that discussed this study tersely summed up the situation: "Only 3 in 10 children asked to draw a scientist drew a woman. But that's more than ever." [7] As the authors of the 2018 article remark, "women remain under-represented in several science fields, and information about such imbalances is filtered through multiple sources such as mass media and social interactions". Children are simply representing the world they see. Indeed, according to UNESCO statistics published the following year, about 30% of scientists around the world (excluding Canada and the USA, for which the data appear to have been unavailable) are women, with significant variations according to countries [8]. But presence and visibility are not the same thing. Today, the scientists who appear in the media, those who are awarded prizes, and those who give expert advice are still overwhelmingly male. Again, this remains true even though one sees many more women as award recipients than a few decades ago.

Let us take the case of chemistry, the discipline that children in these experiments often take to represent science. Since the Nobel Prizes were created in 1901, the prize in chemistry has been awarded to 178 men, and to seven women. They are, in chronological order: Marie Curie (1911), Irène Joliot-Curie (1935), Dorothy Crowfoot Hodgkin (1964), Ada E. Yonath (2009), Frances H. Arnold (2018), Emmanuelle Charpentier (2020) and Jennifer A. Doudna (2020). If one looks back only to the last decade, 23 laureates are men and three are women. As in children's drawings, there is no denying that the proportion of women is increasing here too. There is no denying either that the gap remains wide. Chemistry, of course, is not an exception. Since 1901, four women and 211 men have been awarded the prize in physics; twelve women and 110 men have been awarded the prize in physiology and medicine. The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel, founded in 1969, has been awarded to two women, and to 84 men [9]. In mathematics, the Fields Medal, first awarded in 1936, has been awarded to 59 men and only one woman. The gender imbalance is overwhelming across disciplines, whether in the social sciences or in mathematics and natural sciences. This discrepancy between visibility and reality is both a consequence of the history of women's participation in science, and a hindrance to the reduction of the gender gap in science. The visibility of women scientists, perhaps even more than facts and figures about them, is crucial to representations of science and of scientists among the public. In what follows, I would like to reflect on these representations in science in a historical perspective.

Saying the Names of Women Scientists

Continuing to focus on chemistry, let us look back further than the twentieth century, at the received narrative of the history of this discipline, exemplified by the Wikipedia article on the subject. It is a long genealogy of men of science, most of them European. The illustrations in the article include 23 portraits of chemists; among them one finds the photograph of one woman, Marie Curie (who is doomed to represent "the woman scientist"), and a "Portrait of Monsieur de Lavoisier and his Wife" (1788) [10]. The wife in question, Marie-Anne Paulze Lavoisier (1758-1836), has only recently begun to be studied as a chemist in her own right [START_REF] Bret | Madame Lavoisier's Diffusion and Defense of Oxygen Against Phlogiston: Her Translations of Richard Kirwan's Essays[END_REF]. The Wikipedia article devotes a paragraph to her: we are told that she made a contribution, not directly to chemistry, but instead to her husband's refutation of the then prevailing theory of phlogiston, according to which combustible bodies contained an element (phlogiston) that was released during combustion. The other twenty-one portraits in the Wikipedia article represent men; one collective photograph of the participants of the first Solvay conference (1911) shows Marie Curie and 23 men. The only representative of Antiquity depicted is Democritus (c. 460 -c. 370 BCE), the atomist philosopher. And yet, one of the earliest figures in the alchemical tradition of expert practice was a woman, referred to in English as Mary the Jewess, who lived in Alexandria, probably in the 1st century CE. She is credited with the invention of the alembic and of a device mentioned in most Western cookbooks: the eponymous bain-marie, though for her it functioned as a source of gentle heating for alchemical processes rather than as a culinary technique [START_REF] Alic | Women and technology in ancient Alexandria: Maria and Hypatia[END_REF]. These are three splendid archetypes: Marie Curie is the exception that confirms the rule; Marie-Anne Paulze Lavoisier is somebody's wife; Mary the Jewess has been immortalised in the most feminine of domestic spaces, the kitchen.

The preceding paragraph should not be read as a criticism of Wikipedia. As academics well know, Wikipedia reflects received knowledge, and is what its users make it: since 2012 it has a working group dedicated to "ensuring quality and coverage of biographies of women scientists," the WikiProject Women Scientists [13]. All this goes toward showing that it is not only the "facts", but also the representations that prevail, that need to be worked on in order to understand the gender gap in science as in societies past and present. Such investigations are essential in attempts to reduce this gap. And one thing they have already shown is that the "Three Marys" mentioned above are not the only women in the history of chemistry, and that there are so many of them that it would be impossible to list them here -but one needs to look for them, in many cases elsewhere than in the very masculine memory of science. It is this memory, rather than history, that is engraved in the list of Nobel Prize winners and of members of academies of science. Granting visibility to women past and present is therefore a major goal to attain [START_REF] Fogg-Rogers | Catch 22 -improving visibility of women in science and engineering for both recruitment and retention[END_REF].

Both academic historians and popular authors have written book-length biographical accounts of women scientists. This is indeed a means of granting them visibility, and of celebrating them -to take up the motto put forward by UNESCO in its yearly International Day of Women and Girls in Science ("Celebrating women in science", 11 February). Here again one of the earliest biographies one comes across is that of Marie Curie by her daughter Eve Curie, published four years after her mother's death [START_REF] Curie | Madame Curie[END_REF]. Reading a couple of sentences from it suffices to realise that this was no attempt at a "historically objective" account of Marie Curie's life and work, but rather a heroic one: "She was a woman, she belonged to an oppressed nation, she was poor, she was beautiful. A powerful vocation prompted her to leave her homeland, Poland, to come and study in Paris […]." But, in science as elsewhere in our societies, the experiences of women matter, rather than their looks or their patriotic feelings, Their successes and achievements -not just the difficulties or discrimination they sufferare worth highlighting. The celebration mode, however, has its limits and risks when it focuses on one or a few individuals. As Patricia Fara remarked in her review of biographies of women scientists, many biographies use the heroic mode as a selling point, and by doing so perpetuate the myth that "it is impossible for someone to be both a normal woman and a firstclass scientist." [START_REF] Fara | Women in science: weird sisters?[END_REF] 4. The Gaze on Women Scientists Although women scientists write their autobiographies less often than men scientists [START_REF] Abir-Am | Nobelesse Oblige: Lives of Molecular Biologists[END_REF], these autobiographies may provide a necessary counterbalance to heroic narratives. Yvonne Choquet-Bruhat, the first woman elected as a full member of the French Academy of Sciences in 1979, is best known for providing the earliest mathematical proof of the existence of solutions to Einstein's equations. Her autobiography, written at the request of her children, describes her work, the travels that it entailed, and her private life. She recounts that during the year she spent at the Institute for Advanced Study, Princeton, in 1951-1952, she discussed her work with Einstein himself. Before that, she evokes her "everyday life" at the Institute for Advanced Study with her husband Léonce Fourès, also a mathematician, and their one-year old daughter Michelle: "...We hired a 15 year old girl to come after her school hours and look after Michelle. Michelle and the baby-sitter were very happy with this. When the baby-sitter was away, Léonce sometimes replaced me. Fortunately, one can do mathematics anywhere, so that the presence of our daughter did not hinder my work. I had, by the way, noticed when taking tea in the common room that some colleagues were sitting there, absorbed in playing that game of Chinese origin called go. Once, as I was leaving the common room to take over when the baby-sitter had to go home, one of them told me: 'Obviously, a woman can't be a mathematician, she has to look after her children.' I retorted: 'She can't stay late to play go.'" [18, p. 99] Yvonne Choquet-Bruhat's concerns about childcare arrangements will seem familiar to many readers of the present article, who like her combine a scientific career and family life. Alas, so will the comment made by her male colleague (to Choquet-Bruhat, it goes without saying that the colleagues she mentions were men; in the original French text the gender of the speakers is specified by a pronoun). Together, they encapsulate the everyday experience of women scientists. The anecdote she recounts is now backed by the conclusion of the worldwide survey conducted in 2018, that is, more than sixty years after her stay at the Institute for Advanced Studies. It should be emphasised, however, that this experience is not specific to the scientific profession: both the responsibility for childcare falling to mothers and sexist comments that question the legitimacy of women's status as professionals are common everywhere in our societies. As an institution, science in not immune to phenomena that pervade the societies in which it is established.

Today, 27% of scientists in France are women, and more than fifty out of the 286 members of the French Académie des Sciences are now women [19]. However, the cliché that being a woman and being a good scientist are incompatible proclaimed by Yvonne Choquet-Bruhat's go-playing colleague remains vivid all around the world. Thus in 2014, when the Fields Medal was awarded to a woman for the first time, some comments on the social media targeted the laureate, Maryam Mirzakhani (1977-2017), for her allegedly masculine looks. This is nothing more than the continuation of a long and disheartening tradition. Women who have come down in history as great mathematicians, such as Émilie du Châtelet (1706-1749), Sofia Kovalevskaya (1850-1891) and Emmy Noether (1882-1935), have been subjected to similar comments -coming from women as well as men-that denied the compatibility between being female and being a mathematician of some standing [START_REF] Boucard | Women in mathematics: historical and modern perspectives[END_REF].

The Discourse of Science on Women

But the legitimacy of women in scientific professions has been put in question by more than unpleasant comments from male colleagues. Historians of science usually date the establishment of science as a modern institution to the nineteenth century, with the setting up of universities and the definition of academic disciplines taught there [START_REF]Modern Science in National, Transnational, and Global Context[END_REF]. At the time, it went without saying was that a scientist was a male scientist. Since then, male scientists have not only been at best slow to admit women among their ranks; they have also used science to argue that women must (for their own good) be kept out of science. I will mention a single example: Edward H. Clarke (1820-1877), who was a Professor of Materia Medica at the Harvard Medical School from 1855 to 1872, by no means a marginal figure. In 1872, in the context of increasing pressure in favour of higher education for women, he published a book entitled Sex in Education, or A Fair Chance for the Girls, which was successful enough to undergo seventeen editions in twenty years. Contrary to what its title suggests to twenty-first century readers, the book's main argument is that the intellectual demands placed on boys at university are too hard for girls, in whom they trigger "physiological disasters," leading to "nervous collapse and sterility." Clarke describes seven medical cases in support of his argument. The description of these cases might make one smile today, if it were not for the fate of his patients, some of whom ended up being consigned by him to an asylum [START_REF] Zschoche | Dr. Clarke Revisited: Science, True Womanhood, and Female Collegiate Education[END_REF]. The point I want to make here is that the discourse produced by Clarke, on whom the institutions of his day bestowed weighty scientific authority, barred girls and women not only from studying and practicing science, but more broadly from the intellectual training provided in higher education that had been devised for young men.

Today one would swiftly characterize Clarke's analysis of his cases and treatment of his patients as an example of harmful exercise of power by a prejudiced medical authority. Still, a study conducted less than ten years ago on anatomy textbooks used at university concluded that these textbooks were not gender-neutral, and that while the students who use them "recognise the importance of gender issues and do not wish to associate with sexism, most are unaware of the possible negative aspects of sexism within anatomy" [START_REF] Morgan | Sexism and anatomy, as discerned in textbooks and as perceived by medical students at Cardiff University and University of Paris Descartes[END_REF]. Sexism pervading scientific discourse and practice is not a thing of the past. Over the last three decade, an abundant literature has shown how it continues to affect medical treatment of women, and therefore their health [START_REF] Hamberg | Gender bias in medicine[END_REF]. It also reduces their ability to share the social space equally with men in all quarters.

6. Women Scientists, Across Disciplines, Around the World In sum, scientific progress does not of and by itself purge out gender biases either from the behaviour of scientists or from the knowledge that they produce and teach. Instead, there is a continued need to address these biases as social phenomena present in science as elsewhere in our societies. And despite the historical continuities in ideas and behaviours that have hindered the access of girls and women to science, it is here that one can find some ground for optimism. Acknowledging that science, like all human activity, is produced in social contexts that contribute to shaping it opens possibilities for bringing to light the mechanisms that underlie the gender gap that pervades scientific studies and professions and thereby for remedying it. Women scientists, who a century ago were few enough to be singled out as heroic "exceptions that prove the rule" of women's incapacity in that field, are now too many for their presence in science to be accounted for individually in the heroic mode: one is dealing with figures and statistics. As well as "say their names," we now can now "count their numbers." This is what the worldwide survey undertaken as part of the Gender Gap in Science Project has done. This project was carried out jointly by a number of international scientific unions, joined by a few other international organisations. For the first time, scientists themselves have taken the matter into their hands in a cross-disciplinary way. The results of this survey give a clear measure of the gender gap. These results vary from discipline to discipline, but the overall situation is the same for all of them. This being said, the quantitative approach, valuable as it is, does not replace the sharing of individual experiences, and does not make qualitative approaches to the gender gap in science obsolete. This sharing remains as necessary as ever, first and foremost to account for the diversity of ways in which the gender gap manifests itself in various parts of the world. Thus, at the closing conference of the project [25], one of the speakers, Shobhana Narasimhan, recounted that, at the Career Development Workshop for Women in Science that took place in Trieste in 2013, one of the participants said to her: "This is the first conference of any kind that I have attended… this is the first conference my husband allowed me to attend. He allowed me to attend this one because I told him that only women would be there." [26, slide 13] To most of those coming from Europe and the Americas who heard Shobhana Narasimhan's report, the fact that a scientist might need permission from her husband to attend a conference was new, even though we were all aware, in principle, of the diversity of situations of women scientists around the world; this kind of situation is not, obviously, specific to a particular discipline. Exchanges of this nature highlight the crucial need to develop worldwide and cross-disciplinary networks of support for women scientists, among other things in order to exchange experience and adjust best practices to local conditions. The awareness of this diversity was one of the motivations for the scientific unions who took part in the Gender Gap in Science Project in the decision to set up a permanent structure in order to continue to work together at an international level and across disciplines to monitor the gender gap in the profession and to foster measures in favour of gender equality. This structure, the Standing Committee for Gender Equality in Science (SCGES), started working with its nine founding partners in September 2020. As of January 2021, it has 15 partners, six international unions having joined in, and more of them are expected to join [27].

One especially noteworthy feature is that three of the partners, the International Union of History and Philosophy of Science and Technology (IUHPST), the International Geographical Union (IGU) and the International Union of Psychological Science (IUPsyS) represent disciplines that lie at the intersection of the natural and the social sciences. This reflects the fact that the natural and the social sciences form one single institution. Thus SCGES aims at bringing together all the disciplines represented among the members of the International Science Council (ISC). This will be beneficial in two ways: firstly, because the social sciences have developed research on gender in science (and more generally at large in our societies), on which the present contribution relies, and secondly because the gender gap exists in all academic disciplines. All the sciences, be they natural, mathematical or social, stand to gain by coordinating their efforts to work towards gender equality.

The first action of SCGES, undertaken at the request of one of its partners, has been to issue a statement "Standing for gender equality in times of Covid-19" on the consequences of the coronavirus pandemic on women scientists' working conditions and career prospects. This statement aims to specify the actions that can be taken by scientific institutions, funding agencies, recruitment committees, journals, associations and conference organisers. Individuals and institutions engaged in science can and should join forces in supporting women colleagues whose research careers are jeopardized by gender-specific pressures caused by the pandemic [28]. The various forms of support recommended in this statement can and should be perpetuated beyond the present crisis.