Imen Ayadi

Gabriel Turinici Ceremade

Stochastic Runge-Kutta methods and adaptive SGD-G2 stochastic gradient descent

Keywords: Machine Learning, ICML, SGD, stochastic gradient descent, adaptive stochastic gradient, deep learning optimization, neural networks optimization

The minimization of the loss function is of paramount importance in deep neural networks. On the other hand, many popular optimization algorithms have been shown to correspond to some evolution equation of gradient flow type. Inspired by the numerical schemes used for general evolution equations we introduce a second order stochastic Runge Kutta method and show that it yields a consistent procedure for the minimization of the loss function. In addition it can be coupled, in an adaptive framework, with a Stochastic Gradient Descent (SGD) to adjust automatically the learning rate of the SGD, without the need of any additional information on the Hessian of the loss functional. The adaptive SGD, called SGD-G2, is successfully tested on standard datasets.

Introduction and related literature

Optimization algorithms are at the heart of neural network design in deep learning. One of the most studied procedures is the fixed step Stochastic Gradient Descent (SGD) [START_REF] Bottou | Numerical results for the SGD-G2 algorithm and the ADAM algorithm on the CIFAR10 database[END_REF]; although very robust, SGD may converge too slow for small learning rates or become unstable if the learning rate is too large. Each problem having its own optimal learning rate, there is no general recipe to adapt it automatically; to address this issue, several approaches have been put forward among which the use of momentum [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF], Adam [4], RMSprop [START_REF] Tieleman | Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] and so on. On the other hand, recent research efforts have been directed towards finding, heuristically or theoretically, the best learning rate [START_REF] Seong | Towards Flatter Loss Surface via Nonmonotonic Learning Rate Scheduling[END_REF][START_REF] Leslie | Super-convergence: Very fast training of neural networks using large learning rates[END_REF][START_REF] Leslie | Cyclical learning rates for training neural networks[END_REF] with [START_REF] Yedida | Lipschitzlr: Using theoretically computed adaptive learning rates for fast convergence[END_REF], which uses an estimate of the Lipschitz constant, being a very recent example.

Another interpretation of the minimization procedure it to see it as a time evolution (a flow) in the space R d of neural network parameters. Denote such a parameter by X and let X ∈ R d → f (X) ∈ R be the the loss functional; the flow interpretations recognizes that the minimization of f (X) is related to the solution of the following evolution equation

X (t) = ∇f (X(t)). (1)
In this work we consider the flow (1) and apply two numerical schemes to evolve it in time: the Explicit Euler scheme (which will correspond to the SGD algorithm) and a numerical scheme of second order in time, labeled "SH" (like in "stochastic Heun") belonging to the class of stochastic Runge-Kutta methods; this second scheme allows to have a more precise estimation of the flow and in turn provides essential information to adapt the learning rate of the SGD.

We prove theoretically in section 3 that the SH scheme is indeed of second order; then we explain how it allows to choose the optimal learning rate for the SGD and build the SGD-G2 algorithm. Numerical results on standard datasets (MNIST, F-MNIST, CIFAR10) are presented in section 4 followed by a discussion and concluding remarks.

Notations

The fit of the neural networks is formalized through the introduction of a loss functional f depending on the network parameters X and the input data ω ∈ Ω presented to it. In full generality the input data belongs to some probability space (Ω, P) and the optimization aims to find the value X opt minimizing the mapping X → E ω f (ω, X). However, for instance for classification purposes, not all ω have a label attached to it, so in practice only a limited amount of values ω 1 , ..., ω N ∈ Ω can be used. So the loss functional becomes:

f (X) := 1 N N i=1 f (ω i , X). (2)
For i ≤ N we introduce the functions f i = f (ω i , •) : R d → R to represent the loss due to the i th training sample ω i .

To minimize the loss functional one can think of an deterministic procedure (of gradient descent type) which can be written as:

X n+1 = X n -h∇f (X n ), (3)
where h > 0 denotes the learning rate (also called "step size"). Note that this update rule requires N gradient evaluations per step which is prohibitively large in applications in deep learning that involve networks with many parameters (tenths of thousands up to billions). To this end, the deterministic procedure is replaced by its stochastic counterpart, the Stochastic Gradient Descent (SGD). Let (γ n ) n≥1 be i.i.d uniform variables taking values in {1, 2, .., N }. Then, the SGD is defined as:

X n+1 = X n -h∇f γn (X n ), X 0 = X(0). (4)
The advantage of the stochastic algorithm is that the gradient is evaluated once per iteration which makes its computational complexity independent of N . This explains why this method is preferred for large data sets. 1

2.1 The construction of the SGD-G2 algorithm: the principle

The state X n+1 in (4) can be also seen as an approximation of the solution X(t) of the flow in (1) at the "time" t n+1 = (n + 1)h: X n+1 X(t n+1 ). But there are many ways to obtain approximations of X(t n+1 ), for instance one can use a second order in h scheme (such as the so-called Runge-Kutta schemes to name but a few [START_REF] Press | Numerical Recipes 3rd Edition: The Art of Scientific Computing[END_REF]) and construct another approximation Y n+1 at the price of computing another gradient. If Y n+1 is a better approximation then it closer to X(t n+1 ) and thus at the leading order Y n+1 -X n+1 is an estimation of the error X n+1 -X(t n+1 ). With such an approximation one can extrapolate the behavior of f near X n and compute for what values of the learning rate h we still have stability (the precise computations are detailed in the next section). We adapt then the learning rate to go towards the optimal value, that is, large enough to advance fast but still stable. This will be encoded in the SGD-G2 algorithm we propose. Two questions arise:

-how to design a high order scheme consistent with the equation (1): this is the object of section 3; -is the numerical procedure performing well in practice : this is the object of section 4.

Theoretical results

Choice of the stochastic Runge-Kutta scheme

First we need to choose a numerical scheme that solves the equation (1) by using only partial information on the samples, i.e., we have to use some stochastic numerical scheme. Among the possible variants we choose the stochastic-Heun method described below (see also [START_REF] Ruthotto | Deep Neural Networks Motivated by Partial Differential Equations[END_REF][START_REF] Zhu | Convolutional Neural Networks combined with Runge-Kutta Methods[END_REF] for related works, although not with the same goal); while the SGD updates the state by relation (4) the stochastic Heun scheme (named "SH" from now on) reads:

Y 0 = X(0) Ỹn+1 = Y n -h∇f γn (Y n ) Y n+1 = Y n - h 2 ∇f γn (Y n ) + ∇f γn ( Ỹn+1 ) ( 5
)
1 In practice γn are drawn without replacement from {1, ..., N } until all values are seen. This is called an epoch. Then all values re-enter the choice set and a new epoch begins. We will not take discuss this refinement in our procedures which is independent of the segmentation in epochs or not. Same for mini-batch processing which consists in drawing several γ k at once; the method proposed in the sequel adapts out-of-the-box to such a situation too.

Note that a step requires two evaluations of the gradient, but this is the price to pay for higher precision. Note also that the same random sample γ n is used for both gradient computations. In fact SH this can be also seen as a SGD that uses a sample twice to advance and then do a linear combination of the gradients thus obtained.

In order to prove relevant properties of SH scheme, we need to make clear some details concerning the evolution equation ( 1). In fact, since only one sample is used at the time, the evolution X n will depend on the order in which samples ω γn are chosen. This means that in fact there is some randomness involved and we cannot hope to approach exactly the solution X(t) of (1). In fact, see [5,[START_REF] Li | Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations[END_REF], it is known that the output of, let's say, the SGD algorithm is close (in mathematical terms "weakly converging") when h → 0 to the solution of the following stochastic differential equation:

dZ t = b(Z t )dt + σ(Z t )dW t , Z(0) = X(0), (6)
with b(z) = -∇f (z) and σ(z) = (hV(z)) 1/2 , where

V(z) = N k=1 (∇f γ k (z) -∇f (z))(∇f γ k (z) -∇f (z)) T N , (7)
is the covariance matrix of ∇ X f (ω, z) taken as a random variable of the samples ω (see also [5] equation ( 4)). Here W t is a standard Brownian motion. With these provisions we can formally state the following result:

Theorem 1 (Convergence of SGD and SH schemes). Suppose f , f k are Lipschitz functions having at most linear increase for |X| → ∞ 2 . The SGD scheme converges at (weak) order 1 (in h) to the solution Z t of (6) while the SH scheme (5) at (weak) order 2.

Proof: Te recall what weak convergence means we need to introduce some notations: we designate by G the set of function having at most polynomial growth at infinity and W 1,∞ the set of Lipschitz functions. Given a numerical scheme U n of step h (SGD, SH, etc.) that approximates the solution Z t of the SDE (6) with U n Z nh , weak convergence at order p means that, for any t (kept fixed) and any function G ∈ G we have3 :

|EG(U t/n ) -EG(Z t )| = O(h p ). ( 8
)
It is known from [5] (Theorem 1 point "i") that SGD is of weak order 1. It remains to prove that SH is of weak order 2; the proof uses Theorem 2 from the same reference (see also [START_REF] Milshtein | Weak Approximation of Solutions of Systems of Stochastic Differential Equations[END_REF]) that is recalled below: Theorem 2 (Milstein,1986). Suppose ∀i ≥ 1 : ∇f, ∇f i ∈ G ∩ W 1,∞ , and have at most linear growth at infinity. Suppose that

Z 0 = U 0 = z ∈ R d . Let ∆ = (∆ 1 , ..., ∆ d ) = Z h -z and ∆ = (∆ 1 , ..., ∆ d ) = U 1 -z. If in addition, there exist K 1 , K 2 ∈ G
such that for any s ∈ {1, 2, ..., 2p + 1} and any z:

E( s j=1 ∆ ij ) -E( s j=1 ∆ ij ) ≤ K 1 (x) h p+1 , (9)
and

E( 2p+1 j=1 |∆ ij |) ≤ K 2 (x) h p+1 , (10)
the numerical scheme U n converges at weak order p.

We return to the proof of theorem 1. Let L be an operator acting over sufficiently smooth functions ζ : R d → R by:

Lζ = -< ∇f, ∇ζ > + h 2 d i,j=1 V ij ∂ 2 ij ζ. (11)
Let Φ ∈ G and suppose it is 6 times differentiable; using a classical result of semi-groups expansions (see [START_REF] Hille | Functional Analysis and Semi-groups[END_REF]), we have:

E(Φ(Z h )) = Φ(z) + hLΦ(z) + h 2 2 L 2 Φ(z) + O(h 3 ). ( 12
)
For ξ ∈ R d , we define Φ ξ : y → e i ξ,y-z . Let M : R d → R defined by M(ξ) = e i<ξ,∆> . Note that the function M belongs to the class C ∞ (R d ) and for s = 1, .., d,

∂ s M s j=1 ∂ξ ij ξ=0 = i s E( s j=1 ∆ ij ). ( 13
)
To determine the partial derivatives of M in 0 , we use [START_REF] Leslie | Cyclical learning rates for training neural networks[END_REF] to get:

M(ξ) = 1 + hLΦ t (z) + h 2 2 L 2 Φ ξ (z) + O(h 3 ). ( 14
)
After calculating the explicit expressions of LΦ ξ (z) and L 2 Φ ξ (z), we obtain:

M(ξ) = 1 -ih ∇f (z), ξ +h 2 i 2 d k=1 ∂ k f (z) ∂ k ∇f (z), ξ + ∇f (z), ξ 2 - 1 2 ξ T Vξ +O(h 3 ). ( 15
)
Therefore,

E(∆ j ) = -h∂ j f (z) + h 2 2 d k=1 ∂ k f (z)∂ 2 kj f (z) + O(h 3 ) (16) E(∆ j ∆ l ) = h 2 [∂ j f (z)∂ l f (z) + V jl ] + O(h 3 ) (17) E( s j=1 ∆ ij ) = O(h 3 ), f or s ≥ 3. ( 18
)
For the SH scheme:

E(∆ k ) = -h∂ k f (z) - h 2 2 < ∂ i ∇f (z), ∇f (z) > +O(h 3 ), (19)
E(∆ k ∆ l ) = 1 N h 2 N =1 ∂ k f (z)∂ l f (z) + O(h 3 ), (20)
E( s j=1 ∆ ij ) = O(h 3 ), for s ≥ 3. ( 21
)
Therefore, all hypotheses of the theorem 2 are satisfied for p = 2 which gives the conclusion.

Given theorem 1 we can trust SH to produce high order approximations of the solution which in turn will help obtain information on the Hessian and thus calibrate automatically the learning rate.

Rationale for the adaptive step proposal

In what follows, X, Y denotes the usual scalar product in R d and X the associated euclidean norm. For a matrix A ∈ R d×d , A denotes the matrix norm defined as:

A = sup X∈R d ,X =0 AX|| X .
If the loss function f is smooth enough, a Taylor expansion around a current point Y allows to write:

f (X) = f (Y ) + ∇f (Y ), X -Y + 1 2 ∇ 2 f (Y )(X -Y ), X -Y + O( X -Y 3 ), ( 22
)
where ∇f is the gradient of f computed at Y and ∇ 2 f (Y ) is the Hessian matrix of f at the same point. Note that the Hessian provides detailed information over the behavior of f around the current point Y but in practice ∇ 2 f (Y ) is a high dimensional object and it is impossible to deal with it directly. We will not try to compute it but will exploit its structure.

Neglecting higher order terms, the loss functional can be written as:

f (X) 1 2 AX, X -b, X , (23)
for some matrix A in R d×d and vector b in R d . To explain our method we will consider that in equation ( 23) we have equality. Then, ∇f

(X) = AX -b. If X opt ∈ R d is the minimum of f , then AX opt = b and thus ∇f (X) = A(X -X opt ). ( 24
)
We will forget for a moment that the gradient of f is not computed exactly (only an unbiased estimator being available in practice under the form of ∇f γ ). To minimize the function f , the gradient descent (also called Explicit Euler) scheme with learning rate h n > 0 reads:

X n+1 = X n -h n ∇f (X n ). ( 25
)
Then, denoting I d the identity matrix of d dimensions, we obtain:

X n+1 -X opt = (I d -h n A)(X n -X opt ). ( 26
)
On the other hand the Heun scheme (which is a Runge Kutta method of the second order) reads:

Y n+1 = Y n - h n 2 (∇f (Y n ) + ∇f (Y n -h n ∇f (Y n ))). (27)
Then,

Y n+1 -X opt = I d -hA + h 2 n A 2 2 (Y n -X opt ). (28)
At the step n, suppose that the two schemes start from the same point i.e., X n = Y n . Therefore, using (26) and (28):

Y n+1 -X n+1 = h 2 n A 2 2 (X n -X opt ) = h 2 n A 2 ∇f (X n ) (29)
On the other hand from (25) and ( 27) we can also write:

Y n+1 -X n+1 = h n 2 (∇f (X n ) -∇f (X n+1 )) (30)
Combining ( 29) and (30) we get:

(I d -h n A)∇f (X n ) = ∇f (X n+1 ). (31)
From ( 24) and (26) the stability criterion for the gradient descent (Explicit Euler) scheme is that ∇f (X n ) needs to be bounded, which is verified in particular when I d -h n A < 1. If this condition is true then for each step n:

(I d -h n A)∇f (X n ) ∇f (X n ) < 1. ( 32
)
Although the reciprocal is false, it is not far from true because, if (I d -hnA)∇f (Xn) ∇f (Xn) > 1 for some n, then this means in particular (I d -h n A) > 1 (because the matricial norm is a supremum) and then, except degenerate initial conditions X 0 , we obtain that X n will diverge (unless h n is adapted to ensure stability). So to enforce stability we have to request:

∇f (X n+1 ) ≤ ∇f (X n ) . ( 33
)
On the other hand, at every iteration n, we attempt to choose the biggest possible learning rate that guarantees (33), that is we accelerate the rate of convergence without breaking the stability criterion.

A natural question arises : what is the maximum value of h so that (33) still holds ?

Let us denote

ξ n (h) = ||(I d -hA)∇f (X n )|| 2 .
With X n being given, this is a second order polynomial in h; let h opt n be the maximum value of the learning rate h such (33) still holds. In other words, ξ n (h opt n ) = ∇f (X n ) 2 . Note that:

ξ n (h) = A∇f (X n ) 2 h 2 -2 A∇f (X n ), ∇f (X n ) h + ∇f (X n ) 2 . (34) Then, ξ n (h opt n ) = ||∇f (X n )|| 2 implies that h opt n = 0 or h opt n = 2 A∇f (Xn),∇f (Xn) A∇f (Xn) 2 . Since (I d -h n A)∇f (X n ) = ∇f (X n+1 ), we have that A∇f (X n ) = ∇f (Xn)-∇f (Xn+1)
hn . Then, unless ∇f (X n ) = ∇f (X n+1 ), which would imply that a critical point has already been reached:

h opt n = max 0, 2h n ∇f (X n ) -∇f (X n+1 ), ∇f (X n ) ||∇f (X n ) -∇f (X n+1 )|| 2 . ( 35
)
Note that it is important that in (35) the matrix A, which is impossible to compute, does not appear; only appear ∇f (X n ) and ∇f (X n+1 ) that are known.4 . To conclude: if the current learning rate is h n then it should be put to h opt n (given in equation ( 35)) to have the best convergence and stability properties. Note that when A is definite positive, the scalar product p n := A∇f (X n ), ∇f (X n ) must be positive. Therefore, if in a given iteration n, ∇f (X n )-∇f (X n+1 ), ∇f (X n ) (which equals h n p n ) happens to be negative this means that the second order assumption (23) made on f breaks down around the current point X n ; we cannot trust any computation made above and thus when p n < 0 we can set for instance h opt n = h n . Therefore, denoting now h n the learning rate at step n we will define:

h opt n = 2h n pn ||∇f (Xn)-∇f (Xn+1)|| 2 if p n > 0 h n otherwise. ( 36
)

Update policy

At a given iteration n, if h n h opt n , this means that the gradient descent is progressing too slow with the current learning rate and thus we need to accelerate it by increasing the learning rate. In practice, to not break the stability condition, the new learning rate must not be very close to h opt n . To avoid this risk, we choose a gradual update with an hyper-parameter β close to 1:

h n+1 = βh n + (1 -β)h opt n . (37)
Suppose now that h n h opt n . This means that the current learning rate breaks the convergence criteria. Then, we have to decrease it; contrary to previous policy, here we do not want a slow update because instability is already set in. A drastic measure is required, otherwise the whole optimization may become useless. We propose the following update rule:

h n+1 = (1 -β)h opt n . ( 38
)
This update is not necessarily close to h n but is a conservative choice to enter again the stability region, which, in practice gives good results.

The SGD-G2 algorithm

We present in this section the algorithm resulting from the above considerations; due to the stochastic nature of the gradient that is to be computed, for each iteration n in all the formulas in the former section the full gradient ∇f must be replaced by ∇f γn . Then, our suggested adaptive SGD called "SGD-G2" is described by the Algorithm 1 which includes the provision for mini-batch processing.

Algorithm 1 SGD-G2

Set hyper-parameter: β, mini-batch size M , choose stopping criterion Input: initial learning rate h 0 , initial guess X 0 Initialize iteration counter: n = 0 while stopping criterion not met do

select next mini-batch γ m n , m = 1, ..., M Compute g n = 1 M M m=1 ∇f γ m n (X n ) Compute gn = 1 M M m=1 ∇f γ m n (X n -h n g n ) Compute h opt n = 2hn gn-gn,gn gn-gn 2 if g n -gn , g n > 0 h n otherwise. if h opt n ≥ h n then h n+1 = βh n + (1 -β)h opt n else h n+1 = (1 -β)h opt n end if Update X n+1 = X n -h n+1 g n Update n → n + 1 end while Remark 1.
Several remarks are in order:

1. The computation of both g n and gn allows in principle to construct a more precise, second order in the learning rate, estimate of the next step X n+1 of the form X n -hn 2 (g n + gn ); this is not what we want here, the precise estimate is only used to calibrate the learning rate, in the end the SGD update formula is invoked to advance the network parameters X n to X n+1 .

2. It is crucial to have the same randomness in the computation of gn as the one present in the computation of g n . This ensures that a consistent approximation is obtained as detailed in Theorem 1.

3. We recommend to take the initial learning rate h 0 very small, in order to be sure to start in the stability region around X 0 , for instance h 0 = 10 -6 ; however numerical experiments seem to be largely insensitive to this value as detailed in section 4, figure 2.

Numerical experiments

Network architecture

We conducted experiments on three different data sets (MNIST, Fashion MNIST and CIFAR-10) using neural networks (CNNs) developed for image classification. We follow in this section the specifications in [3,[START_REF] Tensorflow | Convolutional Neural Network (CNN) TensorFlow Core, retrieved 2020[END_REF] and in [2] and reproduce below the corresponding architectures as given in the reference: MNIST/Fashion-MNIST (28 × 28 sized images): three dense 256 neurons feed-forward ReLU layers followed by a final dense 10 neurons feed-forward layer. CIFAR-10 (32 × 32 images with 3 color layers): a convolution layer with 3 × 3 filters, a 2 × 2 max pooling layer, a convolution layer with 3 × 3 filters, a 2 × 2 max pooling layer, a convolution layer with 3 × 3 filters followed by a flatten layer, a dense layer with a 64 neurons and a final dense layer with 10 neurons. All activations are ReLU except last one which is a softmax. The last layer returns the classification result. All hyper-parameters are chosen as in the references: the loss was minimized with the SGD / SGD-G2 algorithms and hyper-parameters β = 0.9 or as indicated in the figures; we used 10 epochs. The batch size is 32.

Discussion

First, we compare the performance of the adaptive SGD algorithm for different choices of the h 0 parameter. The results do not vary much among databases, we plot in figure 2 the ones for FMNIST. The variability is similar for MNIST but slightly larger for CIFAR10.

We come next to the heart of the procedure and we compare the performance of the adaptive SGD algorithm with the standard SGD algorithm for different initial learning rates. Recall that the goal of the adaptive procedure is not to beat the best possible SGD convergence but to identify fast enough the optimal learning rate. We recommend thus to start from a very small value of h 0 ; the algorithm will increase it up to the stability threshold. This is indeed what happens, see figures 1, 3 and 4. As the adaptive algorithm uses two (mini-batch) gradient evaluations per iteration, we take as x-axis in the plots the number of gradient evaluations and not the iteration counter, (in order not to favor the adaptive procedure which is more costly per iteration). We see that in all situations, starting from a tiny value of h 0 , the SGD-G2 algorithm quickly reaches the stability region and converge accordingly.

For MNIST and FMNIST the SGD-G2 cannot be surpassed by SGD, even for the optimal SGD learning rate; on the contrary for CIFAR10 the SGD with the optimal learning rate (which has to be searched through repeated runs) does converge better than the SGD-G2, but this is due to the counting procedure: if instead of number of gradient evaluations we count the iterations, the two are comparable as shown in figure 5 (same considerations apply for the Adam algorithm as illustrated in figure 6); so one can imagine that the adaptive part is only switched on from time to time (for instance once every 10 iterations), which will make its overhead negligible and still reach the optimal learning rate regime. Such fine tuning remains for future work.

In conclusion, an adaptive SGD algorithm (called SGD-G2) is proposed which is both computationally convenient and provides a stable, optimal learning rate value. The procedure is tested successfully on three image databases (MNIST, FMNIST, CIFAR10). [2] Feiyang Chen, Nan Chen, Hanyang Mao, and Hanlin Hu. Assessing four neural networks on handwritten digit recognition dataset (mnist), 11 2018. arXiv:1811.08278.

[3] Akinori Hidaka and Takio Kurita. Consecutive dimensionality reduction by canonical correlation analysis for visualization of convolutional neural networks. Proceedings of the ISCIE International Symposium on Stochastic Systems Theory and its Applications, 2017:160-167, 12 2017.

[4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. arXiv:1412.6980.

Figure 1 :

 1 Figure 1: Numerical results for the SGD and SGD-G2 algorithms on the MNIST database. Here β = 0.9.

Figure 2 :

 2 Figure 2: Numerical results for the SGD-G2 algorithm on the FMNIST database with several choices of the initial learning rate h 0 . Here β = 0.9. Similar results are obtained for the MNIST and CIFAR10 databases.

Figure 3 :

 3 Figure 3: Numerical results for the SGD and SGD-G2 algorithms on the FMNIST database. Here β = 0.9.

Figure 4 :

 4 Figure 4: Numerical results for the SGD and SGD-G2 algorithms on the CIFAR10 database. Here β = 0.9.

Figure 5 :

 5 Figure 5: Numerical results for the SGD and SGD-G2 algorithms on the CIFAR10 database. Here β = 0.9, and we compare in number of iterations instead of gradient evaluations.

[ 5 ]

 5 Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages

The linear growth at infinity is a technical hypothesis. It is for instance true when the parameter domain is closed and bounded and the function continuous.

We used the notation x to designate the integer part of a real number x, that is the largest integer smaller than x.

Recall that do not discuss here the stochastic part; in practice an unbiased estimator of the gradient ∇f (X) is available.


E Thomas Ewing
email: etewing@vt.edu

Everybody had it in Imagination if not in Reality: Information Flows during a Global

Keywords: Influenza, epidemic, information, history of medicine, digital humanities, text mining

The Russian Influenza, which reached the United States in late 1889, caused widespread disease as well as widespread reporting on the disease in newspapers and among medical professionals. A case study of one newspaper and one physician in a small town in Connecticut illustrates the value of deploying text mining, digital humanities, and close reading tools and techniques to understand the information ecosystem in which epidemic influenza affected individual lives and transformed medical reporting.

Introduction: In Imagination if Not In Reality

In a report submitted in May 1890 on the epidemic of so-called "Russian influenza," Connecticut doctor Cyrus B. Newton began with this remarkable statement about the relationship between information and disease: "The epidemic, on account of its foreign name, caused as much mischief as the disease itself, and the press made the matter still worse till everybody had it in imagination if not in reality, and wherever led the foot of man the grippe was sure to go, like Mary's little lamb, and no amount of quinine or antipyrine could drive it out." [START_REF] Newton | To the Committee on Matters of Professional Interest[END_REF] Newton, who practiced medicine and served on the Board of Health in Stafford, declared that some people "admit that they are in good health, but fear they have it in their system and that it is liable to reappear at any time, so strongly were they mentally impressed by it." Yet Newton declared that the actual number of deaths was overshadowed by the breadth of the disease's global impact: "So far as can be learned by statistics, the mortality from pneumonia was no greater during last Winter than in previous Winters. The movement of this ailment with speed of the wind over so vast an area of the globe was surely phenomenal."

Newton's statement suggests the importance of three kinds of information flow during a global epidemic which can be studied productively using methods drawn from historical analysis, digital humanities, and text mining. First, the reference to the "foreign name" of the "Russian influenza" and the "movement of this ailment with the speed of wind over so vast an area of the globe" calls attention to the spread of long-distance information. In an era when the introduction of transoceanic cables had transformed the transmission of news faster and further than ever before in human history, Newton's statement raises questions about the relationship between international news and local knowledge. How did a doctor integrate distant news into his understanding of a disease as it approached and then reached his own location? Second, Newton's statement that "the press made the matter still worse" reveals the tension in information flows between popular attitudes and expert knowledge. Embedded in Newtown's statement is a presumption that he, as an expert, had a better understanding of a disease outbreak than the general public whose information came from sensationalist sources such as newspapers. Finally, the declaration that "everyone had it in imagination if not in reality" calls attention to the tension in information flows between actual experiences, practices, and activities, on the one hand, and attitudes, perceptions, and anxieties, on the other. For Newton, the distinction between who actually had the disease and who imagined that they had the disease because of what they read in newspapers and heard from others in the community. For the historian, Newton's statement calls attention to need for using multiple tools, methods, and sources to ask how the flow of information globally, regionally, and locally corresponded to the lived experiences of individuals in particular contexts.

These three questions guide the structure of this unique effort by a historian to apply tools, methods, and sources of text mining and the digital humanities. The first question is explored by examining news reports about the Russian influenza in a local Connecticut newspaper, The Press, from Stafford Springs, Newton's hometown, to understand the emergence, transformation, and impact of international, regional, and local information networks. The second question is examined by comparing newspaper reporting about the outbreak with Newton's own contributions to public discussion in the form of letters, interviews, and actions covered in the same newspapers. Taking advantage of the methods available for large scale text analysis and visualization, this paper uses the tool of term collocation to construct networked relationships that suggest potential meanings and interpretations. The third question is addressed by examining mortality statistics at the local and state level to determine the actual impact of this disease and thus to evaluate, as Newton asked, whether it was more imagination or reality. This paper uses the Russian influenza as a case study to explore historical analysis at the intersection of data mining and digital humanities. (Bresalier 2012; Honigsbaum 2010; Mussell 2007) . This approach builds on previous scholarship that examines the spread of ideas during the epidemic, such as the concept of contagion, the relationship between influenza and other diseases, and patterns of mass anxiety, as well as explorations of how leading medical experts shaped public and expert understanding of the disease [START_REF] Ewing | Using Digital Humanities Tools to Interpret Information Dissemination during the Russian Flu, 1889-1890[END_REF][START_REF] Ewing | Will It Come Here?' Using Digital Humanities Tools to Explore Medical Understanding during the Russian Flu Epidemic, 1889-90[END_REF][START_REF] Ewing | The Two Diseases Are So Utterly Dissimilar': Using Digital Humanities Tools to Advance Scholarship in the Global History of Medicine[END_REF]). This study takes advantage of tools for largescale text analysis and visualization to supplement a close reading of articles with the goal of indicating how global, regional, and local information networks predicted the spread of disease, advocated preventive measures, and evaluated treatments. [START_REF] Mcshane | Visualising the Reception and Circulation of Early Modern Nuns' Letters[END_REF]van Vugt 2017) Yet the apparent discrepancy between the attention devoted to the Russian influenza and the actual number of cases and deaths, "as can be learned by statistics" published by health departments, revealed the importance of individual agents embedded in an information ecosystem. Recognizing how physicians functioned as nodes in this information econsystem is particularly important, because these agents served as both recipients and transmitters of information. Documenting the relationship between the rhetoric of an epidemic and the actual numbers of deaths focuses on causation patterns and temporal changes that may be ignored or diminished in diagrams. (Thirteenth Annual Report 1891) Yet as the opening example also demonstrates, the use of text mining and digital humanities tools must be combined with traditional interpretive techniques that identify the complex meanings visible only through a critical reading of texts and data in historical contexts. (Ewing and Randall 2019) Just as Newton used his medical training and analytical skills to understand a disease that "everybody had it in imagination if not in reality," this article uses methods of distant and close reading (Moretti 2013) to recognize, model, and interpret the complexities of information ecosystems. This paper explores information flows at the global, regional, and local level. As a worldwide pandemic, the Russian influenza was both a global network of disease transmission and a globalized network of information dissemination. Yet influenza and information also flowed at the regional level, as the disease travels from place to place, and at the local or even individual, as people get sick, seek medical treatment, and even die. This article builds on previous collaborations between humanities scholars and computer scientists applying text mining tools to understand historical epidemics of influenza. [START_REF] Amiri | Tracking and Analyzing Dynamics of News-Cycles during Global Pandemics: A Historical Perspective[END_REF][START_REF] Ewing | Gaining Insights into Epidemics by Mining Historical Newspapers[END_REF][START_REF] Ewing | Mining Coverage of the Flu: Big Data's Insights into an Epidemic[END_REF]. Exploring local information systems makes it possible to think about individuals not just as nodes in a network, but as agents, using their positions to interpret information, recirculate news, and act on the basis of received and transmitted knowledge. In the case of Cyrus Newton, this project examines information primarily in the local weekly newspaper, The Press, for Stafford Springs, Connecticut. By examining the movement of information along, between, and across these networks, this article contributes to historical understanding by exploring a unique moment when information flows converged at global, regional, and local levels.

Tracking Information Across Time: Term Frequency

An analysis of data mining and digital humanities can begin by charting term frequency, to determine how often a particular term appeared in an information source. Newspapers are particularly valuable for term frequency analysis, because each issue can be connected to a specific moment in time that fully indicates changes across time periods. In the case of a weekly newspaper, such as The Press, the sequencing is even more accurate than a daily newspaper, where the number of pages might vary from day to day, thus providing a misleading sense of coverage. Measuring term frequency in a specific newspaper title over time is more accurate than looking at multiple titles within a single database, particularly when the database is only partially representative, weekly and daily newspapers are both included, and each issue has a range of pages. In a similar manner, data bases of books and journals, such as the Google Books corpus or the Hathi Trust Library, can provide misleading information about term frequency over time, because the dates recorded by catalogers may not match up the actual publication dates, particularly for journals, volumes with annual publication dates, and multiple editions with different publication dates.

Figure 1, for the weekly newspaper, The Press in Stafford Springs, Connecticut, compares the number of times keywords influenza or grippe appeared in this four page newspaper. These terms did not appear at all before December 19, at least as indicated by a keyword search. Starting with the December 19 issue, when "grippe" appeared once, the frequency of these two keywords increased significantly, with the January 30, 1890 issue included more than fifteen instances of these two keywords. The terms then became less frequent, although they were still appearing throughout the remainder of the year. During these eleven weeks, grippe appeared 51 times, almost twice as frequently as 31 appearances of influenza. and not a serious threat to public health: "An influenza epidemic has appeared in Europe, and is traveling at express speed all over the continent. Big bugs and little folks, kings and hod lifters alike succumb to it. It isn't dangerous, but might unpleasant." By this time, on December 19, the disease "has appeared in New York," and was anticipated to "have its little spree" in the United States, but the likely effects were not expected to be serious, thus encouraging a more light hearted approach to its coming: "It is hoped that it may not tarry long, and if we all must have it, we may as well nerve ourselves and pray to have it quickly over with. The disease not being dangerous to life, one can, therefore, afford to follow its well marked course over this continent with amused interest."

This article concluded that the European elites --grand duke and ambassadors --had suffered the most, and yet already the threat was receding: "In St. Petersburg, where it was first detected and labelled, it still rages, but with diminished force."

On January 2, 1890, The Press offered its reader this reassuring (and mostly accurate) statement: "The 'grippe,' so far, is not dangerous in this country. Deaths from it are rare, and generally attributable to other diseases which it aggravates." The mortality rate had increased significantly in Paris, the article stated, "but its force as an epidemic was soon spent." The remaining coverage of the disease addressed evidence of outbreaks elsewhere in the United States, but with the same reassuring message that the disease was not serious. In Washington, for example, physicians were "sceptical" [sic] that the many cases of "mild influenza" or common colds were related to the "so-called 'grippe'" in Europe and now in New England. The physicians in Washington attributed the large number of cases to "mild and unseasonable weather," and to more human causes: "Imagination is also alleged to be strongly at work."

The latter statement introduces an interpretation of the epidemic as something imaginary rather than real, which Newton would repeat five months later in his review of the epidemic. This same issue of The Press also revealed that the epidemic was beginning to affect the community, yet the evidence of this impact was quite uneven and selective. Many people, for example, were attempting "preventives" of influenza such as burning sulphur or "champing" garlic. Evidence of actual impacts, by contrast, came from physicians, including reports attributed to three physicians: "Dr. Newton reports an unusual number of cases of hard colds, but in his opinion no epidemic influenza is prevalent here...Dr. Watts has treated about thirty cases of la grippe, none of them severe, and estimated that there are about 100 cases in town, some of them so mild as to need no treatment. Dr. Smith reports quite a number of cases, some quite sick, and others only slightly afflicted."

Finally, this same page included an advertisement that specifically invoked the disease and the widespread fear of this disease: "Influenza: If you are troubled with this dreaded malady use Lathrop's Spruce Gum Cough Syrup, and be relieved."

On January 16, 1890, in a letter published in The Press, Dr. Cyrus Newton offered his own interpretation of the coming epidemic and its likely effects at the local level. The letter began with a scientific explanation for the disease, yet quickly shifted into a denial that this disease was connected to other diseases, including pneumonia: "'La grippe,' or influenza, is a disease affecting the nasal or bronchial membranes, and has no relation to pneumonia as a cause, or of inflammation of the tonsils, or of the joints in the form of rheumatism." Dr. Newton's statement was incorrect, at least according to current medical research, which affirms that influenza can reduce the body's capacity to fight off an infection like pneumonia and thus contribute to death from the latter. (CDC 2018) The medical misinformation continued in the subsequent sentences, yet in ways that are revealing of how medical experts interpreted this surprisingly widespread outbreak of a familiar disease. According to Newton, influenza or la grippe "occur under favoring atmospheric conditions or influences at the same time, and the onset of the former disease is so sudden as to preclude the idea of contagion. Its movement across the continent is as rapid as a cold or thermal wave." Echoing the widespread belief at the time that influenza was spread by atmospheric conditions, Newton used the evidence of how quickly and widely the disease spread to argue against it being a contagious disease. Newton then amplified scientifically incorrect perspective with anecdotal evidence likely taken from newspaper reports: 8 "A prevailing west wind would bring the epidemic to a close; the vast land area does not afford the moisture as does the ocean. Dampness, with wintry temperature, causes all the diseases of the respiratory system. The theory of its germinal origin is maintained by many, but can hardly be sustained. Conditions of great humidity from the continual east wind, which is a sea breeze, with low thermometry at this season, is always productive of diseases of the air passages."

Newton thus seemed to believe that the influenza came to the United States, and Connecticut specifically, because an east wind, a "sea breeze," brought the disease across the moist air over the ocean. Cold and damp temperatures thus caused respiratory diseases, seemingly regardless of their transmission between humans. The statement that the "germinal origin" of the disease reflected the widespread adherence to the belief that atmospheric conditions accounted for the spread of disease, even as the so-called "bacterial revolution" was underway [START_REF] Bresalier | A Most Protean Disease': Aligning Medical Knowledge of Modern Influenza, 1890-1914[END_REF]. The following sentences provide a more scientifically valid statement that even though influenza was widespread, individual experiences would vary considerably: "The symptoms of this epidemic differ in no important respect from the ordinary influenza of every winter; the difference is now it affects everyone in greater or less degree. Some have it so light that a change in handkerchief once in six hours is sufficient treatment; others are decidedly ill for several days. Everyone who has so far escaped expects to have it before the epidemic is over."

Newton then offered a series of seemingly loosely associated comments further emphasizing that the disease did not represent a serious threat to public health: "If complicated with la grippe, the danger is not so greatly increased as it would be with a more formidable disease. Of its expectancy an amusing instance occurred quite recently, the attack ending in so pleasant a surprise that they said they would name the baby La Grippe. No doubt in many cases a common cold is called by the above name. The name influenza is a better one: it is more descriptive of the cause, an Italian word meaning influence. No one but a Frenchman knows what la grippe means."

After further speculation about the atmospheric causes of the disease, Newton concluded by associating the influenza with an electrically charged atmosphere, ocean currents, magnetic pulls, and the weather: "Another explanation is that there is something about the electrical condition of the air, it being negatively electric. Meat sent high up in the air by means of a kite has returned putrid. The Gulf Stream has been greatly blamed for causing this trouble. I would not be surprised to hear that possibly the north pole is doing all this mischief. The people say it's this nasty weather, and I think they are right."

The final sentence is perhaps most revealing of Newton's role as an information broker during this influenza epidemic. On the one hand, he uses the public platform of a newspaper to suggest interpretations that related to his expertise as a physician, including possible paths of transmission that included atmosphere and contagions. While Newton's rejection of the "germinal" transmission of the disease would ultimately be proven incorrect by the discovery of the virus that causes influenza, his views were within the medical mainstream of this transitional era. By contrast, the suggestion that an electrically charged atmosphere could spread disease represented a far less conventional explanation. In his final sentence, by contrast, Newton seemingly returned to an almost common-sense explanation --that the weather had contributed to the spread of the disease, a view reportedly held by "the people," and now approved by the leading local medical authority.

The January 16, 1890 issue of The Press offered further evidence of the local impact of this epidemic. In Stafford, mills, churches, festivals, and schools had closed or postponed activities because of many illness-related absences. Two doctors, Tiffany and Middleton, had been sick with "the fashionable disease," but continued to see patients. Towns reported that the disease was "assailing" people "vigorously," causing "much sickness," and producing victims whose names were published in the paper.

One week later, the January 23, 1890 issue reported on more victims, yet also began to cite evidence that the disease was having a broader impact on local society, including these specific examples: in Wales, "almost every family in town has been afflicted with la grippe," whereas in Westford, "most everyone here has had la grippe." A report on Stafford did not indicate the numbers affected, but emphasized instead how people perceived the disease: "Those who have encountered 'the grip' are not given to joking about it, but speak of it with respectful consideration, the more so that some authorities claim that it can work the same ground over twice." The escalation in the disease found a striking expression in the text for an advertisement in the January 30, 1890 issue: "Don't Let that Cough Run. The coughs and colds that hang on after 'la grippe' should be attended to or pneumonia may 'step in' without being invited. One of the best cough remedies known is Lathrop's Spruce Gum Cough Syrup 25c a bottle."

The same issue included the statement that in Stafford, 1890 "will be known in years to come as the winter of the grip," as the "doctors are still kept very busy."

In its February 6, 1890 issue, The Press reported that as many as twenty families at a time were 'in part or entirely prostrated" by the influenza. One victim, James Cavanaugh, had been so ill that two physicians had "pronounced his case hopeless" and predicted that he had less than an hour to live. Yet he rallied by the next day, leading to a surprise change in the course of his disease: "A consultation of his former physicians, in conjunction with Dr. Newton, was held, and treatment again commenced, and, strange to relate, he has continued to improve, and has now for several days been up and about, though not yet able to leave the house. This marvelous recovery is probably mostly due to his young and very vigorous constitution."

In Tolland, "grippe patients are slowly recovering, and no new cases are reported," while in Coventry, "the doctors speak of resting, because the grip is on the wane." The February 13, 1890 issue, for the first time in a month, did not mention any Stafford residents sick with influenza or related diseases. In Willington, a nearby town, Dr. F. Baker died suddenly at age 58: "He was taken with influenza Thursday evening, and that developed rapidly into pneumonia, and the best of medical skill could not prevent it from resulting fatally."

One week later, on February 20, 1890, in addition to reporting that Josiah Sparks "is one of the latest victims of the influenza," Herbert Lillibridge "is on the gain from a very severe attack of pneumonia," and Zalman A. Storrs is "very low with pneumonia," the newspaper suggested, in a more humorous way, that "some of us" in Willington who heard the thunder storms earlier in the week asked: "Was it to celebrate the cessation of the grip in these parts?"

The combination of methods of extracting term frequency and close reading provide a clear sense of both the narrative of the epidemic and the range of responses within a local community. After receiving almost no attention in the previous year, as documented by term frequency measures, this disease suddenly became a focus of attention in the newspapers for several weeks in early 1890. Even after the of the epidemic passed, the keywords "influenza" and "grippe" continued to appear, although increasingly in the form of advertising copy. A close reading of all the reporting on this disease reveals a similar pattern, as initial awareness of the disease as something occurring at a distance yielded to efforts to identify, explain, and control local manifestations of the disease. Yet these tools of humanities inquiry also reveal substantive differences in how the newspaper made sense of the epidemic, as the increased attention to the disease was always accompanied by questions about the severity of the disease. The fact that the newspaper made a joke about the disease in the February 20, 1890 issue, after the disease had been present in the community for at least six weeks, is further evidence that nuanced readings are essential for understanding the relationship between disease, information, and interpretation.

Medical Term Collocation: Text Mining

How can the method of close reading enhance the use of data mining tools for large scale textual analysis? More specifically, what do patterns of collocation and contextual proximity reveal about changing reports on the influenza epidemic? Figure 2 shows the terms most commonly collocated (within fifteen words) with keywords grippe and influenza in the approximately one hundred issues of the weekly newspaper, The Press, published in 1889-1890. The list was edited to exclude common stop words, misspelled words due to Optical Character Recognition errors, and terms not relevant to the diseases. The resulting list provides several insights into disease reporting in this newspaper. First, commonly collocated terms include many examples of symptoms associated with this disease: chill, cough, fever, inflammation, irritation, and throat. Second, some terms suggest levels of intensity associated with this disease: attack, bad, deaths, died, good, positive, prevalent, quick, severe, stubborn, suffering, and victims. Finally, the fact that the term, "sloan's" appears relatively frequently is puzzling, until it becomes clear that the newspaper regularly featured an advertisement for 'Sloan's Chill and Fever Remedy," which included influenza among a list of seven ailments (the others included chills, fevers, rheumatism, blind staggers, epizootic, and pink eye). An automated text extraction method, combined with close reading of the newspaper, identified ten significant terms related to health: five general terms (death, disease, doctor, epidemic, and fever), and five specific diseases (cancer, cholera, consumption, diphtheria, and pneumonia). These terms appeared frequently in the newspaper during these two years, with nearly 2,500 total appearances in nearly 100 issues with more than 2 million words. As shown in Figure 3, comparing the frequency of these key words in these two years, the general terms showed considerable consistency from 1889 to 1890, with a change of just 5% from 1889 to 1890. By contrast, the keywords "grippe" and "influenza" appeared only 4 times, combined, in 1889, compared to 108 appearances in 1890. These patterns suggest that whereas newspaper reporting generally about disease, medicine, and health showed significant continuity from 1889 to 1890, the epidemic of late 1889 and early 1890 significantly changed reporting about this specific disease. As shown in Figure 4, collocates included symptoms (catarrh, chilblains, cough, fever, pains, scrofula, sores, and ulcerated), measures of severity (afflicted, attack, bad, contagious, death, healed, obituaries, rate, strength, and suffering), references to anatomy (body, heart, kidney, lung, skin, and stomach), and terms related to treatment (balm, bottle, cure, doctor, druggists, medicine, physicians, pills, remedy, salve, sarsparilla, sloan's, and sulphur). 176 deaths in January 1890, nearly three times as many as recorded in January 1889. Examining the number of deaths during an epidemic provides yet another approach that connects data mining and the digital humanities. The number of deaths was affected by the severity of the disease as it spread through the population. which presumably functioned independently of how people were talking about the disease, thus further illuminating the tension embedded in Newton's claim about real and imaginary aspects of the epidemic. Measuring influenza is difficult, however, because the mild symptoms of most cases may not require medical care, the disease is often mistaken for related conditions, and in the case of death, influenza is often a secondary cause for deaths attributed to pneumonia, bronchitis, or some other acute lung disease. (Ewing 2019) Yet even with these constraints, a statistical analysis of deaths for the state of Connecticut provides an evidentiary basis for evaluating Newton's claims that the disease had no real impact on the population, that is, the disease network was less significant than the information network. In fact, death totals spiked in Connecticut in early 1890 by comparison to the same month in previous years (Monthly Bulletin, 1890), which certainly calls into question Newton's statement that grippe was mostly a matter of imagination with no real impact on society. Figure 6 indicates how the total number of deaths from all causes, including pneumonia and influenza, spiked in early 1890. In particular, January 1890 shows a remarkable increase, with the 1,693 deaths almost two-thirds higher than the total of 912 recorded in January 1889 as well as the monthly average of just over 1000 deaths in 1889. In February 1890, however, the total number of deaths decreased significantly, and remained at that level and even slightly decreased by May 1890, when Newton made his comments about death rates.

Given the remarkable increase in deaths in January, it is surprising that Newton would suggest that the effect of the epidemic was primarily one of imagination rather than reality. Two possible explanations can be suggested. First, while the death total spiked in January, with nearly 500 deaths twice as high as any previous monthly total, the number returned to normal levels by February, and declined through the spring and summer. When Newton submitted his report in May 1890, the situation was seemingly normal. In other words, Newton may have simply forgotten that the death rate was so high in January, yet this explanation is seemingly at odds with this confident statement: "So far as can be learned by statistics, the mortality from pneumonia was no greater during last Winter than in previous Winters." A second explanation is that Newton adhered to the belief common among physicians that influenza was not a serious or deadly disease, and he kept to that opinion despite the contradictory experience of the epidemic in the winter of 1890. More specifically, he may have continued to believe that reports about the deadly effects of the epidemic were generated by the press, and did not correspond to reality. Yet such a belief required disregarding state health department about deaths all causes and specifically from influenza and pneumonia, in order to believe that few people were seriously affected by the epidemic.

In keeping with an approach that examines evidence across scales, combining tools for text mining with close reading from the humanities, this section asks about the number of deaths in Newton's own community, the people he would have known and quite likely treated during their illness. According to the Connecticut Monthly Health Bulletin, Newton's home town of Stafford recorded 12 deaths from January to March 1890. Of these twelve deaths, one each in February and March were attributed to La Grippe, but none in January. Pneumonia also accounted for 2 deaths, both recorded in March. In other words, even though statistical reports at the state level demonstrate a significant change in the number of deaths from all causes and the number attributed to pneumonia and influenza, reports at the local level suggest relatively little impact from the disease, which is where a physician like Newton would have observed first hand.

The town clerk for Stafford Springs. W. H. Black, also reported on deaths in the community, thus providing an additional data point. For the year 1889, The Press of January 9, 1890 quoted Black's data indicating that not a single person in this community died of influenza. One death each were attributed to bronchitis and pneumonia, two respiratory diseases often associated with influenza. The disease that accounted for the most deaths was tuberculosis, recorded as phthisis, which caused eleven deaths, nearly 17% of the total of 65 deaths. One year later, in the January 22, 1891 issue of The Press, the same town clerk reported that the total number of deaths increased to 70, including one death attributed to influenza and five deaths attributed to pneumonia. The deadliest month in the year was September 1890, with 13 deaths, and the lowest number of deaths was recorded in February 1890, with only three deaths.

According to these figures, the low number of deaths in February suggests that the high level of public attention devoted to the influenza epidemic in the early months of 1890 did not correspond to a significant increase in total deaths --yet the increase from one death to six deaths from pneumonia is suggestive of some kind of change.

Conclusion: Dr. Cyrus Newton in Context

This question of what Newton observed as a practicing physician is an important reminder that a distinct contribution of historical study is recognizing that people make up information ecosystems --not just as nodes connecting edges, but also as the contributors of content, the interpreters of meaning, and the agents whose actions shape, and are shaped by, these systems. Age, gender, race, social status, and other elements of identity may be significant in determining how a person interacts with information systems.

In the case of Cyrus Newton, his elevated social standing as a physician makes it possible to understand how his interactions with information ecosystems durin were only part of an entire life narrative that these broader patterns even as he fashioned a distinctive understanding of the epidemic's impact within his own community.

Figure 1 .

 1 Figure 1. Keywords, "Grippe" or "Influenza," The Press, December 1889 to February 1890

Figure 2 .

 2 Figure 2. Collocated Terms with Keywords "Influenza" or Grippe, The Press, 1889-1890

Figure 3 .

 3 Figure 3. Disease Terms in The Press, 1889 and 1890

Figure 4 .

 4 Figure 4. Collocates with Disease Terms, The Press, 1889-1890

Figure 5 :

 5 Figure 5: Collocations with Keywords "Influenza" and "Grippe," and Disease Terms, 1889-1890

Figure 6 .

 6 Figure 6. Connecticut Deaths, All Causes and Pneumonia & Influenza, June 1888 to May 1890

Health of the State of Connecticut for the Year Ending November 30, 1889, with the Registration

Report for 1888, Relating to Births, Marriages, Deaths, and Divorces (New Haven: Tuttle, Morehouse, and Taylor, 1890).

State

Board of Health, Bureau of Vital Statistics, State of Connecticut. (1891) "Registration Report, for the Year Ending December 31, 1889," printed in Thirteenth Annual Report of the State Board of Health of the State of Connecticut for the Year Ending November 30, 1890, with the Registration Report for 1889, Relating to Births, Marriages, Deaths, and Divorces (New Haven: Tuttle, Morehouse, and Taylor, 1891). State Board of Health, Bureau of Vital Statistics, State of Connecticut. (1892) "Registration Report, for the Year Ending December 31, 1890," printed in Thirteenth Annual Report of the State Board of Health of the State of Connecticut for the Year Ending November 30, 1891, with the Registration Report for 1890, Relating to Births, Marriages, Deaths, and Divorces (New Haven: Tuttle, Morehouse, and Taylor, 1892). The Press, weekly newspaper, Stafford Springs, Connecticut. Available in full text from United States Library of Congress, Chronicling America: https://chroniclingamerica.loc.gov/lccn/sn92051501. Thirteenth Annual Report of the State Board of Health of the State of Connecticut for the Year ending November 30, 1890 (New Haven: Tuttle, Morehouse, & Taylor Printers, 1891). van Vugt, Ingeborg. (2017) "Using Multi-Layered Networks to Disclose Books in the Republic of Letters," Journal of Historical Network Research, Vol. 1 (2017):25-51. Zhao, Zhehuan, Zhihao Yang, Ling Luo, Lei Wang, Yin Zhang, Hongfei Lin, and Jian Wang. (2017).

"Disease named entity recognition from biomedical literature using a novel convolutional neural network," BMC Medical Genomics, Vol. 10, Suppl. 5:73.

Dr. Newton's statement cited above occurred in the context of an information ecosystem, as he was reporting on behalf of the committee on professional interests to the ninety-ninth annual convention of the Connecticut Medical Society, meeting in New Haven on May 28-29, 1890. His report was then published in the Proceedings of the Connecticut Medical Society, which allowed his statements to circulate more broadly beyond those assembled for the meeting. In this way, Newton was both a consumer of information from newspapers, medical journals, professional dialogues, and consultations with patients, and a contributor of information to the local newspaper, the Connecticut medical journal, the physicians' society, and his immediate community of doctors, patients, and others. Text mining and other tools for large scale analysis make important contributions to understanding the scope and scale of this reporting, while also pointing to connections between diseases, symptoms, treatment, and severity. A close reading of newspaper article provides a deeper appreciation of the tension between reporting on disease and experiencing an epidemic, while text mining and mortality statistics provide empirical evidence for situating these interpretations in social context. Most importantly, this combination of methods facilitates an interpretation of who an individual physician, Dr. Cyrus Newton, engaged with