Namig J Guliyev
email: njguliyev@gmail.com

Vugar E Ismailov
email: vugaris@mail.ru

A SINGLE HIDDEN LAYER FEEDFORWARD NETWORK WITH ONLY ONE NEURON IN THE HIDDEN LAYER CAN APPROXIMATE ANY UNIVARIATE FUNCTION

Keywords: 2010 Mathematics Subject Classification. 41A30, 65D15, 92B20 Sigmoidal functions, λ-monotonicity, Calkin-Wilf sequence, Bernstein polynomials, Continued fractions, Smooth transition function

The possibility of approximating a continuous function on a compact subset of the real line by a feedforward single hidden layer neural network with a sigmoidal activation function has been studied in many papers. Such networks can approximate an arbitrary continuous function provided that an unlimited number of neurons in a hidden layer is permitted. In this paper, we consider constructive approximation on any finite interval of R by neural networks with only one neuron in the hidden layer. We construct algorithmically a smooth, sigmoidal, almost monotone activation function σ providing approximation to an arbitrary continuous function within any degree of accuracy. This algorithm is implemented in a computer program, which computes the value of σ at any reasonable point of the real axis.

Introduction

Neural networks are being successfully applied across an extraordinary range of problem domains, in fields as diverse as computer science, finance, medicine, engineering, physics, etc. The main reason for such popularity is their ability to approximate arbitrary functions. For the last 30 years a number of results have been published showing that the artificial neural network called a feedforward network with one hidden layer can approximate arbitrarily well any continuous function of several real variables. These results play an important role in determining boundaries of efficacy of the considered networks. But the proofs are usually do not state how many neurons should be used in the hidden layer. The purpose of this paper is to prove constructively that a neural network having only one neuron in its single hidden layer can approximate arbitrarily well all continuous functions defined on any compact subset of the real axis.

The building blocks for neural networks are called neurons. An artificial neuron is a device with n real inputs and an output. This output is generally a superposition of a univariate function with an affine function in the n-dimensional Euclidean space, that is a function of the form σ(w 1 x 1 + • • • + w n x n -θ). The neurons are organized in layers. Each neuron of each layer is connected to each neuron of the subsequent (and thus previous) layer. Information flows from one layer to the subsequent layer (thus the term feedforward). A feedforward neural network with one hidden layer has three layers: input layer, hidden layer, and output layer. A feedforward network with one hidden layer consisting of r neurons computes functions of the form

c 0 + r i=1 c i σ(w i • x -θ i ).
(1.1)

Here the vectors w i , called weights, are vectors in R n ; the thresholds θ i and the coefficients c i are real numbers and σ is a univariate activation function. The following are common examples of activation functions:

σ(t) = 1 1 + e -t
(the squashing function),

σ(t) =        0, t ≤ -1, t + 1 2 , -1 ≤ t ≤ 1, 1, t ≥ 1
(the piecewise linear function),

σ(t) = 1 π arctan t + 1 2
(the arctan sigmoid function),

σ(t) = 1 √ 2π t -∞
e -x 2 /2 dx (the Gaussian function).

In many applications, it is convenient to take the activation function σ as a sigmoidal function which is defined as

lim t→-∞ σ(t) = 0 and lim t→+∞ σ(t) = 1.
The literature on neural networks abounds with the use of such functions and their superpositions. Note that all the above activation functions are sigmoidal.

In approximation by neural networks, there are two main problems. The first is the density problem of determining the conditions under which an arbitrary target function can be approximated arbitrarily well by neural networks. The second problem, called the complexity problem, is to determine how many neurons in hidden layers are necessary to give a prescribed degree of approximation. This problem is almost the same as the problem of degree of approximation (see [START_REF] Barron | Universal approximation bounds for superposition of a sigmoidal function[END_REF][START_REF] Cao | The estimate for approximation error of neural networks: A constructive approach[END_REF][START_REF] Hahm | Extension of localized approximation by neural networks[END_REF]). The possibility of approximating a continuous function on a compact subset of the real line (or n-dimensional space) by a single hidden layer neural network with a sigmoidal activation function has been well studied in a number of papers. Different methods were used. Carroll and Dickinson [START_REF] Carroll | Construction of neural nets using the Radon transform[END_REF] used the inverse Radon transformation to prove the universal approximation property of single hidden layer neural networks. Gallant and White [START_REF] Gallant | There exists a neural network that does not make avoidable mistakes[END_REF] constructed a specific continuous, nondecreasing sigmoidal function, called a cosine squasher, from which it was possible to obtain any Fourier series. Thus their activation function had the density property. Cybenko [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF] and Funahashi, [START_REF] Funahashi | On the approximate realization of continuous mapping by neural networks[END_REF] independently from each other, established that feedforward neural networks with a continuous sigmoidal activation function can approximate any continuous function within any degree of accuracy on compact subsets of R n . Cybenko's proof uses the functional analysis method, combining the Hahn-Banach Theorem and the Riesz Representation Theorem, whiles Funahashi's proof applies the result of Irie and Miyake [START_REF] Irie | Capability of three-layered perceptrons[END_REF] on the integral representation of f ∈ L 1 (R n ) functions, using a kernel which can be expressed as a difference of two sigmoidal functions. Hornik, Stinchcombe and White [START_REF] Hornik | Multilayer feedforward networks are universal approximators[END_REF] applied the Stone-Weierstrass theorem, using trigonometric functions.

Kůrková [START_REF] Kůrková | Kolmogorov's theorem and multilayer neural networks[END_REF] proved that staircase like functions of any sigmoidal type can approximate continuous functions on any compact subset of the real line within an arbitrary accuracy. This is effectively used in Kůrková's subsequent results, which show that a continuous multivariate function can be approximated arbitrarily well by two hidden layer neural networks with a sigmoidal activation function (see [START_REF] Kůrková | Kolmogorov's theorem is relevant[END_REF][START_REF] Kůrková | Kolmogorov's theorem and multilayer neural networks[END_REF]).

Chen, Chen and Liu [START_REF] Chen | A constructive proof of Cybenko's approximation theorem and its extensions[END_REF] extended the result of Cybenko by proving that any continuous function on a compact subset of R n can be approximated by a single hidden layer feedforward network with a bounded (not necessarily continuous) sigmoidal activation function. Almost the same result was independently obtained by Jones [START_REF] Jones | Constructive approximations for neural networks by sigmoidal functions[END_REF].

Costarelli and Spigler [START_REF] Costarelli | Constructive approximation by superposition of sigmoidal functions[END_REF] reconsidered Cybenko's approximation theorem and for a given function f ∈ C[a, b] constructed certain sums of the form (1.1), which approximate f within any degree of accuracy. In their result, similar to [START_REF] Chen | A constructive proof of Cybenko's approximation theorem and its extensions[END_REF], σ is bounded and sigmoidal. Therefore, when σ ∈ C(R), the result can be viewed as a density result in C[a, b] for the set of all functions of the form (1.1).

Chui and Li [START_REF] Chui | Approximation by ridge functions and neural networks with one hidden layer[END_REF] proved that a single hidden layer network with a continuous sigmoidal activation function having integer weights and thresholds can approximate an arbitrary continuous function on a compact subset of R. Ito [START_REF] Ito | Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory[END_REF] established a density result for continuous functions on a compact subset of R by neural networks with a sigmoidal function having only unit weights. Density properties of a single hidden layer network with a restricted set of weights were studied also in other papers (for a detailed discussion see [START_REF] Ismailov | Approximation by neural networks with weights varying on a finite set of directions[END_REF]).

In many subsequent papers, which dealt with the density problem, nonsigmoidal activation functions were allowed. Among them are the papers by Stinchcombe and White [START_REF] Stinchcombe | Approximating and learning unknown mappings using multilayer feedforward networks with bounded weights[END_REF], Cotter [START_REF] Cotter | The Stone-Weierstrass theorem and its application to neural networks[END_REF], Hornik [START_REF] Hornik | Approximation capabilities of multilayer feedforward networks[END_REF], Mhaskar and Micchelli [START_REF] Mhaskar | Approximation by superposition of a sigmoidal function and radial basis functions[END_REF], and other researchers. The more general result in this direction belongs to Leshno, Lin, Pinkus and Schocken [START_REF] Leshno | Multilayer feedforward networks with a non-polynomial activation function can approximate any function[END_REF]. They proved that the necessary and sufficient condition for any continuous activation function to have the density property is that it not be a polynomial. For a detailed discussion of most of the results in this section, see the review paper by Pinkus [START_REF] Pinkus | Approximation theory of the MLP model in neural networks[END_REF].

It should be remarked that in all the above mentioned works the number of neurons r in the hidden layer is not fixed. As such to achieve a desired precision one may take an excessive number of neurons. This, in turn, gives rise to the problem of complexity (see above).

Our approach to the problem of approximation by single hidden layer feedforward networks is different and quite simple. We consider networks (1.1) defined on R with a limited number of neurons (r is fixed!) in a hidden layer and ask the following fair question: is it possible to construct a well behaved (that is, sigmoidal, smooth, monotone, etc.) universal activation function providing approximation to arbitrary continuous functions on any compact set in R within any degree of precision? We show that this is possible even in the case of a feedforward network with only one neuron in its hidden layer (that is, in the case r = 1). The basic form of our theorem claims that there exists a smooth, sigmoidal, almost monotone activation function σ with the property: for each univariate continuous function f on the unit interval and any positive ε one can chose three numbers c 0 , c 1 and θ such that the function c 0 + c 1 σ(t -θ) gives ε-approximation to f . It should be remarked that we prove not only the existence result but also give an algorithm for constructing the mentioned universal sigmoidal function. For a wide class of Lipschitz continuous functions we also give an algorithm for evaluating the numbers c 0 , c 1 and θ.

For numerical experiments we used SageMath [START_REF] Stein | Sage Mathematics Software (Version 6.10)[END_REF]. We wrote a code for creating the graph of σ and computing σ(t) at any reasonable t ∈ R. The code is open-source and available at http://sites.google.com/site/njguliyev/papers/sigmoidal.

The theoretical result

We begin this section with the definition of a λ-increasing (λ-decreasing) function. Let λ be any nonnegative number. A real function f defined on (a, b) is called λ-increasing (λ-decreasing) if there exists an increasing (decreasing) function

u : (a, b) → R such that |f (x) -u(x)| ≤ λ, for all x ∈ (a, b).
If u is strictly increasing (or strictly decreasing), then the above function f is called a λ-strictly increasing (or λ-strictly decreasing) function. Clearly, 0-monotonicity coincides with the usual concept of monotonicity and a λ 1 -increasing function is

λ 2 -increasing if λ 1 ≤ λ 2 .
The following theorem is valid.

f (t) -c 1 σ α b -a t -θ -c 0 < ε for all t ∈ [a, b].
Proof. Let α be any positive number. Divide the interval [α, +∞) into the segments [α, 2α], [2α, 3α], . . .. Let h(t) be any strictly increasing, infinitely differentiable function on [α, +∞) with the properties (1) 0 < h(t) < 1 for all t ∈ [α, +∞);

(2) 1 -h(α) ≤ λ;

(3) h(t) → 1, as t → +∞. The existence of a strictly increasing smooth function satisfying these properties is easy to verify. Note that from conditions (1)-(3) it follows that any function w(t) satisfying the inequality h(t) < w(t) < 1 for all t ∈ [α, +∞), is λ-strictly increasing and w(t) → 1, as t → +∞.

We are going to construct σ obeying the required properties in stages. Let {u n (t)} ∞ n=1 be the sequence of all polynomials with rational coefficients defined on [0, 1]. First, we define σ on the closed intervals [(2m -1)α, 2mα], m = 1, 2, . . ., as the function

σ(t) = a m + b m u m t α -2m + 1 , t ∈ [(2m -1)α, 2mα],
or equivalently,

σ(αt + (2m -1)α) = a m + b m u m (t), t ∈ [0, 1], (2.1)
where a m and b m = 0 are chosen in such a way that the condition

h(t) < σ(t) < 1 (2.2)
holds for all t ∈ [(2m -1)α, 2mα].

At the second stage we define σ on the intervals [2mα, (2m + 1)α], m = 1, 2, . . ., so that it is in C ∞ (R) and satisfies the inequality (2.2). Finally, in all of (-∞, α) we define σ while maintaining the C ∞ strict monotonicity property, and also in such a way that lim t→-∞ σ(t) = 0. We obtain from the properties of h and the condition (2.2) that σ(t) is a λ-strictly increasing function on the interval [α, +∞) and σ(t) → 1, as t → +∞. Note that the construction of a σ obeying all the above conditions is feasible. We show this in the next section.

From (2.1) it follows that for each m = 1, 2, . . .,

u m (t) = 1 b m σ(αt + (2m -1)α) - a m b m . (2.3)
Let now g be any continuous function on the unit interval [0, 1]. By the density of polynomials with rational coefficients in the space of continuous functions over any compact subset of R, for any ε > 0 there exists a polynomial u m (t) of the above form such that

|g(t) -u m (t)| < ε, for all t ∈ [0, 1]. This together with (2.3) means that |g(t) -c 1 σ(αt -s) -c 0 | < ε, (2.4)
for some c 0 , c 1 , s ∈ R and all t ∈ [0, 1]. Note that (2.4) proves our theorem for the unit interval [0, 1]. Using linear transformation it is not difficult to go from [0, 1] to any finite closed interval [a, b]. Indeed, let f ∈ C[a, b], σ be constructed as above and ε be an arbitrarily small positive number. The transformed function g(t) = f (a + (b -a)t) is well defined on [0, 1] and we can apply the inequality (2.4). Now using the inverse transformation t = x-a b-a , we can write that

|f (x) -c 1 σ(wx -θ) -c 0 | < ε, (2.5)
where w = α b-a and θ = αa b-a + s. The last inequality (2.5) completes the proof.

Remark 2.1. The idea of using a limited number of neurons in hidden layers of a feedforward network was first implemented by Maiorov and Pinkus [START_REF] Maiorov | Lower bounds for approximation by MLP neural networks[END_REF]. They proved the existence of a sigmoidal, strictly increasing, analytic activation function such that two hidden layer neural networks with this activation function and a fixed number of neurons in each hidden layer can approximate any continuous multivariate function over the unit cube in R n . Note that the result is of theoretical value and the authors do not suggest constructing and using their sigmoidal function.

Using the techniques developed in [START_REF] Maiorov | Lower bounds for approximation by MLP neural networks[END_REF], we showed theoretically that if replace the demand of analyticity by smoothness and monotonicity by λ-monotonicity, then the number of neurons in hidden layers can be reduced substantially (see [START_REF] Ismailov | On the approximation by neural networks with bounded number of neurons in hidden layers[END_REF]). We stress again that in both papers the algorithmic implementation of the obtained results is not discussed nor illustrated by numerical examples.

In the next section, we propose an algorithm for computing the above sigmoidal function σ at any point of the real axis. The code of this algorithm is available at http://sites.google.com/site/njguliyev/papers/sigmoidal. As examples, we include in the paper the graph of σ (see Figure 1) and a numerical table (see Table 1) containing several computed values of this function.

Algorithmic construction of the universal sigmoidal function

Step 1. Definition of h(t). Set

h(t) := 1 - min{1/2, λ} 1 + log(t -α + 1)
.

Note that this function satisfies the conditions 1)-3) in the proof of Theorem 2.1.

Step 2. Enumerating the rationals. Let a n be Stern's diatomic sequence:

a 1 = 1, a 2n = a n , a 2n+1 = a n + a n+1 , n = 1, 2, . . . .
It should be remarked that this sequence first appeared in print in 1858 [START_REF] Stern | Über eine zahlentheoretische Funktion[END_REF] and has been the subject of many papers (see, e.g., [25] and the references therein). The Calkin-Wilf [START_REF] Calkin | Recounting the rationals[END_REF] sequence q n := a n /a n+1 contains every positive rational number exactly once and hence the sequence r 0 := 0, r 2n := q n , r 2n-1 := -q n , n = 1, 2, . . . , is the enumeration of all the rational numbers. It is possible to calculate q n and r n directly. Let (11 . . . 1

f k 00 . . . 0 f k-1 11 . . . 1 f k-2 . . . 00 . . . 0 f1 11 . . . 1 f0 ) 2
be the binary code of n. Here, f 0 , f 1 , . . . , f k show the number of 1-digits, 0-digits, 1-digits, etc., respectively, starting from the end of the binary code. Note that f 0 can be zero. Then q n equals the continued fraction

[f 0 ; f 1 , . . . , f k ] := f 0 + 1 f 1 + 1 f 2 + 1 . . . + 1 f k . (3.1)
The calculation of r n is reduced to the calculation of q n/2 , if n is even and q (n+1)/2 , if n is odd.

Step 3. Enumerating the polynomials with rational coefficients.

It is clear that every positive rational number determines a unique finite continued fraction [n 0 ; n 1 , . . . , n l ] with n 0 ≥ 0, n 1 , . . . , n l-1 ≥ 1 and n l ≥ 2.

Since each non-zero polynomial with rational coefficients can uniquely be written as r k0 +r k1 t+. . .+r k d t d , where r k d = 0 (i.e. k d > 0), we have the following bijection between the set of all non-zero polynomials with rational coefficients and the set of all positive rational numbers:

r k0 + r k1 t + . . . + r k d t d → [k 0 ; k 1 + 1, . . . , k d + 1]
We define u 1 (t) := 0 and

u m (t) := r n0 + r n1-1 t + . . . + r n d -1 t d , m = 2, 3, . . . , where q m-1 = [n 0 ; n 1 , . . . , n d ].
Step 4. Construction of σ on [(2m -1)α, 2mα].

Set M := h((2m+1)α). Besides, for each polynomial u m (t) = d 0 +d 1 t+. . .+d k t k , set and

A 1 := d 0 + d 1 -|d 1 | 2 + . . . + d k -|d k | 2 (3.2)
A 2 := d 0 + d 1 + |d 1 | 2 + . . . + d k + |d k | 2 . (3.3)
It is not difficult to verify that

A 1 ≤ u m (t) ≤ A 2 , t ∈ [0, 1].
If u m is constant, then we put

σ(t) := 1 + M 2 .
Otherwise, we define σ(t) as the function

σ(t) = a m + b m u m t α -2m + 1 , t ∈ [(2m -1)α, 2mα], (3.4)
where

a m := (1 + 2M )A 2 -(2 + M )A 1 3(A 2 -A 1 ) , b m := 1 -M 3(A 2 -A 1 ) . (3.5)
Note that a m , b m are the coefficients of the linear function

y = a m + b m x mapping the closed interval [A 1 , A 2 ] onto the closed interval [(1 + 2M )/3, (2 + M )/3]. Thus, h(t) < M < 1 + 2M 3 ≤ σ(t) ≤ 2 + M 3 < 1, (3.6)
for all t ∈ [(2m -1)α, 2mα].

Step 5. Construction of σ on [2mα, (2m + 1)α].

To define σ on the intervals [2mα, (2m + 1)α] we will use the smooth transition function

β a,b (t) := β(b -t) β(b -t) + β(t -a) ,
where Table 1. Some computed values of σ

β(t) := e -1/t , t > 0, 0, t ≤ 0. It is easy to see that β a,b (t) = 1 for t ≤ a, β a,b (t) = 0 for t ≥ b and 0 < β a,b (t) < 1 for a < t < b. Set K := σ(2mα) + σ((2m + 1)α) 2 t σ t σ t σ t σ t σ α 0.
(α = 1, λ = 1/2)
Since both σ(2mα) and σ((2m + 1)α) belong to the interval (M, 1), we obtain that K ∈ (M, 1). First we extend σ smoothly to the interval [2mα, 2mα+α/2]. Take ε := (1-M )/6 and choose δ ≤ α/2 such that

a m + b m u m t α -2m + 1 -(a m + b m u m (1)) ≤ ε, t ∈ [2mα, 2mα + δ]. (3.7)
Let us show how one can choose this δ. If u m is constant, it is sufficient to take δ := α/2. If u m is not constant, we take

δ := min εα b m C , α 2
,
where C is a number satisfying |u m (t)| ≤ C for t ∈ (1, 1.5). Now define σ on the first half of the interval [2mα, (2m + 1)α] as the function

σ(t) := K -β 2mα,2mα+δ (t) × K -a m -b m u m t α -2m + 1 , t ∈ 2mα, 2mα + α 2 . (3.8)
Let us verify that σ(t) satisfies the condition (2.2). Indeed, if 2mα + δ ≤ t ≤ 2mα + α 2 , then there is nothing to prove, since σ(t) = K ∈ (M, 1). If 2mα ≤ t < 2mα + δ, then 0 < β 2mα,2mα+δ (t) ≤ 1 and hence from (3.8) + ε , for t ∈ [2mα, 2mα + δ). Since ε = (1 -M )/6, the inclusion A(t) ∈ (M, 1) is valid. Now since both K and A(t) belong to (M, 1), we finally conclude that

h(t) < M < σ(t) < 1, for t ∈ 2mα, 2mα + α 2 .
We define σ on the second half of the interval in a similar way:

σ(t) := K -(1 -β (2m+1)α-δ,(2m+1)α (t)) × K -a m+1 -b m+1 u m+1 t α -2m -1 , t ∈ 2mα + α 2 , (2m + 1)α ,
where

δ := min εα b m+1 C , α 2 , ε := 1 -h((2m + 3)α) 6 , C ≥ sup [-0.5,0] |u m+1 (t)|.
One can easily verify, as above, that the constructed σ(t) satisfies the condition (2.2) on [2mα, 2mα + α/2] and

σ 2mα + α 2 = K, σ (i) 2mα + α 2 = 0, i = 1, 2, . . . .
Step 6. Construction of σ on (-∞, α).

Finally, we put

σ(t) := 1 -β(α -t) 1 + h(3α) 2 , t ∈ (-∞, α).
It is not difficult to verify that σ is a strictly increasing, smooth function on (-∞, α). Note also that σ(t) → σ(α) = (1 + h(3α))/2 (see Step 4), as t tends to α from the left and σ (i) (α) = 0, for i = 1, 2, . . ..

Step 6 completes the construction of the universal activation function σ, which satisfies Theorem 2.1.

4.

The algorithm for evaluating the numbers c 0 , c 1 and θ Although Theorem 2.1 is valid for all continuous functions, in practice it is quite difficult to calculate algorithmically c 0 , c 1 and θ in Theorem 2.1 for badly behaved continuous functions. The main difficulty arises while attempting to design an efficient algorithm for the construction of a best approximating polynomial within any given degree of accuracy. But for certain large classes of well behaved functions, the computation of the above numbers is doable. In this section, we show this for the class of Lipschitz continuous functions.

Assume that f is a Lipschitz continuous function on [a, b] with a Lipschitz constant L. In order to find the parameters c 0 , c 1 and θ algorithmically, it is sufficient to perform the following steps.

Step 1. Going to the unit interval.

Consider the function g

(t) = f (a + (b -a)t), which is Lipschitz continuous on [0, 1] with a Lipschitz constant L 1 = L(b -a). Denote by B n (x) := n k=0 g k n n k x k (1 -x) n-k
the n-th Bernstein polynomial of the function g. Let ε > 0 be given.

Step 2. Finding the position of a given rational number. Define the functions

p q : Q + → N, p r : Q → N ∪ {0},
which return the positions of a positive rational number and a rational number in the sequences {q n } and {r n }, respectively (see Section 3). We start with the computation of p q . Let q be a positive rational number. If (3.1) is the continued fraction representation of q with k even (we may always consider [f

0 ; f 1 , . . . , f k -1, 1] instead of [f 0 ; f 1 , . . . , f k ] if needed)
, then the binary representation of the position p q (q) of q in the Calkin-Wilf sequence is p q (q) = (11 . . . 1

f k 00 . . . 0 f k-1 11 . . . 1 f k-2 . . . 00 . . . 0 f1 11 . . . 1 f0 ) 2 .
Now we can easily find p r (r) by the formula p r (r) =      0, r = 0, 2p q (r), r > 0, 2p q (-r) -1, r < 0.

Step 3. Finding n ∈ N such that |B n (x) -g(x)| ≤ ε/2, x ∈ [0, 1]. We use the inequality (see [START_REF] Sikkema | Der Wert einiger Konstanten in der Theorie der Approximation mit Bernstein-Polynomen[END_REF]) If u m = p is not constant then we put

|B n (x) -g(x)| ≤ χω 1 √ n , x ∈ [0,
c 1 := 1 b m , c 0 := - a m b m ,
where a m and b m are computed using the formulas (3.2), (3.3) and (3.5).

Step 7. Evaluating the numbers c 0 , c 1 and θ.

In this step, we return to our original function f and calculate the numbers c 0 , c 1 and θ (see Theorem 2.1). The numbers c 0 and c 1 have been calculated above (they are the same for both g and f ). In order to find θ, we use the formula θ = αa b-a + s. Remark 4.1. Note that some computational difficulties may arise while implementing the above algorithm in standard computers. For some functions, the index m of a polynomial u m in Step 5 may be extraordinarily large. In this case, a computer is not capable of producing this number, hence the numbers c 0 , c 1 and θ.

Theorem 2 . 1 .

 21 For any positive numbers α and λ, there exists a C ∞ (R), sigmoidal activation function σ : R → R which is strictly increasing on (-∞, α), λ-strictly increasing on [α, +∞), and satisfies the following property: For any finite closed interval [a, b] of R and any f ∈ C[a, b] and ε > 0 there exist three real numbers c 0 , c 1 and θ for which

Figure 1 .

 1 Figure 1. The graph of σ on [0, 10] (α = 1, λ = 1/2)

  88087 100 0.95548 200 0.96034 0 0.55682 -100 0.00868 10 0.95095 110 0.94162 210 0.96064 -10 0.07655 -110 0.00790 20 0.95879 120 0.97124 220 0.94790 -20 0.04096 -120 0.00725 30 0.96241 130 0.97163 230 0.96119 -30 0.02796 -130 0.00670 40 0.96464 140 0.94397 240 0.97430 -40 0.02122 -140 0.00623 50 0.94931 150 0.97230 250 0.95743 -50 0.01710 -150 0.00581 60 0.96739 160 0.97259 260 0.97461 -60 0.01432 -160 0.00545 70 0.93666 170 0.94573 270 0.95793 -70 0.01232 -170 0.00514 80 0.96910 180 0.94622 280 0.94979 -80 0.01081 -180 0.00485 90 0.93951 190 0.94669 290 0.96670 -90 0.00963 -190 0.00460

3 -ε, 2+M 3

 33 it follows that for each t ∈ [2mα, 2mα + δ), σ(t) is between the numbers K andA(t) = a m + b m u m t α -2m + 1 . On the other hand, from (3.7) we obtain that a m + b m u m (1) -ε ≤ A(t) ≤ a m + b m u m (1) + ε,which together with (3.4) and (3.6) yields that A(t) ∈ 1+2M

2 ,Step 6 .

 26 where • is the ceiling function defined as x := min{k ∈ Z | k ≥ x}.Step 4. Finding a polynomial p with rational coefficients such that |p(x) -B n (x)| ≤ ε/2, x ∈ [0, 1]. If B n (x) = a 0 + a 1 x + . . . + a k x k then it is sufficient to choose d 0 , d 1 , . . . , d k ∈ Q such that |a 0 -d 0 | + |a 1 -d 1 | + . . . + |a k -d k | ≤ ε 2 and put p(x) := d 0 + d 1 x + . . . + d k x k .Step 5. Finding m ∈ N such that u m ≡ p. For the definition of u m see Section 3. If p ≡ 0 then clearly m = 1. Otherwise let n i := p r (d i ) be the positions of the numbers d i in the sequence {r n }. Then m = p q ([n 0 ; n 1 + 1, . . . , n k + 1]) + 1. Evaluating the numbers c 0 , c 1 and s in (2.4). Set s := (1 -2m)α. If u m = p is constant then we put c 1 := 1, c 0 := d 0 -1 + h((2m + 1)α) 2 .