Radhia Bessi
email: radhia.bessi@enit.utm.tn

Lamsin

Approximation with activation functions and applications

Keywords: Function approximation, interpolation, Runge's phenomenon, Chebychev points, neural network, universal approximation theorem, numerical integration, finite element. Resumé Approximation d'une fonction, interpolation, phénomène de Runge, points de Chebychev, réseau neuronal, théorème universel d'approximation, intégation numérique, éléments finis

Function approximation arises in many branches of applied mathematics and computer science, in particular in numerical analysis, in finite element theory and more recently in data sciences domain. From most common approximation we cite, polynomial, Chebychev and Fourier series approximations.

In this work we establish some approximations of a continuous function by a series of activation functions. First, we deal with one and two dimensional cases. Then, we generalize the approximation to the multi dimensional case. Examples of applications of these approximations are: interpolation, numerical integration, finite element and neural network. Finally, we will present some numerical results of the examples above.

I INTRODUCTION

Motivated by the architecture of the human brain, neural networks are composed of multiple hidden layers. Each hidden layer has multiple hidden nodes. Each node is an activation function of an affine transformation of the outputs of the previous layer. Examples of activation functions:

• Heaviside function H : x → 1 if x ≥ 0 0 if x < 0 • Relu function : Relu : x → max(0, x).
• Quadratic Relu function: Relu 2 : x → 1 2 max(0, x) 2 . • Sigmoid function: σ : x → 1 1+e -x . • Softplus function : σ p : x → log(1 + e -x ).

The starting point of the study of neural networks theory was based on the universal approximation theorem. It was proved in [START_REF] Cybento | Approximation by Superpositions of a Sigmoidal Function[END_REF] the density of neural networks with one hidden layer in the space of continuous functions.

Later, it was proved in [START_REF] Leshno | Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function[END_REF] that multilayer feed-forward network with a locally bounded piecewise continuous activation function can approximate any continuous function if and only if the network's activation function is not polynomial.

The authors in [START_REF] Liu | Optimal Function Approximation with Relu Neural Networks[END_REF] proposed an algorithm to find the optimal approximations of convex univariate functions with feed-forward rectified linear unit (Relu) neural networks. They studied the minimal approximation error and the number of approximating linear pieces.

Artificial neural networks with Relu activation function were used in [START_REF] Petersen | Optimal approximation of piecewise smooth functions using deep ReLU neural networks[END_REF] to approximate discontinuous piecewise functions up to L 2 error. Optimal rates for approximating these piecewise functions by Relu neural networks, measuring the complexity of the networks in terms of the number of nonzero weights, were established.

An L ∞ and L 2 error bounds for functions of many variables that are approximated by linear combinations of Relu and squared Relu functions were established in [START_REF] Klusowski | Approximation by Combinations of ReLU and Squared ReLU Ridge Functions With 1 and 0 Controls[END_REF]. Approximation of continuous multi-variate functions with Deep Relu neural networks and conventional fullyconnected architectures was developed in [START_REF] Yarotsky | Optimal approximation of continuous functions by very deep ReLU networks[END_REF].

Authors in [START_REF] He | ReLU Deep Neural Networks and Linear Finite Elements[END_REF] looked into the relationship between deep neural networks with Relu function as the activation function and continuous piecewise linear functions. Also, they gave a one dimensional example to illustrate that a finite element method using Relu neural network can lead to better approximation result than adaptive finite element method.

Recently, there are increased new research interests for the application of neural network for numerical approximation of nonlinear a PDEs as in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF][START_REF] He | ReLU Deep Neural Networks and Linear Finite Elements[END_REF]. Many forms of interpolation can be constructed by picking a different class of interpolates like, Lagrange or Hermite polynomial interpolation [START_REF] Kress | Numerical Analysis[END_REF], or rational interpolation using Padé approximation [START_REF] Padé | Sur la représentation approchée d'une fonction par des fractions rationnelles[END_REF], or also, trigonometric interpolation, which is interpolation by trigonometric polynomials using Fourier series.

Numerical integration theory is in general based on interpolation functions. It provides a basic and important tool for the numerical resolution of other problems and can be applied, for example, in numerical methods for ordinary or partial differential equations and in finite element method.

In this work, we prove that a continuous function can be explicitly approximated by a neural network with two layers and non linear activation function in the hidden layer. This is a version of the universal approximation theorem in the case of single variate. The novelty here is that we precise weights of the network. But, we do not have linearity of the neural network for multidimensional case using Relu or heaviside as activation function. We prove the approximation for one and two dimensional cases and extend the result to the multi-variables continuous functions. We apply these approximations to interpolation, numerical integration and finite element for one and two-variate functions.

II APPROXIMATION OF UNIVARIATE FUNCTION

Without loss of generality, we consider a function f defined on the interval [0, 1]. Our goal here is to prove that f is a uniform limit of a series of the form

n∈I N α n ϕ(w n x + b n ), for different
types of activation functions ϕ. This means that f can be approximated by one hidden layer neural network with ϕ as activation function.

Approximation by Relu activation function

Proposition 1:

Let f : [0, 1] → IR a continuous function. Then, f is the uniform limit on [0, 1] of

S n (x) = n k=0 α k Relu((k + 1) -nx),
where the vector α = α 0 , α 1 , ..., α n T is given by α = A

R,n F n , for the matrix A

R,n of size (n + 1) |f (x)| and for a given ε > 0, there exists η > 0 such that for all

A (1) R,n =              1 -2 1 0 . . . . . . 0 0 1 - 2
x, x ∈ [0, 1] satisfying |x -x | ≤ η, we have |f (x) -f (x )| ≤ ε 2 . Fixing n ≥ n 0 = E( 1 η ) + 1, (where here E( 1 η ) is the integer part of the real 1 η ). Let x ∈ [0, 1]. If x = 1, then we have clearly S n (1) = α n = f (1) and hence |S n (1) -f (1)| = 0 ≤ ε. If x ∈ [0, 1[, then, there exists 0 ≤ k x ≤ n such that x ∈ [ kx n , kx+1] n ]. S n (x) = k≥kx α k Relu((k + 1) -nx) = n-2≥k≥kx f ( k n ) -2f ( k + 1 n ) + f ( k + 2 n ))(k + 1 -nx) + (f ( n-1 n ) -2f (1))(n -nx) + f (1)(n + 1 -nx) = f ( kx n )(k x + 1 -nx) -f ( kx+1 n )(k x -nx). Then |S n (x) -f (x)| = |(f ( k x + 1 n ) -f ( k x n ))(k x -nx) + f ( k x n ) -f (x)|. Since x ∈ [ kx n , kx+1] n ], then |k x -nx| ≤ 1. In addition | kx+1 n -kx n | = 1 n ≤ η, and | kx n -x| ≤ | kx+1 n -kx n | = 1 n ≤ η, it follows that |S n (x) -f (x)| ≤ |(f ( k x + 1 n ) -f ( k x n )| + |f ( k x n ) -f (x)| ≤ ε 2 + ε 2 = ε.
(S n ) converges uniformly to f on [0,1].

Remarks 1:

1. Coefficients (α k ) are such that : S n ( j n ) = f ( j n ) for all j = 0, ..., n. Since every activation function

ϕ k : x → Relu(k + 1 -nx) satisfies ϕ k ( j n ) = 0 if 1 ≤ k + 1 ≤ j ≤ n k + 1 -j if 0 ≤ j ≤ k + 1 ≤ n + 1 ,
then the vector (α k ) 0≤k≤n is solution to the triangular linear system B

(1)

R,n α = F n with B (1) R,n =              1 2 3 ... n n + 1 0 1 2 3 • • • (n -1) n . . . . . . . . . . . . . . . . . . . . . 0 . . . . . . 1 2 3 4 . . . • • • . . . . . . 1 2 3 . . . • • • • • • . . . . . . 1 2 0 • • • • • • • • • 0 0 1              .
It is easy to check that the matrix B

(1) R,n is nothing else but the inverse of the triangular matrix A

(1) R,n . This means that this Relu approximation is also an interpolation of f at (n + 1) distinct nodes x j = j n , j = 0, ..., n. 2. A continuous function f : [a, b] → IR, has similarly a Relu approximation of the form

f (x) = lim n→+∞ n k=0 α k Relu(k + 1 - n b -a (x -a)),
where the vector α = A

(1)

R,n F n , with A (1)
R,n is the same matrix 1 and

F n =       f (a) f (a + b-a n ) ... f (a + (n-1)(b-a) n ) f (b)       . 3.
Similarly, a Heaviside-approximation of a continuous function f is a uniform sum of Heaviside functions as follows :

f (x) = lim n→+∞ H n (x), with H n (x) = n-1 k=0 [f ( k n ) -f ( k + 1 n )]H(k + 1 -nx) + f (1)H(n + 1 -nx). Indeed, for x ∈ [ kx n , kx+1] n ], we easily have |H n (x) -f (x)| = |f ( kx n ) -f (x)|.
Using uniform continuity of f , we easily deduce uniform convergence of (H n ) to f . 4. The approximation of f by a-one hidden neural network contains a bias term. We modify slightly the last Relu approximation and we prove in a similar way that f is also a uniform limit of

T n (x) = n-1 k=0 β k Relu((k + 1) -nx) + β n ,
where the vector β = β 0 , β 1 , ..., β n T is given by β = A n F n , where

A n =              1 -2 1 0 . . . . . . 0 0 1 -2 1 . . . . . . . . . . . . 1 -2 1 . . . . . . . . . . . . . . . . . . . . . . . . 0 . . . . . . . . . . . . 1 -2 1 . . . . . . . . . . . . 1 -1 0 . . . . . . . . . . . . 0 1              . ( 2
)
A n is the inverse of

B n =              1 2 3 ... n 1 0 1 2 3 • • • (n -1) 1 . . . . . . . . . . . . . . . . . . . . . 0 . . . . . . 1 2 3 1 . . . • • • . . . . . . 1 2 1 . . . • • • • • • . . . . . . 1 1 0 • • • • • • • • • 0 0 1             
.

The bias here is β n and we retrieve the universal approximation theorem for the one dimensional case. 5. On the contrary of polynomial, cubic spline, regression or trigonometric Fourier approximation, last Relu or Heaviside interpolations are explicit and simple to compute for every integer n. Moreover, they are very easy to integrate and to differentiate.

Interpolation by Relu function: general case

As it was mentioned before, approximation by Relu function in proposition ( 1) is also an interpolation of f at the (n + 1) equidistant points (x j ) j=0,...,n of the interval [0, 1]. In general, given a data set (x i , y i ) i=0,...,n , with x 0 < x 1 < ... < x n and y i ∈ IR, we propose here to find a function g of the form

g n (x) = n-1 k=0 α k Relu(w k x + b k ) + α n satisfying g n (x i ) = y i .
Proposition 2: Let g n be the function defined by

g n (x) = n-1 i=0 α i Relu( x i+1 -x x i+1 -x i ) + α n ,
with the vector α = α 0 , α 1 , ..., α n T is given by α = D n F n , where the square matrix D n of size n + 1 and the vector F n are:

D n =           . . . . . . . . . . . . . . . 0 0 . . . 0 1 -x n-1 -x n-3
x n-1 -x n-2

x n-1 -x n-3

x n-1 -x n-2 -1 0 . . . . . . 0 1 -1 0 . . . . . . . . . 0 1          
and F n = y 0 , y 1 , ..., y n-1 , y n T . Then g n (x i ) = y i , for all i = 0, ..., n.

Proof. To determine g of the form

g n (x) = n-1 i=0 α i Relu( x i+1 -x x i+1 -x i ) + α n satisfying g n (x i ) = y i
, for all i = 0, ..., n, it is sufficient to look for (α i ) i=0,...,n solution to the following system :

                           α 0 + x 2 -x 0 x 2 -x 1 α 1 + ... + xn-x 0 xn-x n-1 α n-1 + α n = y 0 α 1 + x 3 -x 1 x 3 -x 2 α 2 + ... + xn-x 1 xn-x n-1 α n-1 + α n = y 1 . . . . . . . . . α i + x i+2 -x i x i+2 -x i-1 α i+1 + ... + xn-x i xn-x n-1 α n-1 + α n = y i . . . . . . α n-2 + xn-x n-2 xn-x n-1 α n-1 + α n = y n-2 α n-1 + α n = y n-1 α n = y n
The matrix of the last linear system is :

D n =             1 x 2 -x 0 x 2 -x 1 x 3 -x 0 x 3 -x 2 . . . . . . xn-x 0 xn-x n-1 1 0 1 x 3 -x 1 x 3 -x 2 x 4 -x 1 x 4 -x 3 . . . xn-x 1 xn-x n-1 1 . . . . . . . . . . . . . . . . . . . . . 0 . . . 0 1 x n-1 -x n-3 x n-1 -x n-2 xn-x n-3 xn-x n-1 1 0 . . . . . . 0 1 xn-x n-2 xn-x n-1 1 0 . . . . . . . . . 0 1 1 0 . . . . . . . . . 0 0 1            
This matrix is clearly non singular and its inverse is the band-upper triangular matrix D n .

Remark 1:

The proof of the last approximation shows the existence and the uniqueness of the coefficients

α i for w i = -1 x i+1 -x i and b i = x i+1
x i+1 -x i , but we do not have uniqueness of the weights w i and bias b i . If the points are equispaced, we find the approximation of type (1) which is similar to the one of proposition (1).

Applications

There are many applications of the two last Relu-approximation functions. Python programming language is used to code some of these applications presented in the rest of this work.

Interpolation

We can use this approximation in interpolation theory. It is known that for Runge function f (x) = 1 1+25x 2 for x ∈ [-1, 1], if we choose equidistant points on [-1, 1], we find Runge's phenomenon which is a problem of oscillation at the edges -1 and 1 of the interval that occurs when using polynomial interpolation with high degree. This is due to the fact that uniform convergence is not guaranteed, unlike approximations by Relu function. Our first numerical application for Chebychev and equidistant points will be tested on the interval [-1, 1], since most existing results for Runge phenomenen are implemented in this interval. For the remaining applications, we deal with f : [0, 1] → IR. Figures [START_REF] Cybento | Approximation by Superpositions of a Sigmoidal Function[END_REF] We notice that uniform convergence is faster with Relu-approximation than Heaviside approximation as (2) and (4) show:

Relu-approximation and quadrature rule

Since Relu functions ϕ k are easy to integrate, we can approximate

[0,1]
f (t)dt by replacing f by its Relu-approximation from proposition (1) to get the following integration quadrature rule:

1 0 f (x)dx 1 2n n-1 k=0 α k (k + 1) 2 + α n , (3)
We denote by

I n (f ) = 1 0 S n (x)dx = 1 2n n-1 k=0 α k (k + 1) 2 + α n .
Replacing α k by its expression and simplifying, we obtain

1 0 f (x)dx I n (f ) = 1 2n f (0) + 2f ( 1 n ) + 2f ( 2 n ) + ... + 2f ( k n ) + ... + 2f ( n -1 n ) + f (1) . (4)
We retrieve the composite trapezoidal rule for n equidistant intervals. Since (S n ) converges uniformly to f on [0, 1], then (I n (f )) converges to

1 0 f (x)dx.

Finite element application.

Without loss of generality, we can restrict our study to the domain ]0, 1[ and we will consider the following boundary-value problem:

Given f ∈ L 2 (]0, 1[) and c ∈ L ∞ (]0, 1[), c ≥ 0, find the function u solving: -u (x) + c(x)u(x) = f (x), ∀ x ∈]0, 1[ u(0) = u(1) = 0. (5)
It is known that ( 5) is optimality condition of the quadratic minimization problem:

min u∈H 1 0 (]0,1[ 1 2 1 0 u (x) 2 dx + 1 2 1 0 c(x)u(x) 2 dx - 1 0 f (x)u(x)dx, (6)
Clearly, the piecewise C 1 continuous functions (ϕ k : x → Relu((k + 1) -nx), k = 0, ..., n) are a linear independent family of the Sobolev space H 1 (]0, 1[). For a fixed n, let V n = Span(ϕ k ) k=0,...,n and

V 0 n = {u ∈ V n / u(0) = u(1) = 0}.
We suppose here that the solution u of ( 5) is continuous, then u = lim

n→+∞ n k=0 α k Relu(k + 1 - nx). The idea of finite element method is to compute u n = n k=0 α k Relu(k + 1 -nx) ∈ V 0 n an
approximation of u solution to the approximate minimization problem.

min un∈V 0 n 1 2 1 0 u n (x) 2 dx + 1 2 1 0 c(x)u n (x) 2 dx - 1 0 f (x)u n (x)dx, (7)
Since we have the Dirichlet conditions u n (0) = u n (1) = 0, then, α satisfies α 0 + 2α 1 + 3α 2 + ... + (n + 1)α n = 0 and α n = 0.

The distributional derivative of u n is

u n = n i=0 α i ϕ i (x) = -n n i=0 α i H(i + 1 -nx).
Replacing u n and u n in [START_REF] Yarotsky | Optimal approximation of continuous functions by very deep ReLU networks[END_REF], the approximation problem consists in solving the quadratic constrained problem:

min α∈I R n+1 1 2 (E n α, α) -(b n , α) α 0 + 2α 1 + ... + nα n-1 + (n + 1)α n = 0 α n = 0 , (8)
where (., .) designates the inner product, the matrix E n and the vector b n are given by :

E n = ]0,1[ ϕ i (x)ϕ j (x) + c(x)ϕ i (x)ϕ j (x)dx 0≤i,j≤n and b n = ]0,1[ f (x)ϕ i (x)dx 0≤i≤n .
Equality constraints raise from boundary conditions u(0) = 0 and u(1) = 0.

Example 1:

Consider the simple homogeneous Laplace problem on one dimensional space:

-u (x) = 1, ∀ x ∈]0, 1[ u(0) = u(1) = 0 (9)
Since we have, for i ≤ j,

1 0 ϕ i (x)ϕ j (x)dx = n 2 1 0 H(i + 1 -nx)H(j + 1 -nx)dx = n 2 i+1 n 0 dx = n(i + 1),
then, the associated symmetric matrix E n is given by

E n = n        1
       . ( 10
)
and the vector b n = ((i + 1) 2 /2n) 0≤i≤n .

The symmetric matrix E n is invertible and its inverse is the classical matrix :

E -1 n = 1 n            2 -1 0 . . . . . . . . . 0 -1 2 -1 0 . . . . . . 0 0 -1 2 -1 0 . . . 0 . . . . . . . . . . . . . . . . . . . . . 0 . . . . . . -1 2 -1 0 0 . . . . . . 0 -1 2 -1 0 . . . . . . . . . 0 -1 1           
.

Moreover, clearly E -1 n is positive definite, then E n is also positive definite. Therefore, the quadratic problem (8) has a unique solution α.

In this work we do not study convergence of this approximation method. However, we compare numerical results for n = 10 and n = 50 with the exact solution u(x) = 1 2 (x -x 2 ). Since we know explicitly the inverse of the matrix E n , resolution of quadratic problem (8) using Karush Kuhn Tucker condition gives the exact solution to this example. Numerical results for n = 10 are good with approximation error u -u 10 ∞ = max

Neural network with one hidden layer

Proposition (1) and remark (1) are nothing else than a proof of the universal approximation theorem for univariate function. It proves the existence of a neural network of one hidden layer with Relu or Heaviside activation function that approximates a continuous function f . But, in practice, we just know values (y i ) of f on some data (x i ). Interpolation of type detailed in proposition (2) can be applied under the condition on training set satisfying x i = x j , for i = j. Loss error of the neural network is zero, which leads to overfitting if we test it on a data

x test / ∈ [min i x i , max i x i ].
This makes this type of approximation not beneficial for neural network regression or classification problems.

Quadratic Relu approximation

We suppose here that f is C 1 ([0, 1]). We consider the Relu-approximation of the continuous f of the form

f (x) = lim n→+∞ n k=0 α k Relu(k + 1 -nx)
where the vector α = A

(1)

R,n G (1)
n , where

A (1)
R,n is the matrix (1) and

G (1) n = f (0), f ( 1 n ), ..., f ( n-1 n ), f (1)
T .

We obtain the following Relu 2 approximation of f .

Proposition 3:

The function f is a uniform limit of (S n ), where

S n (x) = - 1 2n n k=0 α k Relu(k+1-nx) 2 +f (1)+ α n 2n = - 1 n n k=0 α k Relu 2 (k+1-nx)+f (1)+ α n 2n . Proof. It is clear that (S n ) is differentiable on [0, 1] and S n (x) = n k=0 α k Relu(k + 1 -nx), ∀ x ∈ [0, 1].
Then, according to proposition ( 1

III THE TWO DIMENSIONAL CASE

We study next approximation results for the case of a two-variables function f : [0, 1] 2 → IR, x = (x 1 , x 2 ) → f (x). We prove that f can be approximated by a sum of separable variables Heaviside or Relu functions.

For this end, we denote by A B the Kronecker product of two matrices A and B.

Relu-approximation

We study now the following Relu-approximation of f . Proposition 4:

For x = (x 1 , x 2 ) ∈ [0, 1] 2 and m = (n + 1) 2 , let S n (x) = n k,k =0 α k,k Relu(k + 1 -nx 1 ) Relu(k + 1 -nx 2 ),
where the vector

α = α 0 , α 1 , ..., α m T = A (2) R,n F (2)
n . The matrix

A (2)
R,n of size m = (n + 1) 2 is given by

A (2) R,n =               A (1) R,n -2A (1) R,n A (1) R,n O . . . . . . O O A (1) R,n -2A (1) R,n A (1) R,n . . . . . . O O A (1) R,n -2A (
R,n -2A (1) R,n A (1) R,n O . . . . . . O A (1) R,n -2A (1) R,n O . . . . . . . . . . . . O A (1) R,n 
             = A (1) R,n A (1) R,n , with O is null matrix, A (1)
R,n is the (n + 1) square matrix of type (1) and the vector F

n ∈ IR m defined by:

∀ 0 ≤ k, k ≤ n, F (2) n ((n + 1)k + k + 1) = f ( k n , k n ).
Then, the sequence (S n ) converges uniformly to f on [0, 1] 2 .

Proof. The function f is continuous on the compact domain [0, 1] 2 , then it is bounded and uniformly continuous. We denote by M = sup x∈[0,1] 2 |f (x)| and for a given ε > 0, there exists η > 0 such that for all x, x ∈ [0, 1] 2 satisfying x -x ∞ ≤ η, we have

|f (x) -f (x )| ≤ ε 4 . Fixing n ≥ n 0 = E( 1 η ) + 1 and x = (x 1 , x 2 ) ∈ [0, 1] 2 . If x 1 = 1 or x 2 = 1, we prove, similarly to the one dimensional case, that |S n (x) -f (x)| ≤ ε. If x ∈ [0, 1[ 2 , then, there exist 0 ≤ k x ≤ n and 0 ≤ k x ≤ n such that : x ∈ [ kx n , kx+1] n ] × [ k x n , k x +1] n ] and S n (x) = n k =k x n k=kx α k,k (k + 1 -nx 1 )(k + 1 -nx 2 ).
By replacing α k,k and simplifying, we get

S n (x) = f ( kx n , k x n )(k x + 1 -nx 1 )(k x + 1 -nx 2 ) -f ( kx+1 n , k x n )(k x -nx 1 )(k x + 1 -nx 2 ) -f ( kx n , k x +1 n )(k x + 1 -nx 1 )(k x -nx 2 ) + f ( kx+1 n , k x +1 n )(k x -nx 1 )(k x -nx 2 )
.

or

S n (x) = f ( kx n , k x n ) -f ( kx+1 n , k x n ) (k x -nx 1 )(k x + 1 -nx 2 ) + f ( kx n , k x n )(1 + k x -nx 2 ) + f ( kx+1 n , k x +1 n ) -f ( kx n , k x +1 n ) (k x -nx 1 )(k x -nx 2 ) -f ( kx n , k x +1 n )(k x -nx 2 )
.

Then S n (x) -f (x) = f ( kx n , k x n ) -f ( kx+1 n , k x n ) (k x -nx 1 )(k x + 1 -nx 2 ) + f ( kx+1 n , k x +1 n ) -f ( kx n , k x +1 n ) (k x -nx 1 )(k x -nx 2 ) f ( kx n , k x n ) -f ( kx n , k x +1 n ) (k x -nx 2 ) + f ( kx n , k x n ) -f (x)
.

Using uniform continuity, and since, x-

( kx n , k x n ) ∞ ≤ 1 n ≤ η, and (k x -nx 2 ), (k x -nx 1 ), (k x + 1 -nx 2 ) and (k x + 1 -nx 1 ) are bounded by 1, then |S n (x) -f (x)| ≤ ε 4 + ε 4 + ε 4 + ε 4 = ε.
Uniform convergence then holds.

Remark 2:

We prove later in (5) that f is also uniform limit of

n k,k =0 α k,k H(k + 1 -nx 1 )H(k + 1 -nx 2 ),
where the vector α = α 0 , α 1 , ..., α m T = A

(2)

H,n F (2 n . The matrix A (2)
H,n of size m = (n + 1) 2 is given by

A (2) H,n =               A (1) H,n -A (1) H,n O O . . . . . . O O A (1) H,n -A (1)
H,n O . . . . . . O O A (1) H,n -A (1)
              = A (1) H,n A (1) H,n
for the (n + 1) square matrices, O null matrix and

A (1) H,n =              1
             . ( 11
)
And the vector F

n ∈ IR m is defined by:

∀ 0 ≤ k, k ≤ n, F (2) n ((n + 1)k + k + 1) = f ( k n , k n ).

Applications

As for the one dimensional case, we consider some applications presented to interpolation, numerical integration and finite element.

Interpolation

It is easy to check that, for fixed n, the approximated function S n of a given continuous function f on [0, 1] 2 cited in proposition (4) or in remark ( 2) is such that :

S n ( k n , k n ) = f ( k n , k n ), ∀ 0 ≤ k, k ≤ n.
In fact, coefficients α k,k are defined in a way that interpolation conditions are satisfied. This is equivalent to solving the linear system : B

(2)

n α = F (2)
n , with α and F

n are the same vectors in proposition (4) and remark [START_REF] Cybento | Approximation by Superpositions of a Sigmoidal Function[END_REF]. The square matrix of size m = (n + 1) 2 is

B (2) n = B (2)
H,n for Heaviside approximation B

R,n for Relu approximation with B

(2)

H,n , B (2)
R,n are the following upper triangular matrices :

B (2) H,n =               B (1)
H,n B

(1)

H,n B (1)
H,n . . . . . . . . . B

(1) H,n O B

(1)

H,n B (1)
H,n B

(1)

H,n • • • B (1) H,n O O B (1)
H,n B (1)
H,n B (1)
              = B (1) H,n B (1)
H,n , where:

B (1) H,n =              1
             . B (2) R,n =               B (1) R,n 2B (1) R,n 3B (1) R,n . . . . . . . . . (n + 1)B (1) R,n O B (1) R,n 2B (1) R,n 3B (1) R,n • • • nB (1) R,n O O B (1) R,n 2B (1)
              = B (1) R,n B (1)
R,n , and:

B (1) R,n =              1
            
It is easy to check, in both cases, that B

n is the inverse of A

n which confirms the interpolation result. As example, for the parabolic function f (x 1 , x 2 ) = (x 1 -0.5) 2 + (x 2 -0.5) 2 , we present in figure [START_REF] Yarotsky | Optimal approximation of continuous functions by very deep ReLU networks[END_REF] the exact function and its Relu interpolation for n = 20.

Finite element application

We denote by Ω =]0, 1[ 2 . Given f ∈ L 2 (Ω), find the function u solving:

-∆u(x) = f (x), ∀ x ∈ Ω, u = 0 on ∂Ω, (12)
u solution ( 12) is also solution to the quadratic optimization problem u = arg min

v∈H 1 0 (Ω) 1 2 Ω ∇v(x) 2 dx - Ω f (x)v(x)dx.
Let m = (n + 1) 2 , and ϕ : x → ϕ k (x) = Relu(k + 1 -nx). In order to enumerate the set (ϕ k ϕ k ) k,k =0,...,n of the Sobolev space H 1 (Ω), we consider the family (ψ) i=1,...,m = (ϕ k ϕ k ) k,k =0,...,n . We denote by V n = Span(ϕ k,k ) k,k =0,...,n = Span(ψ i ) i=1,...,m and

V 0 n = {v ∈ V n ; v = 0 on ∂Ω}.
We consider then the approximate quadratic optimization problem

u n = arg min vn∈V 0 n 1 2 Ω ∇v n (x) 2 dx - Ω f (x)v n (x)dx.
If we write v n = m i=1 α i ψ i , the last problem becomes equivalent to the quadratic constrained problem:

min α∈I R m 1 2 (E (2) n α, α) -(b (2) n , α), u n = m i=1 α i ψ i = 0 on ∂Ω, (13)
where the matrix E [START_REF] Klusowski | Approximation by Combinations of ReLU and Squared ReLU Ridge Functions With 1 and 0 Controls[END_REF], and the vector b

n = ( Ω ψ i (x)ψ j (x)dx) i,j=1,...,m ) = E n E n , for E n given in (2)
(2) n = ( Ω f (x)ψ i (x)dx) i=1,...,m .
The matrices A

H,n and B

(1)

H,n are the same of those of the bi-dimensional case in section (III).

A (d) H,n =               A (d-
H,n is of the form (11).

In order to prove the last proposition we establish the following general property.

Proof.

Without loss of generality and in order to simplify notations, we prove the lemma for k = 1.

The proof is by induction on d. If d = 1, we have easily

p i=1 α i = n i=1 (A (1)
H,n F ) i = n i=k (F i -F i+1 ) + F n+1 = F 1 .
We suppose that (15) is true for d ≥ 1, we wish to prove that is true for d + 1 ?

Let α = A

VI CONCLUSION

In this paper, we have considered the following approximation problem : given a continuous function f , we proved that f is a uniform limit of a sum of Relu or heaviside activation functions with separable variables. Numerical applications of these approximations for one and two dimensional cases were implemented.

As future work, we plan to extend these approximations using other activation functions and try to apply them to machine learning models like regression or deep learning models.

Figure 1 :

 1 Figure 1: Runge phenomenen: [Wikipedia].

  and (3) display numerical results of Runge function interpolation, first over equidistant points, then over Chebychev points. We present in green color interpolation points (x i ) i=0,...,n .For the same function f , we also test Heaviside approximation whith equidistant points for n = 10 and n = 50. Results are displayed in figure[START_REF] Leshno | Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function[END_REF].

Figure 2 :

 2 Figure 2: Relu interpolation with equidistant points.

Figure 3 :

 3 Figure 3: Relu interpolation with Chebychev points.

Figure 4 :

 4 Figure 4: Heaviside interpolation with equidistant points.

  They are better for n = 50 with approximation error u -u 50 ∞ = 0.0003633180406232603.

Figure 5 :

 5 Figure 5: Relu approximation for finite element method.

  ), (S n ) converges uniformly to f on [0, 1]. Moreover, S n (1) = -αn 2n +f (1)+ αn 2n = f (1) converges to f (1). The uniform convergence of the sequence of derivatives plus the convergence of the sequence of functions at x = 1 imply uniform convergence of (S n ) to f . The Relu 2 approximation interpolates f at ( i n ) i=0,...,n and f just at the point 1. In figure (6), we have used Relu 2 approximation for the classical Runge function f (x) = 1 25(2x-1) 2 +1 , for x ∈ [0, 1] and n = 20.

Figure 6 :

 6 Figure 6: Relu 2 approximation for n=20

  . . . . . . . . . . . . 0 1

  2 3 . . . . . . . . . . . . (n . . . . . . . . . . 0 1

Figure 7 : 1 (

 71 Figure 7: Relu approximation for n=20

Figure 8 :

 8 Figure 8: Relu approximation for n=20

Lemma 1 :

 1 Let d ∈ I N * , p = n + 1 and F = (F 1 , F 2 , ..., F p d ) ∈ IR p d . If α = A (d) H,n F = (α 1 , ..., α p d ), then for all k ≥ 1, we have p d i=k α i = F k .

  F ∈ IR p d+1 , with the block decomposition of α and F : α = (α 1 , α 2 , ..., α p ) andF = (F 1 , F 2 , ..., F p ) ∈ IR p d+1 , α k , F k ∈ IR p d , for k = 1, ..., p. Then H,n F k ) k -H,n F p ) k

Figure 10 :Figure 11 :

 1011 Figure 10: Softplus approximation for d=1 and n=10

  Proof. Since f is continuous on the closed bounded interval, then it is bounded and uniformly continuous. Let M = sup

				1	. . .		. . .
	. . .	. . . 1	-2 1	. . .	. . .
	. . .	. . .	. . .	. . .	. . .	. . . 0
	. . .	. . .	. . .	. . . 1 -2 1
	. . .		. . .	. . .	. . . 1	-2
	0 . . . . . . . . . . . . 0	1

∈[0,1]

  1 1 . . . . . . . . . . . . 1

	0 1 1 1	. . .		1
	0 0 1 1	. . .	. . .	. . .
	. . .	. . .	. . .	. . .	. . .	. . .	. . .
	0	. . .	. . . 0	1 1 1
	0		. . .	. . .	0 1 1
	0 . . . . . . . . .	. . . 0 1

x

-x 0 x 2 -x 1 x 2 -x 0 x 2 -x 1 -1 0 . . . 0 0 1 -x

-x 1 x 3 -x 2 x 3 -x 1 x 3 -x 2 -1 . . . . . .

For the particular example

the exact solution is u(x 1 , x 2 ) = 1 4 x 1 (1 -x 1 )x 2 (1 -x 2 ) and figure [START_REF] He | ReLU Deep Neural Networks and Linear Finite Elements[END_REF] shows numerical solution using Relu approximations for n = 20 for which u -u n ∞ = 0.003246376497983063. To solve the minimizing quadratic problem (13), we used the function 'solvers.qp' of the python software package 'cvxopt'.

Heaviside -approximation

Proposition 5:

The function f is a uniform limit of

H,n is the m = (n + 1) d square matrix given by recurrence as

H,n .

The matrix

H,n is the inverse of the interpolation matrix

H,n which is of the form

Using induction hypothesis, we have

Proof. (of proposition ( 5))

The proof is analogous to the one of the case d = 1. Let ε > 0 and η > 0, such that, for

Replacing S n (x) and applying lemma (1), we deduce

Using uniform continuity, and since, x -

This finishes the proof of the proposition (5).

Relu -approximation

We can prove similarly that a continuous function f on the compact domain [0, 1] d is a uniform limit of

n is the m = (n + 1) d square matrix given by recurrence as

R,n and B

(1) R,n are the same matrices used before in section (III).

V OTHER ACTIVATION FUNCTIONS

If we replace the Relu function by another activation function, like sigmoid or softplus, determining the coefficients α is not explicit and we need to solve a linear system for every choice of n. Numerical results are similar to Relu approximation case. This is an example for Runge function for d = 1 with softplus function then for d = 2 with sigmoid activation function: