Noise Benefits in Combined Nonlinear Bayesian Estimators

Fabing Duan ,Y anPan , François Chapeau-Blondeau , and Derek Abbott , Fellow, IEEE Abstract-This paper investigates the benefits of intentionally adding noise to a Bayesian estimator, which comprises a linear combination of a number of individual Bayesian estimators that are perturbed by mutually independent noise sources and multiplied by a set of adjustable weighting coefficients. We prove that the Bayes risk for the mean square error (MSE) criterion is minimized when the same optimum weighting coefficients are assigned to the identical estimators in the combiner. This property leads to a simplified analysis of the noise benefit to the MSE of the combined Bayesian estimator even when the number of individual estimators tends to infinity. It is shown that, for a sufficiently large number of individual estimators, the MSE of the designed Bayesian estimator approaches a plateau for a wide range of added noise levels. This robust feature facilitates the incorporation of the added noise into the design of Bayesian estimators without tuning the noise level. For an easily implementable Bayesian estimator composed of quantizers, the benefit of the symmetric scale-family noise is demonstrated, and the optimal noise probability density function is approximated by solving a constrained nonlinear optimization problem. We further extend this potential Bayesian estimator to the nonlinear filter design. Finally, examples of the noise benefits in random parameter estimation and nonlinear filtering support the theoretical analyses. Index Terms-Noise benefit, Bayesian estimator, linear combination, nonlinear filtering, stochastic resonance.

I. INTRODUCTION

I T IS well known that the closed form description of an optimal Bayesian estimator is difficult to achieve in general [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF]. For instance, the implementation of a minimum mean square error (MMSE) estimator requires the solution to the mean of the posterior probability density function (PDF) of the observation via difficult integrals [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF]. Thus, in practice, it is reasonable to seek some suboptimal but feasible nonlinear Bayesian F. Duan is with the Institute of Complexity Science, Qingdao University, Qingdao 266071, China (e-mail: fabing.duan@gmail.com).

Y. Pan is with the College of Mathematics and System Science, Shandong University of Science and Technology, Qingdao 266590, China (e-mail: yanpn.pan@gmail.com).

F. Chapeau-Blondeau is with the Laboratoire Angevin de Recherche en Ingénierie des Systèmes, Université d'Angers, 49000 Angers, France (e-mail: chapeau@univ-angers.fr).

D. Abbott is with the Centre for Biomedical Engineering and the School of Electrical and Electronic Engineering, The University of Adelaide, Adelaide, SA 5005, Australia (e-mail: derek.abbott@adelaide.edu.au).

estimators with tractable forms to estimate random parameters or variables [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF]- [START_REF] Fang | Nonlinear Bayesian estimation: From Kalman filtering to a broader horizon[END_REF].

Recently, the noise benefit in nonlinear estimators [START_REF] Rousseau | Noise-improved Bayesian estimation with arrays of one-bit quantizers[END_REF]- [START_REF] Zeng | Securing downlink massive MIMO-NOMA networks with artificial noise[END_REF] and detectors [START_REF] Patel | Optimal noise benefits in Neyman-Pearson and inequality-constrained statistical signal detection[END_REF]- [START_REF] Harmer | A review of stochastic resonance: Circuits and measurement[END_REF] has attracted great attentions of researchers in the field of signal processing, because the accuracy of an estimator and the detectability of a detector can be enhanced by design via intentionally adding noise. Sufficient or necessary conditions have been derived for the existence of the optimal added noise PDF [START_REF] Chen | Noise enhanced parameter estimation[END_REF]- [START_REF] Patel | Optimal mean-square noise benefits in quantizerarray linear estimation[END_REF], [START_REF] Chen | Noise-enhanced information systems[END_REF], [START_REF] Chen | Theory of the stochastic resonance effect in signal detection: Part I-Fixed detectors[END_REF]- [START_REF] Soganci | Optimal signal design for multiparameter estimation problems[END_REF], and the explicit or approximate forms of optimal added noise PDFs [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF]- [START_REF] Zhai | Noise effect on signal quantization in an array of binary quantizers[END_REF], [START_REF] Soganci | Optimal stochastic parameter design for estimation problems[END_REF], [START_REF] Chen | Noise-enhanced information systems[END_REF] have also been of great interest. Among these investigations, it was found that a parallel array of estimators can benefit from mutually independent added noise components in comparison to a single estimator [START_REF] Rousseau | Noise-improved Bayesian estimation with arrays of one-bit quantizers[END_REF], [START_REF] Chapeau-Blondeau | Noise-enhanced Fisher information in parallel arrays of sensors with saturation[END_REF], [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF]- [START_REF] Zhai | Noise effect on signal quantization in an array of binary quantizers[END_REF]. From the parameter estimation standpoint, Uhlich [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF] proposed a novel noise-enhanced estimator by averaging estimates from the same observation added by artificial noise components, and discussed its superiority over the original estimator and the noise-modified estimator derived by Chen et al. [START_REF] Chen | Noise enhanced parameter estimation[END_REF]. Based on the sum of outputs of quantizer arrays, the linear Wiener decoding scheme [START_REF] Mcdonnell | Stochastic Resonance: From Suprathreshold Stochastic Resonance to Stochastic Signal Quantization[END_REF] and the linear MMSE estimation [START_REF] Patel | Optimal mean-square noise benefits in quantizerarray linear estimation[END_REF] of the random inputs were extensively investigated within the framework of suprathreshold stochastic resonance [START_REF] Stocks | Suprathreshold stochastic resonance in multilevel threshold systems[END_REF]. We also used the least-square regression algorithm to numerically study the noise benefit in a quantizer array with optimal weights in comparison with the unweighted array [START_REF] Xu | Decoding suprathreshold stochastic resonance with optimal weights[END_REF].

In this paper, we design a linear combination illustrated in Fig. 1 as a potential noise-enhanced Bayesian estimator θLC , which consists of two modules: Module 1 exploits the benefit of noise by adding mutually independent noise components η i into each estimator θi , and module 2 outputs the linear MMSE estimation based on a set of estimates { θi } from module 1 multiplied by optimally tuned weighting coefficients w i for i =1, 2,...,m and a bias weighting w 0 . Then, the collective responses of all individual estimators yield the combined Bayesian estimator output θLC , as shown in Fig. 1. We first prove that, with the Bayes risk for the MSE criterion and at a given added noise level, the MSE of θLC gradually reduces as the number m of individual estimators increases, leading to the minimum MSE achieved by the combined estimator θLC in the limit of an infinite number. It is also proved that any two identical estimators in a combiner require the same optimum weighting coefficients. This characteristic simplifies theoretical analyses of the convergence of the MSE of θLC with no requirement for matrix inversions, and makes it possible to recognize the noise benefit of the added noise in the limiting case of an infinite number of individual estimators. It is interesting to note that the noise-enhanced estimator θNE of Ref. [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF] is just a special case of the combined estimator θLC with same fixed weighting coefficients, while the linear MMSE estimation [START_REF] Patel | Optimal mean-square noise benefits in quantizerarray linear estimation[END_REF], [START_REF] Mcdonnell | Stochastic Resonance: From Suprathreshold Stochastic Resonance to Stochastic Signal Quantization[END_REF] corresponds to the estimator θLC with a combination of homogeneous quantizers. It is illustratively shown that, for a large enough number of individual estimators, the MSE of θLC can be reduced to a minimum for an optimal added noise level, i.e. the stochastic resonance effect [START_REF] Stocks | Suprathreshold stochastic resonance in multilevel threshold systems[END_REF], [START_REF] Benzi | The mechanism of stochastic resonance[END_REF]. When the response of an original estimator θ has an upper bound, it is demonstrated that, for a very large number of individual estimators, the MSE of the combined estimator θLC approaches a plateau for a wide range of large added noise levels. If this plateau consists of 'local' minima of the MSE that are tolerable levels for practical applications, the noise-enhanced estimator θLC can be operated over a wide range of added noise levels. This robust feature suggests that, as the background noise varies, the designed estimator might be improved by the added noise without tuning levels in practice. For a special combined Bayesian estimator θLC with quantizers in number m>1,w ea l s op r o v et h a t θLC can always benefit from any type of the zero-mean symmetric scale-family added noise, because the MSE initially decreases by adding a small amount of noise. Moreover, using constrained nonlinear optimization methods, the optimal noise PDF is also approximately solved and the obtained MSE is effectively reduced in the considered cases. The designed Bayesian estimator can be also extended to nonlinear filtering with the multiple samples of observations, and some examples demonstrate the applicability of noise benefits in the proposed Bayesian estimators. The obtained novel results indicate that the added noise can be incorporated into the design of nonlinear Bayesian estimation and nonlinear filtering as a potential technique to enhance the accuracy of estimators [START_REF] Rousseau | Noise-improved Bayesian estimation with arrays of one-bit quantizers[END_REF]- [START_REF] Zeng | Securing downlink massive MIMO-NOMA networks with artificial noise[END_REF].

II. PARAMETER MODEL AND BAYESIAN ESTIMATOR

We observe the scalar data model as

x = θ + ξ, (1)
where the parameter θ is a random variable with the prior PDF f θ , and the background white noise ξ, independent of θ, has the PDF f ξ . Then, the observation x accords with the convolved PDF f x (x)= f θ (θ)f ξ (x -θ)dθ.A si nF i g .1 , the same observation x perturbed by the independent noise component η i is operated by each estimator θi (x + η i ) for i =1, 2,...,m, and here m denotes the total number of individual estimators in the combination. Letting the estimate vector θ =[ θ1 (x + η 1 ), θ2 (x + η 2 ),..., θm (x + η m )] ⊤ and the weight vector w =[w 1 ,w 2 ,...,w m ] ⊤ , we design a new Bayesian estimator θLC (x)=w 0 + w ⊤ θ as an unbiased estimation of the parameter θ. Here, w 0 is the bias weight. Then, the unbiased condition of expectations E x,η ( θLC )=E θ (θ) yields the bias weight w 0 =E θ (θ)w ⊤ E x,η ( θ), and then the designed estimator θLC can be written as

θLC (x)=E θ (θ)+w ⊤ θ -E x,η ( θ) , (2)
where E x,η (•) denotes the expectation with respect to the joint PDF of variables x and η and E θ (•) denotes the expectation with respect to the PDF of variable θ. Define ε = θ -θLC as the error of the estimator for a particular sample of x, then the Bayes risk of θLC for the MSE criterion is given by

R m =E x,η (ε 2 )=v ar(θ) -2w ⊤ p + w ⊤ Cw, (3)
where the subscript m is used to denote the number of estimators θi and the variance of θ is var(θ)=E θ (θ 2 ) -E 2 θ (θ). The centralized cross-correlation vector between the parameter θ and the estimate vector θ is p =E x,η θ -E θ (θ) θ -E x,η ( θ) with its elements calculated as

p i =E x,η θ -E θ (θ) θi -E x,η ( θi ) =E x {θE η [ θi (x + η)]}-E θ (θ)E x {E η [ θi (x + η)]}. (4)
The covariance matrix of the estimate vector θ is a symmetric positive definite matrix

C =E x,η θ -E x,η ( θ) θ - E x,η ( θ)
⊤ with its diagonal elements

C ii =E x,η θi -E x,η ( θi ) 2 =E x {E η [ θ2 i (x + η)]}-E 2 x {E η [ θi (x + η)]} (5)
and the non-diagonal elements

C ij =E x,η θi -E x,η ( θi ) θj -E x,η ( θj ) =E x {E η [ θi (x + η)]E η [ θj (x + η)]} -E x {E η [ θi (x + η)]}E x {E η [ θj (x + η)]} (6)
for i, j =1, 2,...,m (i = j). Note that the positive definite property of C requires the individual estimator in the combination not being such degenerate cases as θi = κ for an arbitrary constant κ.

Setting the gradient ∂R m /∂w = -2p +2Cw to zero, we have the optimum weight vector

w o = C -1 p. ( 7
)
Substituting w o of Eq. ( 7) into Eqs. ( 2) and (3), we have the final form of the designed estimator as

θLC (x)=E θ (θ)+p ⊤ C -1 θ -E x,η ( θ) , (8)
and the optimized MSE with respect to the weight vector is

R o m =v ar(θ) -w o⊤ Cw o =v ar(θ) -p ⊤ C -1 p. ( 9
)
Theorem 1: When the covariance matrix C is positivedefinite, the MSE R o m in Eq. ( 9) is a monotonically decreasing function of the number m, i.e.

R o m < R o m-1 , (10)
for a given circumstance of the background noise ξ and the added noise η.

Proof of Theorem 1 is presented in Appendix A. This theorem implies that, for a given circumstance of the background noise ξ and the added noise η, the minimum MSE R o ∞ is achieved by the Bayesian estimator θLC in the limiting case of m →∞. Moreover, the individual estimator θi can differ from each other.

Theorem 2: If two estimators θi = θj (i = j) in a combiner are identical, then optimum weighting coefficients w o i = w o j . Proof of Theorem 2 is given in Appendix B. This theorem suggests that, when a combiner has L groups and each group has m l identical estimators θl,i = θl for l =1, 2,...,L and i =1, 2,...,m l , then each group has m l identical weighting coefficients w l,i = w l . Updating the weight vector as w = [w 1 ,w 2 ,...,w L ] ⊤ and rewriting the estimate vector as θ = [ θ1 , θ2 ,..., θL ] with θl = m l i=1 θl (x + η i ), we then obtain the combined Bayesian estimator of Eq. ( 8) and its MSE of Eq. [START_REF] Mcdonnell | Stochastic Resonance: From Suprathreshold Stochastic Resonance to Stochastic Signal Quantization[END_REF].

Corollary 1: For a combiner composed of m identical original estimators θi = θ, the Bayesian estimator θLC in Eq. ( 8) becomes

θLC (x)=E θ (θ)+w o m i=1 θ(x + η i ) -E x {E η [ θ(x + η)]} , (11)
where the optimum weighting coefficient is

w o = p i /[C ii +(m -1)C ij ]. (12)
The corresponding MSE R o m of Eq. ( 9) is now simplified as

R o m =v ar(θ) -mp 2 i /[C ii +(m -1)C ij ] (13)
with the limit

R o ∞ = lim m→∞ R o m =v ar(θ) -p 2 i /C ij . (14)
Proof of Corollary 1 is presented in Appendix C, where p i , C ii and C ij , as special cases of Eqs. ( 4)- [START_REF] Chen | Noise enhanced parameter estimation[END_REF], are given in Eqs. (37)- [START_REF] Duda | Pattern Classification[END_REF]. Corollary 1 indicates that, for a combiner of m identical original estimators, the combined estimator of Eq. ( 8) with optimally weighting before summation and the estimator of Eq. [START_REF] Xu | Adaptive recursive algorithm for optimal weighted suprathreshold stochastic resonance[END_REF] with optimally weighting after summation can achieve the same MSE R o m of Eq. [START_REF] Zhai | Noise effect on signal quantization in an array of binary quantizers[END_REF]. Moreover, compared with the general expression of R o m in Eq. ( 9), Eq. ( 13) results in considerable simplification of the computation of the MSE R o m with no requirement for matrix inversions in the considered case of m identical estimators θi = θ.

Corollary 2: For a combination of identical estimators θi = θ in the limit case of m →∞, the designed Bayesian estimator θLC in Eq. ( 8) evolves into

θLC (x)=E θ (θ)+w o θNE (x) -E x [ θNE (x)] , (15)
where θNE (x) is the noise-enhanced estimator

θNE (x)=E η [ θ(x + η)] (16)
defined by Uhlich [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF] and the optimum weight is given by

w o = E x θ -E θ (θ) θNE (x) -E x ( θNE ) E x θNE (x) -E x ( θNE ) 2 . ( 17
)
The designed estimator θLC of Eq. ( 15) achieves the MSE R o ∞ of Eq. ( 14), and is never worse than the noise-enhanced estimator θNE of Eq. ( 16).

Corollary 3: The MSE R o
∞ of the estimator θLC in Eq. ( 15) satisfies the inequality

R o ∞ ≥R ms =E x θ -θmmse 2 , ( 18
)
where R ms is achieved by the MMSE estimator [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF] θmmse

(x)=E θ|x (θ|x)= θf θ|x (θ|x)dθ (19)
with the conditional posterior PDF

f θ|x (θ|x)=f θ (θ)f ξ (x - θ)/ f θ (θ)f ξ (x -θ)dθ.
The equality of Eq. ( 18) occurs when

θmmse (x) -E θ (θ)=κ[ θNE (x) -E x ( θNE )],
where κ is an arbitrary constant. Proofs of Corollaries 2 and 3 are presented in Appendices D and E, respectively, which are also illustrated in the following examples.

Example 1: Consider an uniformly distributed parameter θ buried in the Gaussian white noise ξ [1], [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF]. The prior PDF of θ is f θ (x)=1/a for 0 ≤ x ≤ a and otherwise zero, and ξ has the PDF f ξ (x)=exp(-x 2 /2σ 2 ξ )/ 2πσ 2 ξ with zero-mean and variance σ 2 ξ . Then, the maximum a posteriori (MAP) estimator is given by [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF], [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF]

θmap (x)= ⎧ ⎪ ⎨ ⎪ ⎩ 0,x < 0, x, 0 ≤ x ≤ a, a, x > a, (20)
and the MMSE estimator of Eq. ( 19) becomes [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF], [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF] θmmse

(x)=x + σ ξ 2 π e -x 2 2σ 2 ξ -e - (x-a) 2 2σ 2 ξ erf x √ 2σ ξ -erf x-a √ 2σ ξ . (21)
For the interval bound a =2and the background noise level σ ξ =1, it is seen in Fig. 2(a) that the MSE of the MAP estimator θmap of Eq. ( 20) is 0.4832 (magenta dashed line), much higher than the MMSE 0.2492 (red solid line) achieved by the MMSE estimator θmmse of Eq. [START_REF] Zozor | Stochastic resonance in locally optimal detectors[END_REF]. Substituting this suboptimal estimator θmap of Eq. ( 20) into Eq. ( 11) and Eq. ( 16), we can construct the combined estimator θLC and the noise-enhanced estimator θNE . The added noise η is chosen as Gaussian white noise with PDF f η (x) = exp(-x 2 /2σ 2 η )/ 2πσ 2 η and the standard versus the added noise level σ η . For comparison, the MSEs of the MAP estimator θmap in Eq. ( 20) and the MMSE estimator θmmse in Eq. ( 21) are also plotted.

Here, the background noise level σ ξ =1and the interval bound a =2for the parameter θ. Specially, for the limiting case m →∞,MSEsof θLC are also plotted in (b) over a suitable range. (c) MSEs of θLC and θNE constructed form the quantizer θqt of Eq. ( 22) with the threshold γ =0versus σ η . (d) For a combiner of identical θqt , the MSEs of θLC and the MMSE estimator θmmse versus the background noise level σ ξ at a fixed added noise level.

deviation σ η . As in Fig. 2(a), with the benefits of added noise, both θLC and θNE are better than the MAP estimator θmap .

At an optimal noise level σ opt η =2.8184, the noise-enhanced estimator θNE achieves its minimum MSE value of 0.2494. After the optimal added noise level σ η >σ opt η ,theMSEof θNE goes up again, and finally reaches a stable value of 0.33 for very large added noise levels (e.g. σ η > 10 2 ). This can be viewed as a kind of stochastic resonance effect measuring by the MSE that descends to its lowest point at an optimal non-zero noise level. It is also interesting to note in Fig. 2(a) that, as the added noise level σ η increases and the number m ≥ 5,theMSER o m of θLC calculated by Eq. ( 13) can be further reduced by the added noise. In order to observe the behavior of R o ∞ in Eq. ( 14) in detail, Fig. 2

(b) redraws the MSE R o

∞ over a suitable range, and clearly shows that the MSE R o ∞ in Eq. ( 14) achieves a minimum value of 0.2493 at an optimal noise level σ opt η =1.995. When σ η >σ opt η , R o ∞ also rises and gradually evolves into a stable value of 0.2500 for large noise levels (e.g. σ η > 10). Therefore, compared with the estimator θNE ( * ), the designed estimator θLC (•) has the similar noise-enhanced effect, but exhibits a quite stable improvement for a wide range of added noise levels, as shown in Figs. 2(a) and (b). The reason is that, as the added noise level σ η increases, the optimum weighting coefficient w o in Eq. ( 15) can adaptively tune itself for the variety of added noise, and an effectively reduced MSE R o ∞ is achieved by the incorporation of added noise η i and the weights w o in θLC .Itis shown in Figs. 2(a) and (b) that, even without the help of added noise η (when σ η =0), the MSE R o m of θLC can be optimized by the weight vector w o as 0.2550 much lower than the MSE 0.4832 achieved by θNE at σ η =0. This superiority of θLC over θNE is also theoretically proved by Eq. ( 43) in Appendix D.

The benefits of added noise might take effect in other suboptimal estimators. For instance, consider a binary quantizer [START_REF] Mcdonnell | Stochastic Resonance: From Suprathreshold Stochastic Resonance to Stochastic Signal Quantization[END_REF], [START_REF] Stocks | Suprathreshold stochastic resonance in multilevel threshold systems[END_REF]

θqt (x)= 1,x ≥ γ, 0, x<γ, (22)
with the threshold γ. Similarly, based on this suboptimal estimator θqt , the combined estimator θLC and the noise-enhanced estimator θNE can be also obtained. Naturally, the optimal added noise level σ opt η is expected to be obtained for achieving the minimum MSE of the Bayesian estimators θNE and θLC . The solution of σ opt η by setting the derivative ∂R o m /∂σ η =0 may however identify several extrema. As shown in Figs. 2(a)-(c), the equation of ∂R o m /∂σ η =0will yield multiple solutions of σ η . In addition, the second-order derivative ∂ 2 R o m /∂σ 2 η needs to be derived for further determining the optimal level σ opt η . However, it is also seen in Figs. 2 (a)-(c) that, for a sufficiently large number m (e.g. m ≥ 10 4 ), the MSE R o m of the combined estimator θLC reaches a plateau with stationary values of 0.25 for a wide range of added noise levels (e.g. σ η > 1.995 in Fig. 2(a) or σ η > 4.4668 in Fig. 2(c)). This robust feature is very attractive and forms Corollary 4.

Corollary 4: For the combined estimator θLC in Eq. ( 11), if the original estimator θ(x)| x→∞ =Θ< ∞ and the added noise has a symmetric scale-family PDF 14) has a plateau of the local extremum for large added noise levels σ η . Here, f η is the PDF of the standardized noise variable η with zero mean and unit variance.

f η (η, σ η )=(1/σ η ) f η (η/σ η ), then the MSE R o ∞ of Eq. (
Proof of Corollary 4 is given in Appendix F. Corollary 4 only tells us the MSE R o ∞ of θLC reaches a stable value for a large added noise level σ η , but does not indicate how low this stable value is. Due to the optimum weight w o and for a large noise level, it is seen in Figs. 2 (a)-(c) that this stable value of 0.2500 is rather near to the MMSE 0.2492 given by θmmse . This interesting characteristic of θLC suggests that, instead of finding the optimal noise level σ opt η , a fixed large noise level σ η can be used to improve the MSE of the combined estimator θLC with a tolerable accuracy. In practice, the limiting case of m →∞is inaccessible and can only be approached. Then, we here use a combiner of quantizers to construct the estimator θLC with a finite number m =10 5 , and plot its MSE R o m as a function of the background noise level σ ξ in Fig. 2(d). It is clearly seen in Fig. 2(d) that, with the help of added noise (when σ η = 10), the combined estimator θLC achieves a rather comparable performance in comparison to the MMSE estimator θmmse in the considered range of

σ ξ ∈ [0.1, 4].
In Fig. 2, we only consider the added noise η with Gaussian distribution. Since the noise components η i are artificially added to estimators θi , then the characteristic of added noise η can be selected purposefully for improving the performance of the designed estimator θLC . Thus, an important question is whether the addition of any type of noise to the estimators θi is always beneficial to the decrease of MSE. When the initial rate of the MSE is negative, i.e.

lim σ η →0 dR o m dσ η < 0, ( 23
)
the answer is affirmative in certain circumstances, as stated in Corollary 5.

Fig. 3. MSEs R o ∞ of the combined estimator θLC as a function of the added noise level σ η . Here, θLC is constructed form the quantizer θqt of Eq. ( 22) with the threshold γ =0. The added noise η has the generalized Gaussian PDF with exponents α =0.5, 1, 2 and ∞. The other parameters are the same as in Fig. 2 Corollary 5: For the designed estimator θLC composed of identical estimators θi = θqt and the symmetric scale-family added noise η, θLC can benefit from the added noise when the number m>1 and f x (γ) =0.

Proof of Corollary 5 is presented in Appendix G, and an illustrative example is given as follows.

Example 2: Consider the symmetric scale-family added noise η with generalized Gaussian PDF

f η (x)=(c 1 /σ η )exp (-c 2 |x/σ η | α ). Here, c 1 = α 2 Γ 1 2 ( 3 α )/Γ 3 2 ( 1 α ), c 2 =[Γ( 3 α )/Γ ( 1 α )] α 2 , Γ(x)
is the gamma function and the exponent α>0 [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF], [START_REF] Pan | Noise enhancement in robust estimation of location[END_REF]. It is shown in Fig. 3 that the noise-enhanced effects of the MSE also occur clearly for different noise types of α =0.5, 1 (Laplacian noise), 2 (Gaussian noise) and ∞ (uniform noise). The larger the exponent α is, the faster the initial rate of the MSE R o ∞ declines, which also accords to the results in Ref. [START_REF] Patel | Optimal mean-square noise benefits in quantizerarray linear estimation[END_REF]. The corresponding MSE achieved by different noise types also reaches a minimum at an optimal added noise level, and finally tends to the stable value of 0.25 for larger added noise levels, as Corollary 4 indicated.

However, Corollary 5 is only applicable to the prescribed symmetric scale-family added noise η. An important problem is whether the benefit of the added noise (not restricted to the symmetric scale-family) exists or not for a combiner of arbitrary suboptimal estimators θi . The resolution of this problem is in general difficult [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF]- [START_REF] Balkan | CRLB based optimal noise enhanced parameter estimation using quantized observations[END_REF], [START_REF] Pan | Noise enhancement in robust estimation of location[END_REF]- [START_REF] Chen | Noise-enhanced information systems[END_REF], [START_REF] Bayram | Noise enhanced hypothesis-testing in the restricted Bayesian framework[END_REF], even for a combiner of arbitrary identical estimators with the simplified expression of the MSE R o m in Eq. [START_REF] Zhai | Noise effect on signal quantization in an array of binary quantizers[END_REF]. In this case, since the variance var(θ) is given, finding the optimal added noise PDF to minimize the MSE R o m of θLC with a finite number m is the optimization problem

f opt η (η)=argmin f η R o m =argmax f η mp 2 i C ii +(m -1)C ij , s.t. f η (η) ≥ 0, f η (η)dη =1, (24)
which is a constrained minimization of the nonlinear functional R o m (f η ) for the given observation data x, the estimators θi and the number m. This optimization problem is in general theoretically intractable, even for a single estimator θi with m =1.This is due to the fact that the expectation E x {E η [ θi (x + η)]} has no constraint due to the adaptive bias weight w 0 , and the terms E 2

x {E η [ θi (x + η)]} in Eq. ( 5) (Eq. ( 38)) and E x {E 2 η [ θi (x + η)]} in Eq. ( 6) (Eq. ( 39)) are nonlinear functional of f η . Thus, the theoretical determination of optimal added noise PDF f opt η that minimizes R o m can not refer to the convex optimization method proposed by Chen et al. [START_REF] Chen | Noise enhanced parameter estimation[END_REF], [START_REF] Chen | Noise-enhanced information systems[END_REF], [START_REF] Chen | Theory of the stochastic resonance effect in signal detection: Part I-Fixed detectors[END_REF], [START_REF] Chen | Theory of the stochastic resonance effect in signal detection-Part II: Variable detectors[END_REF]. The necessary and sufficient conditions determining the solution of this non-convex optimization problem in Eq. ( 24) are that the first variation of R o m vanishes and the second variation of R o m is nonnegative [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF], [START_REF] Zeidler | Nonlinear Functional Analysis and Its Applications, Part 3: Variational Methods and Optimization[END_REF]. As Uhlich [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF] pointed out, these theoretical conditions are rather difficult to handle in general, and such approximation methods as Parzen widows density estimation [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF] and particle-swarm optimization [START_REF] Bayram | Noise enhanced hypothesis-testing in the restricted Bayesian framework[END_REF] can be employed.

For low computational cost, we here employ an approximate form of added noise PDF as

f opt η (η)= K k=1 ν k ̺ k (η, µ k ,σ k ), (25)
where the normalization coefficients ν k ≥ 0, K k=1 ν k =1, and the Gaussian window function [START_REF] Duda | Pattern Classification[END_REF]. As the number K of ̺ k increases, the estimation form f opt η (η) of Eq. ( 25) gradually converges to f opt η (η) in Eq. ( 24) under certain conditions [START_REF] Uhlich | Bayes risk reduction of estimators using artificial observation noise[END_REF], [START_REF] Bayram | Noise enhanced hypothesis-testing in the restricted Bayesian framework[END_REF], [START_REF] Duda | Pattern Classification[END_REF].

̺ k (η, µ k ,σ k ) = exp[-(η - µ k ) 2 /2σ 2 k ]/ 2πσ 2 k with means µ k and standard deviations σ k ≥ 0 [37],
Example 3: Reconsider the parameter estimation given in Example 1, and choose the estimator θi = θqt of Eq. ( 22) with the threshold γ =0. The number of ̺ k is chosen as K =4, and the constrained nonlinear optimization algorithm of sequential quadratic programming [START_REF] Nocedal | Numerical Optimization[END_REF] is employed to find the approximation PDF of the optimal added noise. Note that, for randomly selected initial values of coefficients 0 ≤ c k ≤ 1, means -∞ <µ k < ∞ and standard deviation σ k ≥ 0, both the sequential quadratic programming method and the particle-swarm optimization method are carried out for finding the minimum MSE R o m ( f opt η ) and the corresponding optimal PDF f opt η of added noise η. At each major iteration of the sequential quadratic programming method, a positive definite quasi-Newton approximation of the Hessian of the objective function is calculated using the BFGS method [START_REF] Nocedal | Numerical Optimization[END_REF], [START_REF] Dennis | Numerical Methods for Unconstrained Optimization and Nonlinear Equations[END_REF]. For different estimator numbers m =1, 2, 10 4 and ∞, we randomly choose 20 groups of initial coefficients c k , means µ k and standard deviation σ k , the minimum MSEs R o m ( f opt η ) with the corresponding vector parameters

ν =[ν 1 ,ν 2 ,...,ν K ] ⊤ , µ =[µ 1 ,µ 2 ,...,µ K ] ⊤ and σ =[σ 1 ,σ 2 ,...,σ K ] ⊤ are recorded in Table I.
It is interesting to note that, for a single estimator with m =1, the optimal added noise PDF is f opt η (η)=δ(η +1.0091) with the Dirac delta function δ(η). As the estimator number m =10 4 , the approximate optimal added noise PDF f opt η (η) have two nonzero normalization coefficients and is composed of two Gaussian window functions, yielding a rather small MSE of 0.2493. Interestingly, for the limiting case m →∞, the optimal added noise PDF obtained in Table I, as shown in Fig. 4 (red

η) of the optimal added noise for the estimator θi = θqt of Eq. ( 22) with the threshold γ =0and the number m = ∞. Here, f opt η (η) is solved for the Gaussian (red solid line) and the Laplacian (blue dashed line) background noise ξ. The other parameters are the same as in Fig. 2. solid line), presents a minimum MSE R o ∞ ( f opt η )=0.2492, which is just the MMSE achieved by the optimal estimator of Eq. [START_REF] Patel | Optimal noise benefits in Neyman-Pearson and inequality-constrained statistical signal detection[END_REF]. We note that the solution f opt η (η) of the optimal added noise PDF is not unique, because the particle-swarm optimization method also presents another solution as

f opt η (η)= exp[-(η -µ η ) 2 /2σ 2
η ]/ 2πσ 2 η with µ η = -1 and σ η =2.004. Using this solution, the corresponding MSE also achieves the MMSE 0.2492. Of course, for various estimators θi , different prior PDF f θ and background noise PDF f ξ , both the sequential quadratic programming method and the particle-swarm optimization method need to be applied anew.

Example 4: In Examples 1-3, the background noise ξ is assumed to be Gaussian distributed. However, the background noise ξ in the real data model of Eq. ( 1) can in some situations come with non-Gaussian distributions, and its type results in the variety of MMSE Bayesian estimators θmmse and the corresponding MMSE values. It is noted that the approximate PDF form in Eq. ( 25) of the optimal added noise and the constrained nonlinear optimization algorithms remain valid. For a uniformly distributed parameter θ buried in a given background noise ξ with its PDF f ξ and cumulative distribution function F ξ , the MMSE Bayesian estimator of Eq. ( 19) becomes

θmmse (x)=E θ|x [θ|x]=x + x-a x uf ξ (u)du F ξ (x) -F ξ (x -a) . (26)
For instance, consider the uniform distributed parameter θ with its PDF f θ (u)=1/2 (u ∈ [0, 2]) corrupted by the the Laplace noise ξ with its PDF f ξ (ξ) = exp(-√ 2|x|/σ ξ )/( √ 2σ ξ ) and standard deviation σ ξ = √ 2, the MMSE achieved by the estimator of Eq. ( 26) is 0.2679. For the limiting case of m →∞,weuse the constrained nonlinear optimization algorithms as in Example 3 with the number K =4to optimize the designed Bayesian estimator θLC constructed by quantizer with threshold γ =0. Then, the approximate form of the optimal added noise PDF is f opt η (η)=[e -(x-m 1 ) 2 /(2σ 2 η ) + e -(x-m 2 ) 2 /(2σ 2 η ) ]/(2 2πσ 2 η ) with two nonzero coefficients ν 1 = ν 2 =0.5, means m 1 = -1.3784, m 2 = -0.6217 and standard deviations σ 1 = σ 2 = σ η =0.3195 (see Fig. 4 in blue dashed line), which is effective for obtaining the corresponding MSE 0.2679, the same as the MMSE achieved by θmmse (x) in Eq. [START_REF] Soganci | Optimal signal design for multiparameter estimation problems[END_REF]. Therefore, a natural question is that, for any type of background noise ξ, can we always find an approximation form f opt η of optimal noise PDF for the designed estimator θLC to achieve the MMSE? We have shown this is feasible for the cases of great practical relevance of a Gaussian background noise and a non-Gaussian (Laplacian) background noise. For other cases, we here leave these interesting open questions of the solution of Eq. ( 24) for further study. We can extend the design of the Bayesian estimator to the finite impulse response (FIR) filter as follows.

III. NOISE-ENHANCED FIR FILTERS

We can extend the noise-benefit combiner G, i.e. module 1 plus module 2 marked in Fig. 1, to estimate a desired signal z n by the current sample x n and the past samples x n-1 ,x n-2 , ...,x n-L , as illustrated in Fig. 5. Here, we assume that the straight-forward observation x n can not be directly obtained and goes through the distortion endowed by the nonlinearity θ in the combiner, representing for example neuronal models [START_REF] Collins | Stochastic resonance without tuning[END_REF], nonlinear estimators [START_REF] Kay | Fundamentals of Statistical Signal Processing: Estimation Theory[END_REF]- [START_REF] Fang | Nonlinear Bayesian estimation: From Kalman filtering to a broader horizon[END_REF] or sensors [START_REF] Chapeau-Blondeau | Noise-enhanced Fisher information in parallel arrays of sensors with saturation[END_REF], [START_REF] Mcdonnell | Stochastic Resonance: From Suprathreshold Stochastic Resonance to Stochastic Signal Quantization[END_REF], [START_REF] Stocks | Suprathreshold stochastic resonance in multilevel threshold systems[END_REF]. In this situation, for L +1samples of x n , L +1combiners G ℓ are equipped and each combiner G ℓ contains m original estimators θℓi , m added noise components η ℓi and the corresponding adjustable weighting coefficients ω ℓi for ℓ =0, 1,...,L and i =1, 2,...,m. Then, collecting all outputs of combiners G ℓ and adding the bias weight ω 0 , we have the designed noiseenhanced finite impulse response (FIR) filter

ẑn = ω 0 + L ℓ=0 m i=1 ω ℓi θℓi (x n-ℓ + η ℓi ). ( 27
)
Here, the number L +1 of combiners G ℓ is the order of the designed FIR filter of Eq. ( 27) and the error signal is defined as

ε n = z n -ẑn .
Furthermore, in each combiner G ℓ , Theorem 2 also holds. Thus, consider the individual estimators θℓi = θℓ in each G ℓ are identical, we have m identical optimum weight coefficients ω ℓi = w ℓ . In this case, we can label these weight coefficients w ℓ as the vector w =[w 0 ,w 1 ,...,w L ] ⊤ and rewrite the estimate vector as θ =[ θ0 , θ1 ,..., θL ] ⊤ with θℓ = m i=1 θℓ (x + η i ). Using the unbiased condition E ẑ (ẑ n )=E z (z n ), the designed filter in Eq. ( 27) becomes

ẑn =E z (z n )+w ⊤ θ -E x,η ( θ) . ( 28
)
Under this circumstance, the MSE J m,L+1 of the FIR filter in Eq. ( 28) is

J m,L+1 =E[ε 2 n ]=v ar(z n ) -2w ⊤ p + w ⊤ Cw. (29)
The covariance matrix

C =E x,η {[ θ-E x,η ( θ)][ θ-E x,η ( θ)] ⊤ } has diagonal elements C ℓ+1,ℓ+1 =E x n-ℓ [E η ( θ2 ℓ )] -E 2 x n-ℓ [E η ( θℓ )] =E x n-ℓ {mE η [ θ2 ℓ (x n-ℓ + η)] + m(m -1) × E 2 η [ θℓ (x n-ℓ + η)]} -m 2 E 2 x n-ℓ {E η [ θℓ (x n-ℓ + η)]} and the non-diagonal elements (ℓ = κ) C ℓ+1,κ+1 =E x n-ℓ ,x n-κ [E η ( θℓ )E η ( θκ )] -E x n-ℓ [E η ( θℓ )]E x n-κ [E η ( θκ )] = m 2 E x n-ℓ ,x n-κ {E η [ θℓ (x n-ℓ + η)]E η [ θκ (x n-κ + η)]} -m 2 E x n-ℓ {E η [ θℓ (x n-ℓ + η)]}E x n-κ {E η [ θκ (x n-κ + η)]}, where E x n-ℓ ,x n-κ (•)= •f x (x n-ℓ ,x n-κ )dx n-ℓ dx n-κ is with respect to the second-order joint PDF f x (x n-ℓ ,x n-κ ). The cen- tralized cross-correlation vector p =E x,η {[z n -E z (z n )][ θ - E x,η ( θ)]} has elements p ℓ+1 = mE z n ,x n-ℓ {z n E η [ θℓ (x n-ℓ + η)]}-mE z (z n ) × E x n-ℓ {E η [ θℓ (x n-ℓ + η)]}.
Then, we obtain the optimum weight vector w o = C -1 p and the optimized MSE J o m,L+1 of the considered FIR filter in Eq. ( 28) is given by

J o m,L+1 =E(ε 2 n )=v ar(z n ) -p ⊤ C -1 p. ( 30
)
Fig. 6. Plots of the MSE J o m,2 of the designed filter in Eq. ( 28) as a function of the added noise level σ η for different numbers m =1, 2, 31, 256, 10 3 , 10 5 and ∞. Here, the saturation estimators of Eq. ( 31) are with the identical parameters β =5, and the external Gaussian noise level σ ξ =0.1.

Notice the multiplicator m in each element p ℓ+1 and the term p ⊤ C -1 p in Eq. ( 30), we find limits of elements as

C ∞ ℓ+1,ℓ+1 = lim m→∞ C ℓ+1ℓ+1 /m 2 =E x n-ℓ {E 2 η [ θ(x n-ℓ + η)]}-E 2 x n-ℓ {E η [ θ(x n-ℓ + η)]}, C ∞ ℓ+1,κ+1 = lim m→∞ C ℓ+1κ+1 /m 2 =E x n-ℓ ,x n-κ {E η [ θℓ (x n-ℓ + η)]E η [ θκ (x n-κ + η)]} -E x n-ℓ {E η [ θℓ (x n-ℓ + η)]}E x n-κ {E η [ θκ (x n-κ + η)]}, p ∞ ℓ+1 = p ℓ+1 /m =E z n ,x n-ℓ {z n E η [ θℓ (x n-ℓ + η)]}-E z (z n ) × E x n-ℓ {E η [ θℓ (x n-ℓ + η)]}.
Substituting these limits into Eq. ( 30), we can also compute the limit value of J o ∞,L+1 as m →∞. Example 5: Consider the noisy input x n = s n + ξ n , where the input signal s n =sin(2πn/N) is a sampled sinusoid with N =16 (N>2) samples per period and the external whitenoise process ξ n is Gaussian distributed. The desired signal is also assumed to be the sampled sinusoid z n =2cos(2πn/N) at the same frequency [START_REF] Widrow | Adaptive Signal Processing[END_REF]. Due to the periodicity of the input and desired signals, the expectations must be computed by averaging over one period, i.e the operator N n=1 (•)/N . Then, the variance of the desired signal is var(z n )=2. When the external Gaussian noise ξ n has a fixed noise level σ ξ =0.1 and the added noise components η ℓi are also Gaussian distributed, we plot the MSE J o m,2 of the filter in Eq. ( 28) as a function of the added noise level σ η for combiners with m identical saturation estimators θℓ (x)=tanh(βx), [START_REF] Dabeer | Signal parameter estimation using 1-bit dithered quantization[END_REF] where the slope parameter β>0. It is clearly seen in Fig. 6 that the MSE J o m,2 of the designed filter in Eq. ( 28) benefits from the increase of the added noise level σ η as the number m is large enough. For instance, as m =10 5 and at an optimal added 6). In addition, Ref. [START_REF] Widrow | Adaptive Signal Processing[END_REF] (pp. 103, Fig. 6.3 in Sect. 6) also considered this example of the two-sample FIR filter, namely Wiener filter, yielding the MSE value of 0.4011. Notice that the weights of Wiener filter are directly deduced by the covariance matrix of x n and the cross-correlation vector between z n and samples of x n [START_REF] Widrow | Adaptive Signal Processing[END_REF]. Obviously, with the benefit of added noise and for a sufficient large order m =10 5 , the obtained result in Fig. 6 is better. In fact, the sinusoid s n = A sin(2πn/N) can be viewed as a variable with PDF f s (u)=1/(π √ A 2 -u 2 ) over the interval u ∈ [-A, A], and the estimate of the desired signal from the observation data is also a Bayesian estimation problem. The corresponding MMSE filter is not the linear Wiener filtering. The designed filter of Eq. ( 28) might achieve a better MSE than the linear Wiener filter, as shown in Fig. 6.

Upon increasing the order L +1 and at the corresponding optimal added noise levels, we list the minimum values of the MSE J o m,L+1 in Table II for the designed FIR filter with the number m =10 5 . As the order L +1 increases, Theorem 1 also holds and the minimum MSE J o m,L+1 of Eq. ( 28) gradually decreases from 0.3558 at order L +1=2 to 0.0069 at order L +1=9. Compared with the results of Wiener filter, the added noise η, as a potential designable variable, clearly manifests its benefits with a performance that is always superior in Table II for the nonlinear FIR filter. In addition, it is also seen in Fig. 6 that, for a large enough level of added noise, the MSE J o m,2 keeps a stable value for a sufficiently large number m →∞. This robust feature of the noise-enhanced filter of Eq. ( 28) also suggests no need of tuning the added noise level. Then, in Table II, we choose the number m =10 5 and list the MSE J o m,L+1 of the noise-enhanced filter in Eq. ( 28) at a fixed added noise level σ η =5, yielding a comparative results of the MSE in Table II. Specially, for L +1=5,Fig. 7(a) shows a realization of the input x n , and the saturation estimator ϑ ℓ (x n ) = tanh(5x n ) is illustrated in Fig. 7(b) at the added noise level σ η =0. When the added noise level σ η =5and m =10 5 , we obtain the optimum weight vector w o = C -1 p =10 -5 × [3.1082, 0.3873, -2.3912, -4.80578, -6.4904] ⊤ and plot the output ẑn (•) of the designed filter of Eq. ( 28) in Fig. 7(c). Note that ẑn starts from n =5, since it is a function of x n and the past four samples x n-ℓ . For comparison, the desired signal z n =2cos(2πn/N) (red solid line) is also given in Fig. 7(c). In this numerical realization of ẑn , the corresponding MSE 0.0261 agrees well with (is slightly larger than) the MSE J o m,5 =0.0212 at the order L +1=5shown in Table II. 28) at the added noise level σ η =5and for m =10 5 , and the desired signal z n = 2 cos(2πn/N). The other parameter are the same as in Fig. 6.

IV. DISCUSSION

Using the Bayes risk for the MSE criterion, this paper mainly investigates the benefits of added noise in the designed Bayesian estimator composed of a combination of individual Bayesian estimators multiplied by the corresponding optimum weighting coefficients. We first demonstrate that the MSE of the combined estimator gradually decreases as the number of individual estimators increases, and then find two equivalent optimum weight coefficients tied with any two identical estimators in the combination. Then, some easily tractable properties of a combination of identical estimators are deduced. It is shown that the MSE of the combined estimator can be optimized not only by the linear weighting coefficients but also by the added noise. Moreover, when the individual estimator has an upper bound and the number of the individual estimators is sufficiently large, we find a robust feature of the combined Bayesian estimator, i.e. the MSE approaching a plateau for a wide range of larger added noise levels. This feature facilitates improvement of the MSE of the designed estimator without tuning the added noise level, and greatly extends the operating range of the designed estimator in practical applications. For the Bayesian estimator composed of identical quantizers, we prove that the injection of a small amount of symmetrical scale-family noise into the observation always improves the MSE of the Bayesian estimator. In order to find the optimal added noise that achieves a "minimum" MSE as close to the MMSE as possible, we adopt the sequential quadratic programming and the particle-swarm optimization method to obtain the approximate form of the optimal noise PDF. The illustrative results show the effectiveness of these constrained nonlinear optimization algorithms. We also extend the combined Bayesian estimator to the nonlinear filtering. In addition, we can extend the results of the scalar parameter model of Eq. (1) to the vector observation model

x = Hϑ + ξ, (32)
where the observation vector x =[x 1 ,x 2 ,...,x N ] ⊤ , the unknown vector parameter ϑ =[ϑ 1 ,ϑ 2 ,...,ϑ k ] ⊤ ,t h eN × k observation matrix is H and the background noise vector ξ = [ξ 1 ,ξ 2 ,...,ξ N ] ⊤ . In Appendix H, the Bayesian estimator θ for the parameter vector ϑ is derived in Eq. ( 56). The further discussions of this general Bayesian estimator vector θ in Eq. ( 56) is significant to the theoretical research and practical applications of noise benefits in the parameter estimation. Finally, two specially important questions that remain in this extended vector model are how to prejudge whether the added noise is beneficial or not, and how to find the exact expression of the optimal added noise PDF that minimizes the MSE of the designed Bayesian estimator formed by a combiner of suboptimal but easily implementable nonlinear estimators.

C m = C m-1 γ m-1 γ ⊤ m-1 C mm = L m-1 0 ℓ ⊤ m-1 1 D m-1 0 0 d m L ⊤ m-1 ℓ m-1 0 1 , (33)
where the sub-matrix C m-1 is just the covariance matrix of the combiner with m -1 individual estimators. This is the optimum nesting property of a matrix (vector) [START_REF] Manolakis | Statistical and Adaptive Signal Processing[END_REF]. The diagonal matrix D m in Eq. ( 33) and the cross-correlation vector p m = [p ⊤ m-1 ,p m ] ⊤ have the optimum nesting property. From Eq. ( 33), the row ℓ m-1 is uniquely given by L m-1 D m-1 ℓ m-1 = γ m-1 , and then L m also possesses the optimum nesting property. Defining an intermediate vector k m = L ⊤ m w o m and from Eq. ( 7), we find

C m w o m = L m D m L ⊤ m w o m = L m D m k m = p m (34)
and

k m =[k ⊤ m-1 ,k m ]
⊤ has the optimum nesting property. Therefore, the MSE R o m of Eq. ( 9) can be expressed as

R o m =v ar(θ) -(w o m ) ⊤ C m w o m =v ar(θ) -k ⊤ m D m k m =v ar(θ) - k m-1 k m ⊤ D m-1 0 0 ⊤ d m k m-1 k m =v ar(θ) -k ⊤ m-1 D m-1 k m-1 -d m k 2 m = R o m-1 -d m k 2 m , (35)
where

R o m-1 =v ar(θ) -k ⊤ m-1 D m-1 k m-1
is just the MSE of the combined estimator θLC with the number m -1.Fromthe positive-definiteness of C m , d m > 0 and k 2 m > 0. Thus, Eq. ( 10) holds.

APPENDIX B PROOF OF Theorem 2

Without loss of generality, we label two identical estimators as θ1 = θ2 .LetC ij denote the element of C at the ith row and the jth column for i, j =1, 2,...,m. From Eqs. ( 5) and ( 6

w o 1 = m j=1 A 1j p j det (C) = m j=1 A 2j p j det (C) = w o 2 . (36)

APPENDIX C PROOF OF COROLLARY 1

For a combiner of m identical estimators θi = θ and according to Theorem 1, all weight coefficients w i = w are equivalent for i =1, 2,...,m. In this case, the weight vector w = w1 for a m × 1 dimensional vector 1 of all ones and the estimate vector θ =[ θ(x + η 1 ), θ(x + η 2 ),..., θ(x + η m )] ⊤ . Thus, the combined estimator θLC in Eq. ( 2) can be simplified as Eq. [START_REF] Xu | Adaptive recursive algorithm for optimal weighted suprathreshold stochastic resonance[END_REF]. In addition, due to identical estimators θi = θ, Eq. (4) becomes [START_REF] Bayram | Noise enhanced hypothesis-testing in the restricted Bayesian framework[END_REF] which indicates the centralized cross-correlation vector p has m equivalent elements p i . Similarly, from Eqs. ( 5) and ( 6), the covariance matrix C has m equivalent diagonal elements

p i =E x {θE η [ θ(x + η)]}-E θ (θ)E x {E η [ θ(x + η)]},
C ii =E x {E η [ θ2 (x + η)]}-E 2 x {E η [ θ(x + η)]} (38)
and m(m -1) equivalent non-diagonal elements

C ij =E x {E 2 η [ θ(x + η)]}-E 2 x {E η [ θ(x + η)]}. (39)
Then, the MSE R m of θLC in Eq. ( 11) can be computed as

R m =E x,η θ -θLC 2 =v ar(θ) -2w m i=1 p i + w 2 m i=1 m j=1 C ij =v ar(θ) -2mwp i + mw 2 [C ii +(m -1)C ij ]. (40)
Since the covariance matrix C is positive definite and

1 ⊤ C1 > 0, then C ii +(m -1)C ij > 0.
Setting the derivative ∂R m /∂w =0, we obtain the optimum weight w o of Eq. ( 12) achieving the minimum of Eq. [START_REF] Nocedal | Numerical Optimization[END_REF]. Substituting the optimum weight w o into Eq. ( 40), the MSE R o m can be simplified as Eq. ( 13). As the number m →∞and C ii < ∞,w eh a v et h e limit R o ∞ = lim m→∞ R o m in Eq. ( 14).

APPENDIX D PROOF OF COROLLARY 2

For the Bayesian estimator in Eq. ( 15), the corresponding MSE is given by

R =E x θ -θLC 2 =v ar(θ) -2wE x θ -E θ (θ) θNE (x) -E x ( θNE ) + w 2 E x θNE (x) -E x ( θNE ) 2 . (41)
Then, setting the derivative ∂R/∂w =0, the optimum weight w o is solved as Eq. ( 17). Substituting the optimum weight w o of Eq. ( 17) into Eq. ( 41), we have the minimum MSE

R o =v ar(θ) - E 2 x θ -E θ (θ) θNE (x) -E x ( θNE ) E x θNE (x) -E x ( θNE ) 2 . (42)
Substituting θNE (x)=E η [ θ(x+η)] of Eq. ( 16) into Eq. ( 42), we find the numerator E 2 x θ -E θ (θ) θNE (x) -E x ( θNE ) = p i of Eq. ( 37) and the denominator E x [( θNE (x) -E x ( θNE )) 2 ] = C ij of Eq. [START_REF] Duda | Pattern Classification[END_REF]. Therefore, the MSE R o of Eq. ( 42) is just the limit R o ∞ of Eq. ( 14). We also find that R o of Eq. ( 42) satisfies

R o =min w E x θ -θLC 2 =min w E x θ -E θ (θ) -w θNE -E x ( θNE ) 2 ≤ E x θ -E θ (θ) -θNE -E x ( θNE ) 2 | w=1 =E x θ -θNE 2 , (43)
where the unbiased condition E x ( θNE )=E θ (θ). This inequality of Eq. ( 43) clearly shows that the estimator θLC in Eq. ( 15) presents a better or at lest equivalent MSE compared to that of the estimator θNE in Eq. ( 16), and this is obtained thanks to purposeful addition of noise and the adjustable weighting coefficients in the combiner of Fig. 1.

APPENDIX E PROOF OF COROLLARY 3

From the MSE R o ∞ of Eq. ( 14), i.e. Eq. ( 42), achieved by the estimator θLC of Eq. ( 15) and by using the Cauchy-Schwarz inequality, we find the numerator

E 2 x θ -E θ (θ) θNE (x) -E x ( θNE ) =E 2 x E θ|x θ -E θ (θ) θNE (x) -E x ( θNE ) =E 2 x θmmse -E θ (θ) θNE (x) -E x ( θNE ) ≤ E x θmmse (x) -E θ (θ) 2 ]E x [ θNE (x) -E x ( θNE ) 2 , (44)
where the equality occurs when θmmse (x) -E θ (θ)=κ[ θNE (x) -E x ( θNE )], i.e. Eq. ( 20), and κ is an arbitrary constant. Sub-stituting the inequality of Eq. ( 44) into Eq. ( 42), we have

R o ∞ ≥ var(θ) -E x θmmse (x) -E θ (θ) 2 ] =v ar(θ) -E x θ2 mmse +2E x θmmse E θ (θ) -E 2 θ (θ) =E θ (θ 2 ) -E x θ2 mmse =E x θ -θmmse 2 = R ms ,
where the unbiased condition E θ (θ)=E x ( θmmse ) and

E x (θ θmmse )=E x [E θ|x (θ) θmmse ]=E x ( θ2 mmse ).
Thus, Eq. ( 18) holds.

APPENDIX F PROOF OF COROLLARY 4

From Eq. ( 14), we have the derivative

∂R o ∞ ∂σ η = p 2 i ∂C ij ∂σ η -2p i C ij ∂p i ∂σ η C 2 ij . (45)
From Eq. ( 37) and Eq. ( 39) and noting the interchange of the order of differentiation and integration in Eq. (45), both derivatives ∂p i /∂σ η and ∂C ij /∂σ η contain the term

∂E η [ θ(x + η)] ∂σ η = θ(x + η) ∂f η (η, σ η ) ∂σ η dη = θ(x + η) - f η ( η σ η ) σ 2 η - ηf ′ η ( η σ η ) σ 3 η dη = θ(x + σ η η) - f η (η) σ η - ηf ′ η (η) σ η dη, (46)
where η = η/σ η and f ′ η = df η /dη. Since θ(x)| x→∞ =Θ< ∞, f η (η)dη =1and the derivative with respect to σ η allowing us to interchange the order of integration and differentiation [START_REF] Patel | Optimal mean-square noise benefits in quantizerarray linear estimation[END_REF], we find

lim σ η →∞ θ(x + σ η η) f η (η) σ η dη = lim σ η →∞ θ(x + σ η η) f η (η) σ η dη = lim σ η →∞ Θ f η (η)dη σ η =0, and
lim σ η →∞ θ(x + σ η η) ηf ′ η (η) σ η dη = lim σ η →∞ Θ ηf η (η)| ∞ -∞ -f η (η)dη σ η = lim σ η →∞ Θ ηf η (η)| ∞ -∞ σ 2 η - 1 σ η =0,
where f η (±∞)=0and lim σ η →∞,η→±∞ ηf η (η)/σ 2 η is infinitesimal of higher order. Thus, lim σ η →∞ ∂E η [ θ(x + η)]/∂σ η =0. Substituting this limit into Eq. (45), we have

lim σ η →∞ ∂R o ∞ ∂σ η =0. ( 47
)
Therefore, under the considered conditions in Corollary 4, the MSE R o ∞ of Eq. ( 14) tends to a (local) extremum asymptotically for large added noise levels.

APPENDIX G PROOF OF COROLLARY 5

For a combiner of identical quantizer estimators θi = θqt of Eq. ( 22), the terms E η [ θqt (x + η)] = E η [ θ2 qt (x + η)] = 1 -F η (γ -x)=F η (x -γ)=F η [(x -γ)/σ η ], where F η (x)= F η (x/σ η ) is the cumulative distribution function of added noise η and F η corresponds to the cumulative distribution function of standardized noise η = η/σ η . From Eq. ( 13), the derivative of R o m with respect to the added noise level σ η is

∂R o m ∂σ η = mp 2 i ∂C ii ∂σ η +(m -1) ∂C ij ∂σ η [C ii +(m -1)C ij ] 2 - 2mp i ∂p i ∂σ η C ii +(m -1)C ij .
Since the symmetrically-distributed noise η has zero mean f η ( η) ηd η =0and let η =(x -γ)/σ η ,wehave

Then, we find

lim σ η →0 ∂p i ∂σ η =0, lim σ η →0 ∂C ii ∂σ η =0. (50)
Due to F η (-η)=1-F η ( η) and f η (-η)=f η ( η), the derivative

lim σ η →0 ∂E x {E 2 η [ θqt (x + η)]} ∂σ η = lim σ η →0 ∞ -∞ 2F η x -γ σ η ∂F η x-γ σ η ∂σ η f x (x)dx = ∞ -∞ lim σ η →0 2F η x -γ σ η f η x -γ σ η -(x -γ) σ 2 η f x (x)dx =2f x (γ) 0 -∞ F η ( η)f η ( η)(-η)d η - ∞ 0 F η ( η)f η ( η) ηd η =2f x (γ) ∞ 0 [1 -2F η ( η)]f η ( η) ηd η. (51)
Thus, for the quantizer number m>1, f x (γ) =0and F η ( η) > 1/2 over the range of η>0, we find

lim σ η →0 ∂R o m ∂σ η = lim σ η →0 m(m -1)p 2 i ∂C ij ∂σ η [C ii +(m -1)C ij ] 2 = ∞ 0 [1 -2F η ( η)]f η ( η) ηd η × 2m(m -1)f x (γ)p 2 i [C ii +(m -1)C ij ] 2 σ η =0 < 0, (52)
where p 2 i > 0 and C ii +(m -1)C ij > 0 for the positive definite covariance matrix C.

APPENDIX H DERIVATION OF PARAMETER VECTOR ESTIMATION

For the observation data model in Eq. ( 32), the unknown k × 1 parameter vector ϑ is with the k × 1 known mean vector E ϑ (ϑ) and the k × k covariance matrix C ϑ .LettheN × k observation matrix H be represented in row form H =[h ⊤ 1 , h ⊤ 2 ,...,h ⊤ N ] ⊤ with its 1 × k row vectors h n , and assume that the N × 1 noise vector ξ is with the common distribution f ξ for mutually independent samples ξ n for n =1, 2,...,N, we have the scalar observation

x n = h n ϑ + ξ n , (53)
which can be also analyzed by the theory developed in Sec. II with the parameter θ replaced by h n ϑ in Eq. (53). We add m noise components η in to the observation x n , respectively, resulting in m × 1 estimate vector θn = θ1n (x n + η 1n ), θ2n (x n + η 2n ),..., θmn (x n + η mn ) ⊤ .

Then, for the observation x n , the centralized correlation vector between ϑ and θn is

p n =E x n ,η [h n (ϑ -E ϑ (ϑ))( θn -E x n ,η (
⎢ ⎢ ⎢ ⎣ h 1 G 1 C -1 1 θ1 -E x 1 ,η ( θ1 ) h 2 G 2 C -1 2 θ2 -E x 2 ,η ( θ2 ) . . . h N G N C -1 N θN -E x n ,η ( θN ) ⎤ ⎥ ⎥ ⎥ ⎦ . (55)
Assuming that H ⊤ H is invertible, the estimate vector θ is given by

θ =(H ⊤ H) -1 H ⊤ θLC =E(ϑ)+(H ⊤ H) -1 H ⊤ ⎡ ⎢ ⎢ ⎢ ⎣ h 1 G 1 C -1 1 θ1 -E x 1 ,η ( θ1 ) h 2 G 2 C -1 2 θ2 -E x 2 ,η ( θ2 )
. . .

h N G N C -1 N θN -E x n ,η ( θN ) ⎤ ⎥ ⎥ ⎥ ⎦ , (56)
with the MSE of each estimation parameter θi as the i-th diagonal element of covariance matrix E x,η [(ϑ -θ)(ϑθ) ⊤ ] for i =1, 2,...,k.

Fig. 1 .

 1 Fig. 1. Block diagram representation of a linear combination of estimators performing as a noise-enhanced Bayesian estimator. In module 1, mutually independent noise components η i are intentionally added into individual estimators θi , and then each θi is multiplied by optimally tuned weighting coefficient w i in module 2, resulting in the combined Bayesian estimator output θLC .

Fig. 2 .

 2 Fig. 2. (a) MSEs of the designed estimator θLC and the noise-enhanced estimator θNE [7] constructed form the suboptimal MAP estimator θmap in Eq.[START_REF] Kay | Can detectability be improved by adding noise?[END_REF] versus the added noise level σ η . For comparison, the MSEs of the MAP estimator θmap in Eq. (20) and the MMSE estimator θmmse in Eq. (21) are also plotted. Here, the background noise level σ ξ =1and the interval bound a =2for the parameter θ. Specially, for the limiting case m →∞,MSEsof θLC are also plotted in (b) over a suitable range. (c) MSEs of θLC and θNE constructed form the quantizer θqt of Eq. (22) with the threshold γ =0versus σ η . (d) For a combiner of identical θqt , the MSEs of θLC and the MMSE estimator θmmse versus the background noise level σ ξ at a fixed added noise level.

Fig. 5 .

 5 Fig. 5. Diagram of the noise-enhanced FIR filter. The combiner G ℓ ,r e p r esenting module 1 plus module 2 in Fig. 1, processes the input sample x n-ℓ .All outputs of combiners G ℓ and the bias weight ω 0 applied at the filter output ẑn .

Fig. 7 .

 7 Fig. 7. (a) A realization of the input x n ; (b) The saturation estimator ϑ ℓ (x n ) = tanh(5x n ) at the added noise level σ η =0; (c) Outputs of the designed filter ẑn of Eq. (28) at the added noise level σ η =5and for m =10 5 , and the desired signal z n = 2 cos(2πn/N). The other parameter are the same as in Fig.6.

1

 1 Let the subscript m to denote the dimension of a matrix of m × m or a vector of m × 1. The positive definite covariance matrix C m has an unique lower-diagonal-upper decomposition C m = L m D m L ⊤ m , where L m is a unit lower triangular matrix and D m =diag(d 1 ,d 2 ,...,d m ) is a diagonal matrix [44]. Since the determinant det(L m )=1, then det(C m )= det(D m )= m i=1 d m and elements d i =detC i / det C i-1 > 0 for i =2, 3,...,m [44]. Notice the matrix C m = L m D m L ⊤ m can be also partitioned as

  ), it is seen that the covariance matrix C has equivalent elements C 11 = C 22 , C 21 = C 12 and C 1k = C 2k for k =3, 4,...,m. The inverse matrix C -1 = A/ det (C), and the matrix A is the cofactor of C with its elements A ij =(-1) i+j B ij .T h e minor B ij of C ij is obtained by deleting the ith row and the jth column of C. Immediately, we find B 11 = B 22 , B 21 = B 12 and B 1k = -B 2k , yielding A 11 = A 22 , A 21 = A 12 and A 1k = A 2k . Furthermore, for identical estimators θ1 = θ2 , the cross correlation vector p has equivalent elements p 1 = p 2 . Thus, the optimum weight vector w o = C -1 p = Ap/ det (C) has equivalent elements

2 η

 2 lim σ η →0 ∂E x {θE η [ θqt (x + η)]} ∂σ η f ξ (x -θ)dxdθ = θf θ (θ) lim σ η →0 f η ( η)(-η)f ξ (σ η η + γ -θ)d ηdθ = -θf θ (θ)f ξ (γ -θ)dθ f η ( η) ηd η =0,(48)andlim σ η →0 ∂E x {E η [ θqt (x + η)]} ∂σ η = -f x (γ) f η ( η) ηd η =0.

- 1 n

 1 θn ))] = G ⊤ n h ⊤ n with the k × m cross-covariance matrix G n =E x n ,η [(ϑ -E ϑ (ϑ))( θn -E x n ,η ( θn )) ⊤ ],and the m × m covariance matrixC n =E x n ,η θn -E x n ,η ( θn ) θn -E x n ,η ( θn )⊤ . Thus, using the designed Bayesian estimator of Eq. (8), we obtainθLC,n = h n θ =E ϑ (h n ϑ)+p ⊤ n C θn -E x n ,η ( θn ) = h n E ϑ (ϑ)+h n G n C -1 n θn -E x n ,η ( θn ) . (54)These scalar estimators of Eq. (54) can be combined into a vector estimator as

  . 2(c) as a function of σ η . It is seen that the noise benefit to the MSE R o ∞ of θLC is more effective, reducing the initial MSE R o ∞ =0.2925 at σ η =0to the minimum of 0.2498 at the optimal added noise level σ opt

	η	=4.4668, as illustrated in
	Figs.2(b) and (c).

Upon increasing the added noise level σ η and the number m,t h eM S E sR o m of θLC are plotted in Fig

TABLE II MSESOFFIR

 II FILTERSnoise level σ η =0.890, a minimum value of the MSE is achieved as J o m,2 =0.3558 ( * marked in Fig.