Marco P Tucci
email: tucci@unisi.it

David A Kendrick
email: kendrick@eco.utexas.edu

Hans M Amman
email: amman@uu.nl

The Parameter Set in an Adaptive Control Monte Carlo Experiment: Some Considerations 1

Keywords: Active learning, dual control, optimal experimentation, stochastic optimization, time-varying parameters, numerical experiments. JEL Classification: C63, E61

 and Amman and Kendrick (1994, 1995), this paper tracks the source of these outliers to two sources: (1) the use of a zero for the penalty weights on the control variables and (2) the generation of near-zero initial estimate of the control parameter in the systems equations by the Monte Carlo routine. This result leads to an understanding of why both the unsophisticated Optimal Feedback (Certainty Equivalence) and the sophisticated Dual methods do poorly in some Monte Carlo comparisons relative to the moderately sophisticated Expected Optimal Feedback method.

Introduction

One of the outstanding problems in the application of stochastic control methods to economic models is the relative performance of different ways of treating the uncertainties in these models and thus of different methods for solving the models. For example, there has been considerable comparison of Optimal Feedback (OF), Expected Optimal Feedback (EOF) and Dual Control (DC) methods using Monte Carlo runs. 2 The first of these methods is a certainty equivalence approach that ignores all the uncertainties in the model except the additive noise terms in the systems equations. The second method considers additive noise terms as well as parameter uncertainty in the system equations but uses passive learning. The third method considers both additive noise and parameter uncertainty and adds consideration of active learning methods that include perturbation early in time to gain improved estimates of the model parameters later in time.

The comparison of these methods has mostly been done applying Monte Carlo methods to simple one-state, one-control quadratic-linear tracking models such as the [START_REF] Macrae | An adaptive learning role for multiperiod decision problems[END_REF] model and the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] model. 3 However, in some cases these comparisons have encountered a significant number of outliers, viz Amman, Kendrick and Tucci (2007). The source of these outliers was a puzzle -particularly so because they occurred prominently in the least sophisticated (Optimal Feedback) and the most sophisticated (Dual Control) methods but not in the intermediate sophistication method (Expected Optimal Feedback).

In this paper we track the source of this puzzle to the choice of parameters (i.e. the parameter set) in these models. In particular, we find that the use of a zero for the penalty weight on the control variables in the criterion function (the λ parameter) can cause problems. This is true when this weight is used in conjunction with a parameter value near zero for the coefficient in the system equation that is multiplied by the control variable (the b coefficient). If the variance of the b parameter is sufficiently large, then the Monte Carlo routine will generate values of this estimate around zero in a meaningful number of runs. The combination of a zero value for λ and a near zero value for b results in division by a number near zero and thus causes large values for some components of the calculations and therefore outliers. However, this occurs in the Optimal Feedback and the Dual Control methods but not in the Expected Optimal Feedback methods as is discussed below.

The [START_REF] Macrae | An adaptive learning role for multiperiod decision problems[END_REF] and the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] models are identical in struc-ture; however they use different sets of parameter values. In particular the Beck and Wieland model uses a zero for the weight, λ, on the control variables in the criterion function. Consequently it is this model where the outlier problem is pronounced, particularly when the variance of the b parameter is relatively large.

In tracking down the source of the outliers we have made considerable use of the research results on nonconvexities in the cost-to-go function of adaptive control problems. This work begin when the [START_REF] Tse | An actively adaptive control for linear systems with random parameters[END_REF] algorithm for solving adaptive control problems as used by Kendrick (1978) and, Norman, [START_REF] Norman | Multiple Relative Maxima in Optimal Macroeconomic Policy: An Illustration[END_REF] found nonconvexities in the cost-to-go function of a ten-period threeparameter version of the classic [START_REF] Macrae | Linear decision with experimentation[END_REF] problem. However, the computer codes were sufficiently complex that it was difficult to be sure that the phenomena were not caused by programming errors. Therefore, at the time it was uncertain whether nonconvexities were (1) caused by a programming error;

(2) caused by the particular choice of parameters;

(3) fundamental to this class of problems.

Over a decade later [START_REF] Mizrach | Non-convexities in an stochastic control problem with learning[END_REF] and Amman andKendrick (1994, 1995) returned to this problem showing, by analytical methods, that nonconvexities are fundamental in this type of models.

In the [START_REF] Tse | An actively adaptive control for linear systems with random parameters[END_REF] approach for solving active learning stochastic control problems, the total cost-to-go is approximated by a function which can be decomposed into three terms: deterministic, cautionary and probing. Nonconvexities are caused by the probing and/or cautionary components of the cost-to-go. When only the parameter associated with the control variable is unknown, the nonconvexities are determined by the changes in the path of future state variables induced by modifying the value of the control. If more than one parameter is unknown the covariance between the unknown parameters, e.g. the covariance between the unknown parameter associated with the control and that associated with the state, and the relative magnitude of their variances is another possible source of nonconvexities [START_REF] Mizrach | Non-convexities in an stochastic control problem with learning[END_REF]). 4 The situation with only one unknown parameter is further investigated in Amman 4 According to [START_REF] Mizrach | Non-convexities in an stochastic control problem with learning[END_REF]: increases in the control variables ultimately do reduce the parametric uncertainty to zero in the limit ... there will often be at least one region in which increases in the control value raise parametric uncertainty before it begins to decline. This phenomenon can generate nonconvexities, see Mizrach (1991, pages 516, 526-534). Furthermore when the planning time horizon is short initial conditions are important. For instance in the 2 periods case, different values of x 0 affect the variance term in an interesting way, Mizrach (1991, page 534).

and [START_REF] Amman | Nonconvexities in stochastic control problems: An analysis[END_REF][START_REF] Amman | Nonconvexities in stochastic control models[END_REF]. They find that nonconvexities depend upon the magnitude of the initial variance of the unknown parameter, the level of the parameter itself and the variance of the additive noise in the system equation. Moreover, they show that particular combinations of the parameters and of the penalty weights in the objective function can turn the usually convex cautionary term of the cost-to-go into a concave term which, under certain circumstances, may generate a concave cost-to-go. These results are important because they clearly indicate that nonconvexities are fundamental in the mathematics of the problem (Amman and Kendrick (1995, page 456)) and can arise even in very simple adaptive control problems.

These conclusions have substantial computational consequences. When nonconvexities are absent, the stochastic control problem can be solved with gradient methods. However the gradient solution may be a local optimum, rather than a global optimum, when nonconvexities are present. For this reason a globally optimizing algorithm should be generally preferred except in the one control case where a grid search may be equally effective. Indeed as shown in [START_REF] Tucci | The nonconvexities problem in adaptive control models: A simple computational solution[END_REF][START_REF] Tucci | The Rational Expectation Hypothesis, Time-varying Parameters and Adaptive Control[END_REF], nonconvexities may be much more common and subtle than the theoretical results suggest. For example, Amman and Kendrick (1995, page 465) find that when the MacRae (1972) parameter set is used, the nonconvexity appears when the variance of the estimate of the unknown parameter is set to 2. On the other hand [START_REF] Tucci | The nonconvexities problem in adaptive control models: A simple computational solution[END_REF][START_REF] Tucci | The Rational Expectation Hypothesis, Time-varying Parameters and Adaptive Control[END_REF] reports nonconvexities in 28% of the cases of a Monte Carlo experiment with the same parameter set and a variance equal to .5 for the unknown parameter. Therefore the need for a globally optimizing algorithm may be more stringent than generally believed.

One side effect of the line of research discussed above on the origin of nonconvexities is that the analytical results contained in these papers allow one to fully characterize these three components of the cost-to-go for the simplest one-state, one-control, one unknown parameter, quadratic linear adaptive control problem with a time horizon of two periods. It is therefore possible to compare the average or representative cost-to-go associated with different parameter sets and to study the impact of a certain parameter set on the individual runs of a Monte Carlo experiment. 5 The former may help to understand the basic characteristics of a certain parameter set. The latter may be useful to reconcile the theoretical results in [START_REF] Mizrach | Non-convexities in an stochastic control problem with learning[END_REF] and [START_REF] Amman | Nonconvexities in stochastic control models[END_REF] with the computational findings in [START_REF] Tucci | The nonconvexities problem in adaptive control models: A simple computational solution[END_REF][START_REF] Tucci | The Rational Expectation Hypothesis, Time-varying Parameters and Adaptive Control[END_REF] and to shed some light on the outlier problem mentioned in [START_REF] Amman | Solving the Beck and Wieland model with optimal experimentation in DUALPC[END_REF].

In the following a simple, but fairly general, one-state, one-control, one unknown parameter, quadratic linear adaptive control problem is presented first (Section 2). Then the probing, cautionary and deterministic components of its dynamic pro-gramming cost-to-go are characterized (Sections 3 through 5). This is done for the case in which the planning time horizon is 2, following closely Amman andKendrick (1994, 1995). In Section 6 it is shown that this model encompasses as special cases both the MacRae (1972) and [START_REF] Beck | Learning and control in a changing economic environment[END_REF] models and parameter sets and the characteristics of the representative cost-to-go associated with these two problems are discussed. At this point the impact of a certain parameter set on the individual runs of a Monte Carlo experiment is investigated (Section 7). Its impact on the OF and EOF solutions is discussed in Section 8. Conclusions are summarized in Section 9.

The Problem

Consider a simple control problem with one state, one control and a time horizon of N periods in which the policy maker wants to find u 0 , u 1 , . . . , u N -1 to minimize

J = E 1 2 w N (x N -xN ) 2 + 1 2 N -1 k=0 w k (x k -xk ) 2 + λ k (u k -ũk ) 2 (1)
where E is the expectation operator, subject to

x k+1 = αx k + β k u k + γ + ε k+1 for k = 0, 1, ..., N -1 (2)
with x k and u k the state and control variables, respectively. Also α, β k and γ are the parameters of the system equation and ε k+1 is an error term identically and independently distributed (i.i.d.) normal with mean zero and variance q. Finally, the initial state x 0 and the penalty weights w's and λ's are given constants. The parameter associated with the control is assumed time-varying with the following law of motion6

β k+1 = dβ k + (1 -d)β + η k+1 for k = 0, 1, ..., N -1 (3)
where d is the transition parameter, β is the unconditional mean of the stochastic parameter and η k+1 is the additive error term i.i.d. normal with mean zero and variance σ 2 η . Also, the states are measured without error.7

It is worth while to point out that equation (3) includes as special cases several relevant situations. For instance, when d and σ 2 η are zero, β k reduces to the usual time-invariant case. On the other hand, if d = 0 but σ 2 η = 0 equation (3) describes a random parameter, i.e. a parameter varying randomly about the fixed mean β. When β = 0, equation (3) describes the usual vector-autoregressive process of order one with mean zero. 8 Finally, parameters following a random walk are modeled setting d = 1. Alternatively, equation (3) may be used to model the lack of knowledge about the parameters. For instance when the true parameter associated with the control is constant but unknown, setting d = 0 and σ 2 η = 0 in the parameter transition equation allows one to interpret β k in equation ( 2) as the time-varying estimate, based on observations up to time k, of the unknown β used to determine optimal control. In this case σ 2 η should be interpreted as the variance of the estimate based on all information available at time k. 9

Following the [START_REF] Tse | An actively adaptive control for linear systems with random parameters[END_REF] method for solving active learning stochastic control problems, [START_REF] Kendrick | Stochastic control for economic models[END_REF]Kendrick ( , 2002) ) and [START_REF] Tucci | Adaptive control in the presence of time-varying parameters[END_REF][START_REF] Tucci | The Rational Expectation Hypothesis, Time-varying Parameters and Adaptive Control[END_REF] compute the approximate cost-to-go at different values of the control and then choose that value which yields the minimum approximate cost. 10 This approximate cost-to-go can be decomposed into three terms and, for the present problem, written as

J N = J D,N + J C, N + J P, N (4)
where J N is the total cost-to-go with N periods remaining and J D, N , J C, N and J P, N are the deterministic, cautionary and probing components, respectively. In equation (4) the deterministic component includes only terms which are not stochastic. The cautionary term includes uncertainty only in the next time period and the probing term contains uncertainty in all future time periods. Thus the probing term includes the motivation to perturb the controls in the present time period in order to reduce future uncertainty about parameter values. 11

3 The Probing Component Amman andKendrick (1994, 1995) consider the case where the planning horizon is N = 2 and all the parameters in equation ( 2) are constant, but β is unknown.

Tucci (2004, Chapters 2-5).

8 See, e.g., Harvey (1981, Chapter 2). 9 Sometimes the more cumbersome notation σ 2 b,k , with σ 2 b,k ≡ σ 2 η , may be preferred to stress the fact that it is the variance of the estimate of the unknown parameter β based on the information available at time k, when this is the appropriate interpretation. 10 See [START_REF] Kendrick | Stochastic control for economic models[END_REF], 2002, Chapters 9-10) or Tucci (2004, Chapter 2) for details. 11 See Kendrick (1981, pages 97-98)) for an introduction to this decomposition.

Therefore the optimal control is determined using the estimate of β based on all the information available at time 0, i.e. β 0 ≡ b, with a variance equal to σ 2 η ≡ σ 2 b . They show that the probing component takes the form12

J P, 2 = 1 2 g(u 0 ) h(u 0 ) (5)
where g(u 0 ) is a quadratic function reflecting the effect of the control u 0 on future states x 1 and x 2 .13 Also, h(u 0 ) is a quadratic function whose inverse defines the relationship between the control and the updated variance of the unknown parameter b. 14 Consequently the probing component is the ratio of two quadratic functions that can be written as15

h(u 0 ) = 1 σ 2 b q (σ 2 b u 2 0 + q) (6)
and

g(u 0 ) = w 2 2 λ 1 + b 2 w 2 (bu 01 + x 02 -x2 ) 2 (7)
with u 01 and x 02 the nominal, or CE, values of u 1 and x 2 defined as16

u 01 = - 1 λ 1 + b 2 w 2 αb 2 w 2 u 0 + α 2 bw 2 x 0 + αbγw 2 + bw 2 (γ -x2 ) -λ 1 ũ1 (8) x 02 = b α -αb 2 w2 λ1+b 2 w2 u 0 + α 2 x 0 + αγ + γ + b -1 λ1+b 2 w2 [α 2 bw 2 x 0 + αbγw 2 + bw 2 (γ -x2 ) -λ 1 ũ1 ] (9)
Substituting ( 8)-( 9) into ( 7) and simplifying yields

g(u 0 ) = φ 1 (φ 2 u 0 + φ 3 ) 2 (10)
with

φ 1 = w 2 2 λ1+b 2 w2 φ 2 = αb 1 -2b 2 w2 λ1+b 2 w2 φ 3 = 2b -1 λ1+b 2 w2 [α 2 bw 2 x 0 + αbγw 2 + bw 2 (γ -x2 ) -λ 1 ũ1 ] + α 2 x 0 + αγ + γ -x2 (11)
So the probing component of the cost-to-go is the ratio of two quadratic functions in u 0 . 17 From ( 5) with h and g defined as in ( 6) and ( 10)-( 11), respectively, it follows that the probing function in this case is

J P,2 = σ 2 b qφ 1 2 (φ 2 u 0 + φ 3 ) 2 σ 2 b u 2 0 + q (12)
From equation ( 11) φ 1 will be positive regardless of b and u 0 when the λ's and w's are positive. Under this condition the first term in parenthesis on the right-hand side of (12) will be positive provided that the variances σ 2 b and q are nonzero. If these conditions are met, J P,2 is non-negative for all values of u 0 .

Next Amman andKendrick (1994, 1995) consider the roots of the probing term. The first derivative of equation ( 12) with respect to the control is 17 To investigate the behavior of this component of the probing cost, [START_REF] Mizrach | Non-convexities in an stochastic control problem with learning[END_REF] studies its partial derivative with respect to the control at time 0

, i.e. [∂g(u 0 )/∂u 0 ] = 2φ 1 (φ 2 u 0 + φ 3 ) [∂ (φ 2 u 0 + φ 3 ) /∂u 0 ] .
Given that φ 1 is positive regardless of b and u 0 when the λ's and w's are positive, the signs of the term in parenthesis and of the partial derivative appearing on the right-hand side are critical. If both of them are positive an increase in the control increases the volatility of the future states path and, depending upon how fast the variance of the uncertain parameter decreases when the control is increased, this can cause the appearance of nonconvexities in the probing component of the cost-to-go. This derivative is indeed positive when the set of parameters is α

= .7, b = -3.5, c = 3.5, q = -3.5, σ 2 b = 2.5, x 0 = 2, λ 1 = λ 2 = w 1 = w 2 = 1 , x0 = x1 = x2 =
0 and ũ0 = ũ1 = ũ2 = 0 . In this case movements in the control variable produce changes in g(u 0 ) large enough to overcome the drop in the updated variance of b caused by the same movement in the control and give rise to nonconvexities in the probing component [START_REF] Mizrach | Non-convexities in an stochastic control problem with learning[END_REF].

∂J P,2 ∂u 0 = σ 2 b qφ 1 (φ 2 2 u 0 + φ 2 φ 3 ) (σ 2 b u 2 0 + q) -σ 2 b u 0 (φ 2 u 0 + φ 3 ) 2 (σ 2 b u 2 0 + q) 2 (13)
Since both the term outside the square brackets in the numerator of equation ( 13) and the denominator are strictly positive as long as the λ's, w's, σ 2 b and q are positive, they focus on the condition under which the bracketed term would be zero. Expansion of the term in brackets yields

-φ 2 φ 3 σ 2 b u 2 0 + φ 2 2 q -σ 2 b φ 2 3 u 0 + φ 2 φ 3 q (14)
This expression is in general a quadratic function in u 0 with roots

r 1 = - φ 3 φ 2 and r 2 = φ 2 q φ 3 σ 2 b (15)
So the first derivative of the probing function with respect to the control in period zero has two roots. In other words the probing function has two extreme points.

At this point Amman andKendrick (1994, 1995) evaluate the function at these two extremes. Substitution of the first root r 1 into equation ( 12) yields J P,2 = 0. To evaluate the function at the second root they substitute r 2 into (12) to obtain

J P,2 = σ 2 b qφ1 2 φ 2 2 q+φ 2 3 σ 2 b φ3σ 2 b 2 φ 2 2 q 2 +φ 2 3 σ 2 b q φ 2 3 σ 2 b -1 A B C (16)
If the variance of the parameter, σ 2 b , and the variance of the additive error term, q, are nonzero then the term A will be strictly positive given the fact from equation (11) that φ 1 is positive. Term B is a square so it will be strictly positive provided that either of the variances are non-zero. Term C will also be positive under the same restriction. Therefore the probing function J P,2 in general will be strictly positive at the second root provided that σ 2 b and q are non-zero.

To check the limit of the probing term as u 0 goes to either plus or minus infinity use (5) with h and g defined as in ( 6) and ( 10)-( 11), respectively, to obtain

lim u0→±∞ J P,2 = σ 2 b qφ 1 φ 2 2 2σ 2 b = qφ 1 φ 2 2 2 (17)
The right hand side of ( 17) is positive so long as q is nonzero so the probing term approaches a positive limit when u 0 approaches either plus or minus infinity. The probing cost function J P, 2

Summarizing the probing function ( 5) with h and g defined as in ( 6) and ( 10)-( 11), respectively, has the general form shown in Figure 1, page 10. 18 The function is non-negative at all values of u 0 . The first derivative of the function has two roots which correspond to the maximum and minimum shown. At the minimum the value of the function is zero. The limits of the function as u 0 approaches plus or minus infinity are strictly positive and identical. The intuition associated with this function is that the probing component is decreased by actions which make u 0 large in either the positive or negative directions. Thus the perturbations to the system which come from making u 0 large in absolute value decrease the variance of the parameters in future time periods and therefore decrease the total cost-to-go.

The Cautionary Component

Following Amman and Kendrick (1994, equation 5.17), the cautionary component of the approximate cost-to-go can be written as

J C,2 = σ 2 b w2 2 (αu 0 + u 01 ) 2 + σ 2 b 2 -1 λ1+b 2 w2 (αbw 2 u 0 + bw 2 u 01 + w 2 x 02 -w 2 x2 ) 2 + q 2 α 2 w 2 + w 2 + w 1 + -1 λ1+b 2 w2 (αbw 2 ) 2 + σ 2 b w1 2 u 2 0 ( 18
)
with u 01 and x 02 defined as in ( 8) and ( 9), respectively. As is shown in Appendix A, when these quantities are replaced by their definitions, equation ( 18) takes the form19

J C,2 = δ 1 u 2 0 + δ 2 u 0 + δ 3 (19)
with

δ 1 = σ 2 b 2 ν 2 1 w 2 - 4b 2 w 2 2 λ1+b 2 w2 + w 1 δ 2 = σ 2 b w 2 ν 1 ν 2 - 2bw2(2bν 2 +ν 3 ) λ1+b 2 w2 δ 3 = σ 2 b 2 w 2 ν 2 2 - w2(2bν 2 +ν 3 ) 2 λ1+b 2 w2 + q 2 α 2 w 2 + w 2 + w 1 -(αbw2) 2 λ1+b 2 w2 ( 20
)
where

ν 1 = α 1 -b 2 w2 λ1+b 2 w2 ν 2 = -1 λ1+b 2 w2 [α 2 bw 2 x 0 + αbγw 2 + bw 2 (γ -x2 ) -λ 1 ũ1 ] ν 3 = α 2 x 0 + αγ + γ -x2 (21)
This component of the cost-to-go is convex if δ 1 > 0. When the penalty weights w 1 and w 2 are positive

1 - 4b 2 w 2 λ 1 + b 2 w 2 = λ 1 -3b 2 w 2 λ 1 + b 2 w 2 > 0 (22)
or, alternatively, λ 1 > 3b 2 w 2 is a sufficient condition for δ 1 to be positive. 20 In summary, even if the cautionary term is concave, the cost-to-go will usually be convex because the deterministic term is always convex and is frequently larger in magnitude than the cautionary term. Only when the parameter variance gets large the concave cautionary term can overcome the deterministic oneto produce a concave cost-to-go (see Amman and Kendrick, 1995, page 466).

5 The Deterministic Component and the Total Cost-to-Go

Finally Amman and Kendrick (1994, equation 6.8) write the deterministic component as

J D,2 = λ 0 2 (u 0 -ũ0 ) 2 + w 2 2 (x 02 -x2 ) 2 + w 1 2 (x 01 -x1 ) 2 + λ 1 2 (u 01 -ũ1 ) 2 (23)
with u 01 and x 02 defined as in ( 8) and ( 9), respectively and

x 01 = αx 0 + bu 0 + γ .
Replacing these quantities in equation ( 23), squaring and rearranging the terms yields a convex quadratic function of the initial control value u 0 which can be written

J D,2 = ψ 1 u 2 0 + ψ 2 u 0 + ψ 3 (24)
with

ψ 1 = λ0 2 + w2 2 (bν 1 ) 2 + w1 2 b 2 + λ1 2 -1 λ1+b 2 w2 2 (αb 2 w 2 ) 2 ψ 2 = -λ 0 ũ0 + w 2 bν 1 (bν 2 + ν 3 ) + w 1 (αx 0 + γ -x1 ) b + -λ1 λ1+b 2 w2 αb 2 w 2 (ν 2 -ũ1 ) ψ 3 = λ0 2 ũ2 0 + w2 2 (bν 2 + ν 3 ) 2 + w1 2 (αx 0 + γ -x1 ) 2 + λ1 2 (ν 2 -ũ1 ) 2 . ( 25
)
Then, as in equation ( 4), the total cost-to-go for period k = 0, i.e. with two periods remaining, can be written as

J 2 = J D,2 + J C,2 + J P,2 (26)
Substitution of ( 12), ( 19) and ( 24) into (26) yields 21

J 2 = (ψ 1 + δ 1 ) u 2 0 + (ψ 2 + δ 2 ) u 0 + (ψ 3 + δ 3 ) + σ 2 b q 2 φ 1 (φ 2 u 0 + φ 3 ) 2 (σ 2 b u 2 0 + q) (27)
6 The Representative Cost-to-Go

The model analyzed in the previous sections encompasses as special cases both the MacRae (1972) and [START_REF] Beck | Learning and control in a changing economic environment[END_REF] models and parameter sets. 22 It is therefore possible to characterize the representative cost-to-go of these two popular problems. The parameter set used in MacRae ( 1972) is

α = .7 β 0 ≡ b = -.5 γ = 3.5 σ 2 ε = q = .2 w k = 1 ∀k λ k = 1 ∀k σ 2 η ≡ σ 2 b = .5 x 0 = σ 2 α = σ 2 γ = 0 (28)
and the desired path for the state and control is set equal to zero. Then, as already noticed in Amman andKendrick (1994, 1995), the coefficients of the probing cost are

φ 1 = 1 1 + b 2 = .8 φ 2 = αb 1 -b 2 1 + b 2 = -.21 (29) φ 3 = (αγ + γ) 1 -b 2 1 + b 2 = 3.57
and this function reaches its minimum 0 at r 1 = 17 and its maximum J P,2 = 2.553 at r 2 = -.023529. The limits of the probing cost function as u 0 approaches plus 21 As noticed in Amman andKendrick (1994, 1995), the derivative of this function is a complex function of u 0 and the function has several extreme points. However by taking the second derivative of the cost-to-go at

u 0 = 0 it yields ∂ 2 J 2 ∂u 2 0 u0=0 = 2ψ 1 + 2δ 1 + σ 2 b ψ 1 ψ 2 2 -σ 4 b φ 1 φ 2 3 q .
Then they conclude that a higher variance, as well as a lower value of q, usually makes nonconvexities more likely. 22 The Excel spreadsheet that we have used for many of the calculations and the creation of the graphs in this and the following section is available from Marco Tucci (tucci@unisi.it).

or minus infinity are equal to J P,2 = .0035 and it has the general form shown in Figure 1, page 10.

The coefficients for the cautionary component of the approximate cost-to-go are

ν 1 = α 1 1 + b 2 = .56 ν 2 = - 1 1 + b 2 b (αγ + γ) = 2.38 (30)
ν 3 = αγ + γ = 5.95 therefore δ 1 = σ 2 b 2 ν 2 1 1-3b 2 1+b 2 + 1 = .2657 δ 2 = σ 2 b ν 1 ν 2 - 2b(2bν 2 +ν 3 ) 1+b 2
= 1.4661

δ 3 = σ 2 b 2 ν 2 2 - (2bν 2 +ν 3 ) 2 1+b 2 + q 2 α 2 + 2 -(αb) 2 1+b 2 = -.8937 (31)
and the cautionary component is convex. Finally the deterministic component

J D,2 = ψ 1 u 2 0 + ψ 2 u 0 + ψ 3 (32)
has coefficients

ψ 1 = 1 2 1 + (bν 1 ) 2 + b 2 + -1 1+b 2 2 (αb 2 ) 2 = .674 ψ 2 = bν 1 (bν 2 + ν 3 ) + γb + -1 1+b 2 αb 2 ν 2 = -3.416 ψ 3 = 1 2 (bν 2 + ν 3 ) 2 + 1 2 γ 2 + 1 2 ν 2 2 = 20.286 . ( 33
)
Using these values the representative cost-to-go for the MacRae (1972) problem can be plotted as in Figure 2, page 15, with the flat looking portion around control 0 due to the fact that in that portion the cost-to-go decreases at a slower pace. The optimal cost in Figure 2, page 15, is 18.8605 at u 0 = 1.32691.23

Next we shift from the MacRae (1972) model to the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] model. The parameter set for that model is

α = 1. β 0 ≡ b = -.5 γ = 0.0 σ 2 ε = q = 1 .0 w k = 1 ∀k λ k = 0 ∀k σ 2 η ≡ σ 2 b = .25 σ 2 α = σ 2 γ = 0 (34)
and the desired paths for the state and control variables are set equal to zero. Also the initial state is x 0 = 1 as set in [START_REF] Amman | Solving the Beck and Wieland model with optimal experimentation in DUALPC[END_REF]. In this case the coefficients of the probing component are

φ 1 = 1 b 2 = 4, φ 2 = αb (-1) = .5, φ 3 = -x 0 = -1 (35)
and this function reaches its minimum 0 at r 1 = -2 and its maximum J P,2 = 1 at r 2 = 2. The limits of the probing cost function as u 0 approaches plus or minus infinity are equal to J P,2 = .5 and, again, it has the general form shown in Figure 1, page 10. From equation ( 35) it follows that setting no penalty on the use of the control variable, i.e. λ k = 0, makes the probing component of the cost-togo extremely sensitive to the value of b. Thus in equation ( 11) with w 2 = 1, the parameter φ 1 reduces to the inverse of b 2 . Furthermore assuming a null intercept in equation ( 11) with w 2 = 1, i.e. γ = 0, makes φ 3 solely dependent on the initial state.

The coefficients for the cautionary component of the approximate cost-to-go are

ν 1 = α 1 - b 2 b 2 = 0, ν 2 = - 1 b 2 α 2 bx 0 = - 1 b = 2, ν 3 = α 2 x 0 = 1 (36) therefore δ 1 = σ 2 b 2 ν 2 1 1 -4b 2 b 2 + 1 = .125 δ 2 = σ 2 b ν 1 ν 2 - 2(2bν 2 +ν 3 ) b = 0 δ 3 = σ 2 b 2 ν 2 2 - (2bν 2 +ν 3 ) 2 b 2 + q = q = 1 (37)
and the cautionary component is convex. From equation ( 37) it is apparent that δ 3 is always equal to q in this model. Moreover these coefficients appear to be independent of the value of b with δ 2 always equal to zero. Finally the deterministic component

J D,2 = ψ 1 u 2 0 + ψ 2 u 0 + ψ 3 (38)
has coefficients

ψ 1 = 1 2 (bν 1 ) 2 + 1 2 b 2 = .125 ψ 2 = bν 1 (bν 2 + ν 3 ) + αx 0 b = -.5 ψ 3 = 1 2 (bν 2 + ν 3 ) 2 + 1 2 (αx 0 ) 2 = .5 (39)
and the representative cost-to-go function for the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] model can be plotted as in Figure 3, page 15. The optimum cost in Figure 3, page 15, is 1.32 at u 0 = 1.25.

The Parameter Set and Monte Carlo Experiments

The analysis carried out in the last section focuses on the shape of the representative cost-to-go. Therefore it sheds little light on the results obtained in each run of a Monte Carlo experiments carried out using the MacRae (1972), or [START_REF] Beck | Learning and control in a changing economic environment[END_REF], models and parameter sets. In a typical Monte Carlo experiment the initial estimate of the unknown parameter β, i.e. b, in each run is drawn from a distribution having the true value of the unknown parameter as mean and the es-timated variance σ 2 b as variance (Figure 4, page 18).24 This means that, when the MacRae problem is used, the actual value for b is between -1.2 and 0.2 in approximately 68% of the Monte Carlo runs, in the interval (-1.9, 0.9) approximately 95% of times and over 99% of time between -2.6 and 1.6. 25 The same intervals for the Beck and Wieland problem are, respectively, (-1.0, 0.0), (-1.5, 0.5) and (-2.0, 1.0).

It is therefore useful to study the shape of the approximate cost-to-go when the parameter b takes on values in these different intervals. Even though the probability that b takes on a specific value is zero, as is well known, the probability that it falls in a certain interval, no matter how small, is not. Amman and Kendrick (1995, page 470) consider the effects of changes in the b parameter and they notice that for the MacRae problem the cost-to-go function will be convex for values of b either substantially above or below zero but will be nonconvex for values of b close to zero. However they do not explicate the relationship between this result and the outcome of Monte Carlo experiments. This is the main task of the present section.

In the MacRae problem, as originally suggested in [START_REF] Amman | Nonconvexities in stochastic control models[END_REF], the approximate cost-to-go shows a nonconvexity when b is close to 0. Indeed a nonconvexity appears in all those runs where the value of b is between -0.45 and 0.45 and it becomes more pronounced as b gets closer to 0 (Figure 5, page 19, and Figure 6, page 19). 26 The probability for b to fall in the interval (-0.45, 0.45) in this problem is equal to the probability for the standard normal to be in the interval (0.07143, 1.3571). Therefore in a Monte Carlo experiment based on the MacRae (1972) problem nonconvexities will be encountered in about 38% of the runs which is even higher than the result, namely 28%, reported in Tucci (1998, page 216) and Tucci (2004, page 105). When the value for b is outside the interval (-0.45, 0.45) the cost-to-go appears as in Figure 7, page 19. Some relevant information about a certain set of parameters may be gained studying the relationship between the stochastic parameter b and the optimal control, for the first period, and its associated cost-to-go. This can be done, for each value of b, by computing the coefficients of the approximated cost-to-go, see Appendix B, and selecting the optimal control. When b is in the interval (-3.0, -0.02), the optimal control is always between 1.00 and 2.00 with a cost ranging from 2.08 to 19.16 (Figure 8, page 20).27

In the Beck and Wieland problem, it follows from equation ( 35) that when b = 0 the parameter φ 1 is infinity and as reported in Appendix B most of the relevant coefficients of the approximate total cost-to go are not available because they include a division by zero term. When the parameter b happens to be around 0, say 0.02 as in Figure 9, page 21, φ 1 is extremely high and the approximate total cost-to go takes on almost completely the shape of its probing component, see Appendix B. Also recall that in the Beck and Wieland model the weight λ 1 on the control variable deviation in the criterion function is set to zero. When this is combined with a Monte Carlo run in which the coefficient b on the control variable term in the systems equations is near zero as was discussed above, then from equation ( 11) the coefficient

φ 1 = w 2 2 λ 1 + b 2 w 2
in the probing term of the cost-to-go becomes very large. As Figure 9, page 21, shows the probing term is large enough in this case to make the entire cost-to-go function nonconvex.

The intuition here is that the probing component in the cost-to-go provides a mea- Relationship between b and the optimal control in the first time period, and its approximate cost-to-go, for the MacRae ( 1972) problem.

sure of future cost depending on the amount of probing. If there is not enough probing to move the state variable x 1 around, the cost is high because the unknown parameter value is not learned effectively and its variance remains high. Thus there is an incentive, as Figure 9, page 21, shows, for the control to be made large in either the positive or negative direction in order to learn the parameter and reduce the cost. However, when b is very small it takes large movements in the control variable u 0 to bring about shifts in the state variable x 1 in order to learn the coefficient b and reduce its variance. Also, in the Beck and Wieland model, with λ 1 equal to zero there is no cost in the criterion function associated with using large positive or negative values of the control. Thus there is a need, when b is near zero, for large values of the control in order to learn and there is no constraint from the criterion cost. These two factors together bring about the shape of the cost-to-go function shown in Figure 9, page 21.

Moreover it should be noticed that in the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem the cautionary cost is constant in the various runs, i.e. is independent of b, as indicated by the δ coefficients reported in Table B.2, page 33, of Appendix B. When the absolute value of b is in the interval (0.03, 0.19) the cost-to-go function has a sombrero like shape as in Figure 10, page 21. 28 The nonconvexity is very pronounced for values of b close to the lower end of the interval and it becomes less and less prominent as b moves toward the higher end of the interval. For b = 0.2 the costto-go appears as in Figure 11, page 21. The flat looking portion around a control equal to 1 is due to the fact that in that portion the cost-to-go increases at a slower pace. Namely, when the control at time zero is increased from 0.5 to 0.75, 1, 1.25, 1.5, 1.75 and 2 the approximate cost-to-go increases from 5.20 to 5.35, 5.45, 5.49, 5.51, 5.52 and 5.54 respectively.

The probability for b to fall in the interval (-0.2, 0.2) in the Beck and Wieland problem is equal to the probability for the standard normal to be in the interval (0.6, 1.4). Therefore, in a Monte Carlo experiment based on this model and parameter set, from mild to severe nonconvexities will be encountered in about 20% of the runs. Half of them will be severe and around 2% will be associated with a cost-togo as in Figure 9, page 21. When the value for b is outside the interval (-0.2, 0.2) the cost-to-go appears as in Figure 12, page 21. When the value for b is outside the interval (-0.2, 0.2) the cost-to-go appears as in Figure 12, page 21.

The probability that the value of b used in the Monte Carlo run falls in the intervals (-0.1, 0.1) and (-0.02, 0.02) in the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem is equal to the probability for the standard normal to be in the intervals (0.8, 1.2) and (.96, 1.04), respectively.

Summarizing (as shown in Figure 13, page 23) when b is in the interval (-3.0, -0.1), the optimal control goes from .25 to 3.75 with a cost ranging from 1.04 to 4.03.29 As b gets closer to 0 both the optimal control and its associated cost skyrocket. At b = -.02 the former is 9.25 and the latter is 21.30.

The discussion of this section reveals that the variance of the unknown parameter estimate plays a dual role in making nonconvexities more or less likely to occur in a certain Monte Carlo experiment. On one hand it enters the coefficients of the approximate cost-to-go and its second order derivative with respect to the control, as pointed out in Amman andKendrick (1994, 1995), effecting directly the shape of the cost function. On the other hand it affects the probability with which values of the parameter drawn from certain intervals of the parameter space are used. Therefore the role of σ 2 b is more subtle then so far expected. Nonconvexities may appear frequently even in cases where the representative run shows no indication of it and a smaller variance does not necessarily imply fewer nonconvexities. For instance, in the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem when σ 2 b = 1.25 the probability that b is drawn from the interval (-0.02, 0.02) is approximately 1%, instead of 2% as with the original data set where σ 2 b = 0.25. 30 This means that cost functions similar to that reported in Figure 9, page 21, will be encountered less frequently with this higher variance. This completes the discussion of the main line of argument of this paper about Relationship between b and the optimal control in the first time period, and its approximate cost-to-go, for the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem.

the effect of parameter sets on the adaptive or dual control solutions. However, before closing it is appropriate to add a short additional section on the effect of the parameter set selected for a certain numerical experiment on the optimal feedback (OF) and expected optimal feedback (EOF) solutions since it is anticipated that comparisons of these two methods may occur with some frequency in the future.

8 The Importance of Penalty Weights on the OF and EOF Solutions Suppose that the optimal feedback (OF) control, the expected optimal feedback (EOF) control and Dual control (DC) are compared. 31 Dual control minimizes the approximate cost-to-go defined in equation ( 27). The EOF control is determined using the feedback rule 31 See [START_REF] Kendrick | A classification system for economic stochastic control models[END_REF] for a description of the three different methods.

u 0 = G 0 x 0 + g 0 (40)
with

G 0 = -[λ 0 + K 1 (b 2 + σ 2 b )] -1 (αK 1 b) g 0 = -[λ 0 + K 1 (b 2 + σ 2 b )] -1 (bK 1 γ + bp 1 ) (41)
and the Riccati quantities K 1 and p 1 defined as

K 1 = w 1 + α 2 K 2 -(αbK 2 ) λ 1 + K 2 b 2 + σ 2 b -1 (αbK 2 ) p 1 = (αγK 2 ) + αp 2 -(αbK 2 ) λ 1 + K 2 b 2 + σ 2 b -1 [bγK 2 + bp 2 ] ( 42
)
where the terminal conditions are given by K 2 = 1 and p 2 = 0.32 The OF control is obtained using equations ( 40)-( 42) with σ 2 b = 0. When the parameter b is very close to zero, say b = 0.02, the EOF control is equal to 0, OF control is infinity and the DUAL control diverges toward very high or low values trying to minimize a function like that depicted in Figure 9 on page 21. Consequently, in a Monte Carlo experiment based on the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem both OF and Dual control will exhibit around 10% of very unfavorable runs and around 2% of severe outliers compared with EOF. As indicated in the sixth column of Table 1, page 32, the EOF control is not too affected by a parameter value for b close or equal to 0 in the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem. It goes from a maximum of 1 to a minimum of minus 1 for the selected b's.

Completely different is the situation for the OF control which, for the set of selected values of b, ranges from infinity to -100 as reported in the last column of Table 1, page 32. The situation looks more balanced when the MacRae problem is used. As reported in Table 2, page 33, the difference between EOF and OF control is not too big and as shown in Figure 6, page 19, the shape of the cost-to-go is not such as to cause the Dual algorithm to diverge.

Conclusion

As work on the application of various stochastic control methods to economic models proceeds Monte Carlo and similar stochastic methods will probably be used to compare the performance of the various methods on simple models. When this is done outliers of the type we have encountered in doing such studies on the MacRae (MacRae (1972)) and the [START_REF] Beck | Learning and control in a changing economic environment[END_REF]) models are likely to occur. We have traced a major cause of these outliers to the choice of parameter sets and in particular to the weight on the control term in the criterion function and the estimate of the unknown parameter that is multiplied by the control variable. When λ = 0 and b is drawn near zero in the Monte Carlo runs the cost-to-go function is likely to be nonconvex and the criterion value may be large.

Therefore, in conducting Monte Carlo experiments to compare various methods of solving stochastic control methods in quadratic-linear tracking models it is important to be careful in the choice of parameter sets. We would recommend that whenever possible that calculations similar to those we have performed here be done before large Monte Carlo runs are done. As indicated above, we are happy to provide to interested users the spreadsheet we have used to make the calculations in Sections 7 and 8 in this paper and which can easily be modified for other parameter sets in one-state, one-control, one uncertain parameter cases. Expected optimal feedback (EOF) control and optimal feedback ((OF) control for the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] 20) -the cautionary cost term. To do so begin with equation ( 18), namely

J C,2 = σ 2 b w2 2 (αu 0 + u 01 ) 2 + σ 2 b 2 -1 λ1+b 2 w2 (αbw 2 u 0 + bw 2 u 01 + w 2 x 02 -w 2 x2 ) 2 + q 2 α 2 w 2 + w 2 + w 1 + -1 λ1+b 2 w2 (αbw 2 ) 2 + σ 2 b w1 2 u 2 0 , (A-1)
and replacing u 01 and x 02 by their definition, that is ( 8) and ( 9) respectively, it yields

J C,2 = σ 2 b w2 2 αu 0 + -1 λ1+b 2 w2 [αb 2 w 2 u 0 + α 2 bw 2 x 0 + αbγw 2 + bw 2 (γ -x2 ) -λ 1 ũ1 ] 2 + σ 2 b 2 -1 λ1+b 2 w2 αbw 2 u 0 + bw 2 -1 λ1+b 2 w2 [αb 2 w 2 u 0 + α 2 bw 2 x 0 + αbγw 2 + bw 2 (γ -x2 ) -λ 1 ũ1 ] + w 2 b α -αb 2 w2 λ1+b 2 u 0 + α 2 0 + αγ γ -1 λ1+b w2 (α 2 bw 2 x 0 αbγw 2 + bw 2 γ -bw 2 x2 -λ 1 ũ1 ) -w 2 x2 2 + q 2 α 2 w 2 + w 2 + w 1 + -1 λ1+b 2 w2 (αbw 2 ) 2 + σ 2 b w1 2 u 2 0 . (A-2) or J C,2 = σ 2 b w2 2 αu 0 - 1 λ1+b 2 w2 αb 2 w 2 u 0 - 1 λ1+b 2 w2 [α 2 bw 2 x 0 + αbγw 2 + bw 2 (γ -x2 ) -λ 1 ũ1 ] 2 + σ 2 b 2 -1 λ1+b 2 w2 αbw 2 u 0 + bw 2 -1 λ1+b 2 w2 αb 2 w 2 u 0 +bw 2 -1 λ1+b 2 w2 [ α 2 bw 2 x 0 + αbγw 2 +bw 2 (γ -x2 ) -λ 1 ũ1 ] + w 2 b α -αb 2 w2 λ1+b 2 w2 u 0 + α 2 x 0 + αγ + γ +b -1 λ1+b 2 w2 (α 2 bw 2 x 0 + αbγw 2 + bw 2 γ -bw 2 x2 -λ 1 ũ1 ) -w 2 x2 2 + q 2 α 2 w 2 + w 2 + w 1 + -1 λ1+b 2 w2 (αbw 2 ) 2 + σ 2 b w1
2 u 2 0 .

(A-3)

Use the quantities ν 1 , ν 2 and ν 3 as defined in equation ( 21), i.e. This appendix contains some of the coefficients of the cost-to-go function, at selected values of b, used to derive Figure 8, page 20, and Figure 13, page 23. Table B.1, page 32, contains the coefficients for the MacRae (1972) problem. It is worthwhile to point out that the sign of b affects only the sign of the linear coefficients in the approximate cost-to-go, namely φ 2 , δ 2 and ψ 2 . Table B.2, page 33, contains the coefficients for the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem. For b = 0 most of the relevant coefficients of the approximate total cost-to go are not available because they include a division by zero term. When the parameter b happens to be around 0, say 0.02 as in Figure 9, page 21, is extremely high and the approximate total cost-to go takes on almost completely the shape of its probing component. Coefficients of the cost-to-go function for the [START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem with

  Figure 1.

Figure 2 .

 2 Figure 2. Approximate cost-to-go in the first time period for the MacRae (1972) model in the Monte Carlo representative run, i.e. when b is -0.5.

Figure 3 .

 3 Figure 3. Approximate cost-to-go in the first time period for the Beck and Wieland (2002) model in the Monte Carlo representative run, i.e. when b is -0.5.

Figure 4 .

 4 Figure 4. Probability density function of the unknown parameter b with the Beck and Wieland (2002) parameter set (NORMAL.25), with the MacRae (1972) parameter set (NORMAL.5) and with the Amman et al. (2007) parameter set (NORMAL1.25)

Figure 5 .

 5 Figure 5. Approximate cost-to-go in the first time period for the MacRae (1972) problem in the Monte Carlo run where b is 0.4.

Figure 6 .

 6 Figure 6. Approximate cost-to-go in the first time period for the MacRae (1972) problem in the Monte Carlo run where b is 0.0.

Figure 7 .

 7 Figure 7. Approximate cost-to-go in the first time period for the MacRae (1972) problem in the Monte Carlo run where b is 0.9. The optimal control is -1.75 with a cost of 14.127.

  Figure 8.

Figure 9 .

 9 Figure9. Approximate cost-to-go in the first time period for the[START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem in the Monte Carlo run where b is 0.02.

Figure 10 .

 10 Figure10. Approximate cost-to-go in the first time period for the[START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem in the Monte Carlo run where b is 0.1.

Figure 11 .

 11 Figure 11. Approximate cost-to-go in the first time period for the Beck and Wieland (2002) problem in the Monte Carlo run where b is 0.2.

Figure 12 .

 12 Figure12. Approximate cost-to-go in the first time period for the[START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem in the Monte Carlo run where b is 0.5.

  Figure 13.

  parameter set with x 0 = 1 at various values of b (Note: n.a.= not available because dividing by zero)

B

  Appendix: The Coefficients of the Cost-To-Go Function for Selected Values of b

  n.a.= not available because dividing by zero).

Table 1

 1

  Intermediate Steps in the Derivation of the Cautionary and Deterministic Components of the Cost-To-GoFirst derive equation (19) and equation (

	Table 2 A Appendix: of b EOF b CASE K 1 p 1 G 0 g 0 u 0 OF K 1 CASE p 1 G 0 g 0 u 0 Expected optimal feedback (EOF) control and optimal feedback ((OF) control for the MacRae (1972) parameter set at various values	-2.60 1.0890 0.4449 0.2225 1.2426 1.2426 1.0631 0.3157 0.2363 1.2820 1.2820	-1.90 1.1438 0.7192 0.2668 1.5739 1.5739 1.1063 0.5315 0.2946 1.6754 1.6754	-1.20 1.2500 1.2500 0.3066 1.9708 1.9708 1.2008 1.0041 0.3696 2.2895 2.2895	-0.50 1.4200 2.1000 0.2407 1.7119 1.7119 1.3920 1.9600 0.3614 2.5341 2.5341	-0.25 1.4704 2.3520 0.1408 1.0260 1.0260 2.3059 0.2343 1.6998 1.6998	-0.20 1.4773 2.3864 0.1150 0.8407 0.8407 1.4712 2.3558 0.1945 1.4175 1.4175	0.00 1.4900 2.4500 0.0000 0.0000 0.0000 1.4900 2.4500 0.0000 0.0000 0.0000	0.20 1.4773 2.3864 -0.1150 -0.8407 -0.8407 1.4712 2.3558 -0.1945 -1.4175 -1.4175	0.25 1.4704 2.3520 -0.1408 -1.0260 -1.0260 1.4612 2.3059 -0.2343 -1.6998 -1.6998	0.30 1.4623 2.3113 -0.1649 -1.1965 -1.1965 1.4495 2.2477 -0.2693 -1.9429 -1.9429	0.90 1.3182 1.5909 -0.3046 -2.0478 -2.0478 1.2707 1.3536 -0.3945 -2.5728 -2.5728	1.60 1.1810 0.9052 -0.2867 -1.7473 -1.7473 1.1376 0.6882 -0.3257 -1.9098 -1.9098

Table B . 1

 B1 Coefficients of the cost-to-go function for the[START_REF] Macrae | Linear decision with experimentation[END_REF] problem at various values of

		ψ 2 ψ 3	-8.406	10.495	-8.367 9.965	-6.248 13.380	-3.416 20.286	-1.855 22.785	-1.501 23.145	0 23.826	1.501 23.145	2.196 22.365	2.836 21.385	5.221 15.905	7.472 11.097
	b
		δ 2 δ 3 ψ 1	-0.044 0.571 4.093		-0.025 1.099 2.497	0.268 2.243 1.365	1.466 -0.894 0.674	1.275 -5.749 0.546	1.096 -6.677 0.529	0 -8.602 0.500	-1.096 -6.677 0.529	-1.404 -4.744 0.565	-1.516 -2.706 0.614	-0.692 2.361 1.015	-0.032 1.524 1.956
		φ 2 φ 3 δ 1	1.351 -4.416 0.245		0.753 -3.369 0.238	0.151 -1.073 0.222	-0.210 3.570 0.266	-0.154 5.250 0.333	-0.129 5.492 0.346	0 5.950 0.373	0.129 5.492 0.346	0.175 4.967 0.319	0.203 4.309 0.291	0.066 0.625 0.220	-0.491 -2.607 0.232
		φ 1	0.129		0.217	0.410	0.800	0.941	0.962	1	0.962	0.917	0.862	0.552	0.281
		ν 2 ν 3	1.994 5.95		2.452 5.95	2.926 5.95	2.380 5.95	1.400 5.95	1.144 5.95	0 5.95	-1.144 5.95	-1.638 5.95	-2.052 5.95	-2.959 5.95	-2.674 5.95
		ν 1	0.090		0.152	0.287	0.560	0.659	0.673	0.7	0.673	0.642	0.603	0.387	0.197
	Table B.2	b	-2.60		-1.90	-1.20	-0.50	-0.25	-0.20	0.00	0.20	0.30	0.40	0.90	1.60

For a classification of various methods of solving stochastic control models see[START_REF] Kendrick | A classification system for economic stochastic control models[END_REF].

This work is a part of the Methods Comparison Project that seeks to compare various ways of solving economic stochastic control models. For examples of the various approaches under consideration see[START_REF] Cosimano | Optimal experimentation and the perturbation method in the neighborhood of the augmented linear regulator problem[END_REF],[START_REF] Cosimano | An algorithm for approximating optimal experimentation problems using the perturbation method[END_REF],[START_REF] Amman | Solving the Beck and Wieland model with optimal experimentation in DUALPC[END_REF] and[START_REF] Beck | Learning and control in a changing economic environment[END_REF].

By parameter set is here intended both the value of the parameters and their covariances and the values used for the penalty matrices, desired paths for the states and controls and the initial state.

[START_REF] Tucci | Adaptive control in the presence of time-varying parameters[END_REF][START_REF] Tucci | The nonconvexities problem in adaptive control models: A simple computational solution[END_REF]) uses a similar model for the parameter associated with the control variable.[START_REF] Tucci | The Rational Expectation Hypothesis, Time-varying Parameters and Adaptive Control[END_REF] and[START_REF] Tucci | Expected optimal feedback with time-varying parameters[END_REF] consider the case where all the parameters may be time-varying.

This is equivalent to setting H = I and R = 0 inKendrick (1981, Chapters 10-11) or

See Amman and Kendrick (1994) for a detailed derivation.

The term g(u 0 ) is the same as the script R term in[START_REF] Mizrach | Non-convexities in an stochastic control problem with learning[END_REF] and[START_REF] Amman | Nonconvexities in stochastic control models[END_REF] and the Θ term inTucci (2004, Chapters 2-5).

The updated variance of the parameter can be obtained using the Kalman filter and is independent of the actual observations. See, e.g.,Tucci (2004, Chapter 2) for a brief discussion orHarvey (1981, Chapter 4) for a more technical presentation.

15 See Amman and[START_REF] Amman | Nonconvexities in stochastic control problems: An analysis[END_REF] for

details.16 Equations (8)-(9) correspond to (2.17) and (2.19) in[START_REF] Amman | Nonconvexities in stochastic control problems: An analysis[END_REF], respectively.

As pointed out by an anonymous referee, it is also possible that the curve cuts the axis with a positive slope.

See also[START_REF] Amman | Nonconvexities in stochastic control models[END_REF].

This condition however is not necessary. It may happen that it is not satisfied and δ 1 is still positive. The bracket term in (20) may be positive thanks to the squared term in (21), or to w 2 , being sufficiently small to make the effect of w 1 overwhelming.

We thank an anonymous referee for providing a better approximation of the optimal cost than the one originally found by the authors.

This is equivalent to assuming that the estimator of the unknown parameter is unbiased.

In the first time period (0 time period) when computing u 0 the only uncertain element is the b , i.e. θ 0 . Therefore u 0 depends only on the uncertain b but u 1 will depend also on the additive noise terms as well as the uncertain parameter.

In the[START_REF] Macrae | Linear decision with experimentation[END_REF] problem the optimal control for period 0 is -1.25, with an associated cost of 19.163, when b = 0.4. A local optimum is at u 0 = 0.75 with a cost of 21.96. The maximum cost between these two optima is 22.68 and is associated with u 0 = 0. When b = -0.4 the cost function looks like in a mirror with the global optimum at u 0 = 1.25, the local optimum at u 0 = -0.75 with a cost of 19.163 and 21.96, respectively, and the maximum cost between the two optima is again 22.68 at u 0 = 0. The cost function is perfectly symmetric around u 0 = 0 when b = 0. In this case there are two global optima at u 0 = -1.25 and u 0 = 1.25 with a cost equal to 18.392, and the maximum cost between the two optima is 24.075 at u 0 = 0.

The optimal control associated with b = 3.0 is the opposite of the optimal control for b = -3.0 and the approximate cost-to-go is the same for the two controls. The same is true for all the positive values of b.

In the[START_REF] Beck | Learning and control in a changing economic environment[END_REF] problem the optimal control for period 0 is 3.75, with an associated cost of 4.034, when b = -0.1. A local optimum is at u 0 = -4.25 with a cost of 8.875. The maximum cost between these two optima is 14.55 and is associated with u 0 = -0.50. From Figure10, page 21, when b = 0.1 the cost function looks like in a mirror with the global optimum at u 0 = -3.75, the local optimum at u 0 = 4.25 with a gobal optimum cost of 4.034 and a local optimum cost of 8.875, respectively, and the maximum cost between the two optima is again 14.55 at u0 = 0.50.

Again the optimal control associated with a positive b is the opposite of the optimal control for a negative b and the approximate cost-to-go is the same in both cases.

This probability is approximately equal to the probability that the standard normal is in the interval (0.43, 0.47).

See, e.g., Kendrick (1981, 2002, Chapter 6).

and rewrite equation (A-3) as

Squaring the terms in equation (A-5) yields

or, alternatively,

which is identical to equation ( 19) when the δ's are defined as in (20).

Next derive equation ( 24) and equation ( 25) -the deterministic component. Begin with equation ( 23) i.e.

Again by replacing u 01 and x 02 by their definition, that is ( 8) and ( 9) respectively, using x 01 = αx 0 + bu 0 + γ , and equation ( 21) one obtains

Squaring and rearranging the terms in equation (A-9) yields

or, alternatively,

Equation (A-11) is identical to (24) when the ψ's are defined as in (25).