H S Qi

K H Cao

C K Woo

R Li

Incremental wind energy development in the Midcontinent electricity markets of the US

Keywords: Wind energy development, generation investment incentives, missing money, Midcontinent electricity markets, United States

This paper projects how much incremental wind energy development (WED) may occur without causing inadequate investment incentives (aka missing money) for wind generation and natural-gas-fired generation in the day-ahead market (DAM) and real-time market (RTM) of the Midcontinent Independent System Operator (MISO) in the US. Using a large sample of hourly data for the 82-month period of 01/01/2014 -10/31/2020, it documents that the DAM's hourly investment incentives move with (a) the day-ahead forecast of daily natural gas price; (b) MISO's day-ahead hourly requirements of ancillary services; (c) MISO's zonal day-ahead hourly schedules of nuclear generation, wind generation, and must-run generation; and (d) MISO's zonal day-ahead forecasts of hourly loads. Findings based on the RTM's hourly data tell a similar story. Further, the negative effect of incremental WED on investment incentives over the forward-looking period of 2023-2042 is offset by the positive effect of rising natural gas price, nuclear plant retirement, declining must-run generation, and growing demand. In the extreme case of nuclear plant retirement and zero must-run generation, incremental WED of up to ~441% of the existing wind generation level may occur as a market-based outcome without missing money in MISO's DAM.

Figures and Table legends

Introduction

At the 2021 G20 Summit held in Rome, the United States (US) reaffirmed its deep decarbonization commitment to tackle the urgent threat of climate change,1 necessitating large-scale wind energy development (WED) [START_REF] Williams | The technology path to deep greenhouse gas emissions cuts by 2050: the pivotal role of electricity[END_REF][START_REF] Williams | Pathways to deep decarbonization in the United States[END_REF][START_REF] Mahone | On the path of decarbonization[END_REF]. 2 Thanks to declining wind capacity costs [START_REF] Wiser | Land-based wind market report[END_REF] and government policies of renewable portfolio standards, investment and production tax credits, and easy transmission access [START_REF] Alagappan | What drives renewable energy development?[END_REF][START_REF] Hitaj | Wind power development in the United States[END_REF], WED in the US has been explosive, as evidenced by the vast increase in total installed wind capacity from ~1 GW in 2010 to ~122 GW in 2020.

An electric grid in the US can accommodate the rising market penetration of wind generation (WG) with transmission expansion [START_REF] Joskow | Facilitating transmission expansion to support efficient decarbonization of the electricity sector[END_REF] and flexible capacity provided by natural-gas-fired generation (NG), demand response resources, and energy storage systems [START_REF] Hargreaves | REFLEX: An adapted production simulation methodology for flexible capacity planning[END_REF]. With a feature like capacity market [START_REF] Spees | Capacity markets -lessons learned from the first decade[END_REF], resource adequacy requirement [START_REF] Cramton | Electricity market design[END_REF], or regulatory price adder (Zarnikau et al., 2020a), the grid's wholesale market design mitigates NG's missing money problem of inadequate investment incentive [START_REF] Joskow | Editorial: symposium on capacity markets[END_REF], which is further exacerbated by WG's electricity price reduction (aka merit order) effect [START_REF] Woo | Blowing in the wind: vanishing payoffs of a tolling agreement for natural-gas-fired generation of electricity in Texas[END_REF][START_REF] Woo | What moves the ex post variable profit of natural-gas-fired generation in California?[END_REF][START_REF] Quint | The impact of wind generation on wholesale electricity market prices in the midcontinent independent system operator energy market: an empirical investigation[END_REF][START_REF] Zarnikau | Market price behavior of wholesale electricity products: Texas[END_REF][START_REF] Zarnikau | Determinants of the wholesale prices of energy and ancillary services in the U.S. Midcontinent electricity market[END_REF][START_REF] Prol | The cannibalization effect of wind and solar in the California wholesale electricity market[END_REF][START_REF] Peña | Cannibalization, depredation, and market remuneration of power plants[END_REF].

As the regional transmission organization (RTO) in the US Midwest, the Midcontinent Independent System Operator (MISO) uses locational marginal pricing [START_REF] Stoft | Power system economics: designing markets for electricity[END_REF] and 10 local resource zones (LRZs) shown in Figure 1 for determining the hourly prices for energy and ancillary services in its day-ahead market (DAM) and real-time market (RTM) [START_REF] Zarnikau | Determinants of the wholesale prices of energy and ancillary services in the U.S. Midcontinent electricity market[END_REF]. This paper empirically answers the substantive research question of how much incremental WED may occur as a market-based outcome without missing money for WG's and NG's investments.

This question is real-world relevant, underscored by WG's and NG's documented missing money problem [START_REF] Woo | Blowing in the wind: vanishing payoffs of a tolling agreement for natural-gas-fired generation of electricity in Texas[END_REF][START_REF] Woo | What moves the ex post variable profit of natural-gas-fired generation in California?[END_REF](Woo et al., , 2019;;Woo and Zarnikau, 2019;[START_REF] Prol | The cannibalization effect of wind and solar in the California wholesale electricity market[END_REF][START_REF] Cao | Energy trading efficiency in the US Midcontinent electricity markets[END_REF][START_REF] Peña | Cannibalization, depredation, and market remuneration of power plants[END_REF] that dims the prospect of relying on market-based incentives to maintain system reliability [START_REF] Joskow | Reliability and competitive electricity markets[END_REF][START_REF] Joskow | Editorial: symposium on capacity markets[END_REF]. It is also policy important because incremental WED weakens the investment incentives for WG and NG that have lower CO2 emissions than coal-fired generation plants owned by MISO's members [START_REF] Zarnikau | Determinants of the wholesale prices of energy and ancillary services in the U.S. Midcontinent electricity market[END_REF]. Further, NG's flexible capacity is necessary for MISO's integration of the rising market penetration of WG [START_REF] Hargreaves | REFLEX: An adapted production simulation methodology for flexible capacity planning[END_REF]. Finally, an assessment of incremental WED's effects on generation investment incentives reveals the factors that can mitigate the missing money problem [START_REF] Mills | Strategies to mitigate declines in the economic value of wind and solar at high penetration in California[END_REF].

There are three reasons justifying our focus on MISO. First, "MISO operates one of the world's largest energy markets with more than $40 billion in annual market energy transactions." 3 Second, MISO's service territory in the US comprises the Northern region of LRZs 1 to 7 and the Southern region of LRZs 8 to 10. As shown by Figure 1, these two regions are geographically distinct, thus enabling an empirical analysis of regional generation investment incentives. Finally, MISO has salient features that make our paper an important and interesting case study, including its large footprint, huge peak demand, diverse market participants, and mix of heterogeneous generation capacities [START_REF] Zarnikau | Determinants of the wholesale prices of energy and ancillary services in the U.S. Midcontinent electricity market[END_REF][START_REF] Cao | Energy trading efficiency in the US Midcontinent electricity markets[END_REF]. Based on the hourly data for the 82-month period of 01/01/2014 to 10/31/2020 used by [START_REF] Cao | Energy trading efficiency in the US Midcontinent electricity markets[END_REF] to determine MISO's energy trading efficiency, our key findings are:

(1) Rising market penetration of WG shrinks WG's and NG's regional investment incentives in MISO's DAM and RTM.

(2) MISO's regional generation investment incentives increase with rising natural gas price, declining must-run generation, 5 growing demand, and declining nuclear generation.

(3) The projected negative effect of incremental WED on MISO's regional generation investment incentives in the forward-looking period of 2023-2042 is offset by the projected positive effect of the factors listed in (2).

(4) In the extreme case of nuclear plant retirement and zero must-run generation, incremental WED of up to ~441% may occur as a market-based outcome without missing money in MISO's DAM.

Our paper makes three contributions that complement recent empirical studies of missing money due to WG's merit order effect (e.g., [START_REF] Quint | The impact of wind generation on wholesale electricity market prices in the midcontinent independent system operator energy market: an empirical investigation[END_REF][START_REF] Zarnikau | Market price behavior of wholesale electricity products: Texas[END_REF][START_REF] Zarnikau | Determinants of the wholesale prices of energy and ancillary services in the U.S. Midcontinent electricity market[END_REF][START_REF] Prol | The cannibalization effect of wind and solar in the California wholesale electricity market[END_REF][START_REF] Cao | Energy trading efficiency in the US Midcontinent electricity markets[END_REF][START_REF] Peña | Cannibalization, depredation, and market remuneration of power plants[END_REF]): 6

 It adopts the perspective of a cost-minimizing load serving entity (LSE) (Woo and Zarnikau, 2019;Woo et al., 2019), leading to a newly developed explanation of the missing money problem based on the profitability of a wind power purchase agreement (WPPA) and a tolling agreement (TA). This explanation matches (a) the business reality 134 non-transmission owners, and generation capacity mix of natural gas (43%), coal (27%), renewables (21%), nuclear (7%) and other (2%).

5 With a zero price offer, must-run generation is the amount of electricity to be dispatched by MISO before supplies with positive price offers. Its presence is attributable to such reasons as (1) thermal power plants fueled by coal and natural gas often have minimum output requirements; (2) nuclear power plants are baseload and largely non-dispatchable; and (3) an integrated electric utility under the cost-of-service regulation may prefer to use its own generation assets instead of wholesale market purchases to meet its retail load obligations. As an increase in must-run generation causes a rightward shift of MISO's generation supply curve, it reduces MISO's wholesale electricity prices (Zarnikau et al., 2020a). 6 Two reasons support our paper's exclusion of a comprehensive literature review of the numerous studies on renewable energy's merit order effect that diminishes generation investment incentives. First, such a review does not add material insights beyond those presented herein. Second, ample literature reviews are already available in the cited papers, thus obviating our duplication of effort.

that a power plant developer in the US often uses such agreements to obtain project financing at a reasonable cost [START_REF] Stern | Merchant power plants: competing for financing in a deregulated industry[END_REF]; and (b) the widespread use of WPPAs that suggests windfarm owners seldom make supply offers like independent power producers of conventionally generated energy in a RTO's DAM and RTM.

 To the best of our knowledge, it is the first empirical study of regional generation investment incentives in MISO's DAM and RTM. Specifically, it documents what matters in MISO's incremental WED, thus complementing the assessment studies by [START_REF] Mills | Strategies to mitigate declines in the economic value of wind and solar at high penetration in California[END_REF][START_REF] Mills | Impacts of variable renewable energy on wholesale markets and generating assets in the United States: A review of expectations and evidence[END_REF].

 Its regression-based approach is applicable to assess incremental WED's effects on generation investment incentives in the wholesale electricity markets operated by other RTOs in North America, as well as those in Europe (e.g., France, Germany, Italy, Spain, and UK), South America (e.g., Brazil and Chile), and Asia Pacific (e.g., Australia, New Zealand, and Singapore).

The rest of this paper proceeds as follows. Section 2 explains WG's and NG's investment incentives. Section 3 develops our regression setup. Section 4 assesses how much incremental WED may occur as a market-based outcome for the 2023-2042 period. Section 5 reports our empirics, the basis for Section 6: conclusions and policy implication.

Generation investment incentive based on a LSE's least-cost procurement

We use E() = expected annual sum of hourly operating profits per MW of installed capacity to measure generation investment incentive [START_REF] Woo | Blowing in the wind: vanishing payoffs of a tolling agreement for natural-gas-fired generation of electricity in Texas[END_REF][START_REF] Woo | What moves the ex post variable profit of natural-gas-fired generation in California?[END_REF]. If E() ≥ F = per MW installed cost ($) × [cost of capital (% per year) + depreciation (% per year)], there is adequate investment incentive. When E() < F, the missing money problem is said to exist.

We calculate hourly operating profit from the perspective of a LSE that minimizes its energy procurement cost (Woo and Zarnikau, 2019;Woo et al., 2019;[START_REF] Woo | Regional revenues of solar and wind generation in Texas[END_REF]. This makes sense based on the following line of reasoning. After signing a 1-year WPPA at levelized price LP = ~$25/MWh [START_REF] Wiser | Land-based wind market report[END_REF],7 each wind MWh delivered in hour h = 1, …, 24 on day d displaces the same amount to be bought at wholesale market spot price Pdh.

As a result, Pdh is the LSE's per MWh avoided energy cost attributable to the signed WPPA.

Made possible by the signed WPPA, dh = Pdh -LP is the LSE's per MWh profit in hour h on day d. When the annual sum of E(dh) > 0, it induces the LSE to sign the WPPA.

As WG's merit order effect reduces E(Pdh), it diminishes the LSE's signing incentive.

We now consider the case of a 1-year TA for a MW-share of a dispatchable and fast ramping NG plant that employs combined cycle gas turbines (CCGTs). After signing the TA with contracted heat rate HR (MMBtu/MWh), the LSE instructs the plant operator to perform economic dispatch to achieve Cdh = min(Pdh, HR × Gd), where Gd = natural gas price ($/MMBtu) on day d [START_REF] Woo | What moves the ex post variable profit of natural-gas-fired generation in California?[END_REF](Woo et al., , 2019)). As a result, Cdh is the LSE's per MWh In summary, the rising market penetration of WG tends to reduce the construction of windfarms and NG plants in wholesale electricity markets like those operated by MISO.

Section 3 below proposes a regression analysis of the hourly data for Pdh and Cdh to reveal the factors that can offset this capacity reduction. Section 4 then uses the regression results to assess the market-based incremental WED in MISO's DAM and RTM.

Regression setup

Specification

For expositional ease, we first focus on MISO's DAM in the Northern region, as the discussions in connection to MISO's DAM in the Southern region and RTMs in both regions are entirely analogous.

Let Pjdh = hourly DAM price of LRZ j = 1, .., 7 in hour h = 1, …, 24 on day d = 01/01/2014, …, 10/31/2020. The data generating process (DGP) for Pjdh is assumed to be the following linear regression with random error jdh that may be heteroskedastic and autocorrelated:

Pjdh = Xjdh  + fixed effect + jdh, (1)
where Xjdh = vector of fundamental drivers;  = vector of coefficients that measure the MMBtu per MWh × $X1dh per MMBtu), as the TA's per MWh non-fuel variable cost is for all practical purposes negligible [START_REF] Woo | What moves the ex post variable profit of natural-gas-fired generation in California?[END_REF].

The DGP for Cjdh is assumed to be the following linear regression with the coefficient vector  that measures the marginal cost effects of Xjdh and random error jdh that may be heteroskedastic and autocorrelated:

Cjdh = Xjdh  + fixed effect + jdh.
(2)

As the regression specification for the Northern region's DAM, equations ( 1) and ( 2 Replacing their regressands and regressors to reflect the Southern region's DAM yields two additional equations to be estimated. Further, replacing the day-ahead data with real-time data leads to four equations for the two regional RTMs. In short, each region has a system of two energy price regressions and two per MWh procurement cost regressions differentiated by market type.

Estimation strategy

Recall the two regional regression systems in Section 3.1, the Northern system, which uses hourly data for LRZs 1 to 7, and the Southern system, which uses hourly data for LRZs 8 to 10. As the two regional systems have unequal sample sizes, we use PROC MODEL of SAS to estimate them separately. 11 We use PROC MODEL's GMM option to obtain heteroscedasticity-autocorrelation-consistent (HAC) standard errors for gauging the statistical significance of the coefficient estimates [START_REF] Wooldridge | Econometric analysis of cross section and panel data[END_REF]. 12 Finally, we perform unit root tests to allay the concern of spurious regressions caused by non-stationary regression residuals [START_REF] Davidson | Estimation and inference in econometrics[END_REF].

Incremental wind energy development

Wind power purchase agreement

Stated below are the steps for assessing how much incremental WED may occur without reducing a WPPA's profitability in MISO's DAM over the 2023-2042 period.

Step 1 uses the regional DAM price regression results to calculate R > 0, the regional average of hourly DAM price increases triggered by the plausible events listed below:

(1) $1/MMBtu increase in the Henry Hub natural gas price in the 20-year period. The Henry Hub price is expected to rise because of (a) increasing US export of LNG to reduce Europe's dependence on Russia's supply of natural gas (EIA, 2022); (b) a slowdown in shale gas development due to environmental concerns about hydraulic fracturing [START_REF] Sovacool | Cornucopia or curse? Reviewing the costs and benefits of shale gas hydraulic fracturing (fracking)[END_REF]; (c) electricity generation's increasing demand for natural gas based on the EIA's Annual Energy Outlook 2022; 13 and (d) inter-fuel substitution between natural gas and petrol products like fuel oil and diesel [START_REF] Li | How price responsive is industrial demand for natural gas in the United States?[END_REF].

( Step 2 assumes  = total percentage increase in day-ahead wind MWh by LRZ at the end of the 20-year period. For illustration, the day-ahead wind MWh in the year 2042 for LRZ 1 in MISO's Northern region is (1 + ) × LRZ 1's day-ahead wind MWh in 2019. We select 2019 as the reference year because it is the most recent year in our sample period, with hourly data for all 12 calendar months.

Step 3 uses the DAM price regression results to calculate S < 0, which is WG's regional average of hourly DAM price reductions based on 's assumed value.

Step 4 varies  from 1% to 500% to find  * that results in R + S = 0. Hence,  * is an estimate for the market-based extent of incremental WED in connection to WPPAs. As there is no local wind generation in the Southern region, we use the Northern region's wind MWh to determine the  * values for both regions.

To find the regional  * values for the RTM, we repeat Steps 1 to 4 after replacing the DAM price regression results with the RTM price regression results and the day-ahead data with the real-time data.

Tolling agreement

Stated below are the steps for assessing how much incremental WED may occur in connection to TAs in MISO's DAM over the 2023-2042 period:

Step 1 uses the regional DAM price and per MWh procurement cost regression results

to calculate R' = regional average of hourly (Pjdh -Cjdh) = TA's projected per MWh profit increase due to the events listed in Step 1 of Section 4.1.

Step 2 assumes  = total percentage increase in day-ahead wind MWh by LRZ at the end of the 20-year period.

Step 3 uses the regression results noted in Step 1 to calculate S' < 0, which is the TA's decremental per MWh profit. Specifically, S' = regional average of hourly (Pdh -Cdh)

based on 's value assumed in Step 2.

Step 4 varies  from 1% to 500% to find  ** that results in R' + S' = M, where M = NG's amount of missing money embodied in the existing hourly data.

To find  ** for the RTM, we repeat Steps 1 to 4 after replacing the DAM price and per MWh procurement cost regression results with their RTM counterpart and the day-ahead data with the real-time data.

Extent of incremental WED

Since wind capacity expansion simultaneously affects a WPPA's and a TA's profitability, M = min( * ,  ** ) is the condition for adequate investment incentives for WG and NG. This makes sense because M is the extent of incremental WED that may occur without causing missing money for WG and NG [START_REF] Woo | Regional revenues of solar and wind generation in Texas[END_REF].

Empirics

This section reports our empirics based on MISO's hourly data from 01/01/2014 to 10/31/2020. After describing the hourly data, it presents our regression results, followed by our assessment of incremental WED for the forward-looking period of 2023-2042. The average energy price and per MWh procurement cost are slightly higher in the Northern region's DAM than in RTM. However, they are lower than the corresponding averages of the Southern region. Further, energy prices and the differentials between energy price and per MWh procurement cost for both regions are highly volatile, as evidenced by their wide ranges and large standard deviations. Finally, per MWh procurement costs are less volatile than energy prices because of a CCGT's economic dispatch based on the relatively stable natural gas prices.

Descriptive statistics and correlations

Regional energy prices are positively correlated with per MWh procurement costs (r > 0.33) and natural gas price (r > 0.28). However, they are weakly correlated with regional wind generation, nuclear generation, load, and must-run generation.

Regional per MWh procurement costs positively correlate with natural gas price (r > 0.74). As in the case of energy prices, they are weakly correlated with regional wind generation, nuclear generation, load, and must-run generation.

While the correlations generally corroborate our expectations, they do not untangle the marginal effects of each fundamental driver on energy prices and procurement costs, thus motivating our use of the regression results presented in the following section.

Regression results

Table 2 reports the regression results that yield the following inferences: When taken together, the preceding findings lend support for using Table 2 to quantify the incremental WED estimates presented below.



Incremental WED estimates

Under the assumption of no nuclear plant retirement, the top half of Table 3 reports that the Northern region's M estimates range from 190% to 267%. Nuclear plant retirement would greatly magnify these estimates. Finally, the M estimates for the Southern region are below those for the Northern region.

To better understand what drives Table 3's voluminous results, we estimate the following OLS regression with intercept  and random error :

Y =  +  N +  MG +  R +  D + , (3)
where Y = M estimate in Table 3; Table 4 reports the OLS regression results that lead to the following inferences:

 The regression has an adjusted R 2 is 0.97. With the exception of D, it has highly significant (p-value < 0.01) coefficient estimates for the remaining regressors. Hence, it is a reasonable characterization of the  estimates reported in Table 3.

 The sample mean of the  estimates is 213%, presaging the potentially large level of incremental WED in MISO's DAM and RTM.

 For the scenario of N = MG = R = D = 0, the  estimate is 223%, chiefly reflecting the combined effect of the $1/MMBtu increase in natural gas price and MISO's cumulative growth of 20% by the end of 2042.

 Nuclear plant retirement increases the  estimates by 118%, making it the biggest driver of MISO's incremental WED.

 A 1% reduction in must-run generation is found to increase the  estimates by 0.54%.

Should the must-run generation become zero, the resulting increase would be 54%.

 The Northern region's M estimates are on average 180% above the Southern region's.

 The DAM's M estimates are on average 13% (p-value = 0.052) above the RTM's.

Table 4's overall message is that large-scale incremental WED may occur as a marketbased outcome in MISO's DAM and RTM, due chiefly to the projected increases in natural gas price, load growth, nuclear plant retirement, and decline in must-run generation. Notes: (1) The average levelized price of recently signed WPPA is ~$25/MWh [START_REF] Mills | Impacts of variable renewable energy on wholesale markets and generating assets in the United States: A review of expectations and evidence[END_REF][START_REF] Wiser | Land-based wind market report[END_REF], below the regional means of hourly energy prices by market type.

Conclusions and policy implication

(2) Based on the cost of new entry (PJM, 2018), a CCGT's per MWh capacity is ~$11.41/MWh, below the regional means of (hourly energy prices -hourly per MWh procurement costs) by market type. (2) The Dickey-Fuller and Phillips-Perron unit root tests decisively reject (p-value < 0.01) the null hypothesis that the regression residuals are non-stationary. Hence, the regression results reported in this table are not spurious [START_REF] Davidson | Estimation and inference in econometrics[END_REF].

(3) This table uses GW instead of MW to measure system AS requirement and GWh instead of MWh to measure energy-related variables to better represent their coefficient estimates that would otherwise be very small in size.

Figure 1

 1 Figure 1 MISO's footprint and local resource zones

  procurement cost in hour h on day d. The LSE's hourly profit in hour h on day d is (Pdh -Cdh -K), where K = per MWh fixed charge = TA's per MW-year capacity cost for a CCGT ÷ 8,760 hours per year = ~$11.41/MWh (PJM, 2018). When the annual sum of [E(Pdh) -E(Cdh) -K] > 0, it induces the LSE to sign the TA. As WG's merit-order effect reduces [E(Pdh) -E(Cdh)] (Woo et al., 2016), it diminishes the LSE's signing incentive.

  marginal price effects of Xjdh; and fixed effect = linear function of the intercept and binary indicators for time of the day, day of the week, month of the year, and LRZs in MISO's Northern region. We decide not to use the log-linear specification, chiefly because the presence of negative prices renders ln(Pjdh) undefined.8 Guided by[START_REF] Zarnikau | Determinants of the wholesale prices of energy and ancillary services in the U.S. Midcontinent electricity market[END_REF] and[START_REF] Cao | Energy trading efficiency in the US Midcontinent electricity markets[END_REF], the regressors in equation (1) are as follows: 9  X1dh = Daily forecast of natural gas price ($/MMBtu) at the Henry Hub that does not vary hourly. 10 Its coefficient 1 > 0 measures the DAM price effect of a $1/MMBtu increase in natural gas price.  X2dh = MISO's system AS requirement (MW) announced on day d-1 for hour h on day d. Its coefficient 2 > 0 measures the DAM price effect of rising AS requirement.  X3jdh = LRZ j's local wind MWh scheduled on day d-1 for delivery in hour h on day d. Its coefficient 3 < 0 measures the DAM price effect of LRZ j's rising WG.  X4jdh = Total non-local wind MWh of all other LRZs scheduled on day d-1 for delivery in hour h on day d. For example, if j = 1, X4jdh = sum of wind MWh for LRZs 2 to 7 because LRZs 8 to 10 do not have WG. Its coefficient 4 < 0 measures the DAM price effect of rising WG in all other LRZs.  X5jdh = LRZ j's local nuclear MWh scheduled on day d-1 for delivery in hour h on day d. Its coefficient 5 < 0 measures the DAM price effect of LRZ j's rising nuclear generation.  X6jdh = Total non-local nuclear MWh of all other LRZs scheduled on day d-1 for delivery in hour h on day d. Its coefficient 6 < 0 measures the DAM price effect of rising nuclear generation in all other LRZs.  X7jdh = LRZ j's local must-run MWh scheduled on day d-1 for delivery in hour h on day d. Its coefficient 7 < 0 measures the DAM price effect of LRZ j's rising must-run generation.  X8jdh = Total non-local must-run MWh of all other LRZs scheduled on day d-1 for delivery in hour h on day d. Its coefficient 8 < 0 measures the DAM price effect of rising must-run generation in all other LRZs.  X9jdh = LRZ j's forecast of local MWh load made on day d-1 for hour h on day d. Its coefficient 9 > 0 measures the DAM price effect of LRZ j's rising demand.  X10jdh = Forecast of non-local MWh load of all other LRZs made on day d-1 for hour h on day d. Its coefficient 10 > 0 measures the DAM price effect of rising demands in all other LRZs. We now turn our attention to the per MWh procurement cost regression. Based on Woo et al. (2016), we assume HR = 7 MMBtu per MWh = engineering-based heat rate of a CCGT, the most popular thermal generation technology deployed in the US. If an LSE serving LRZ j has signed a CCGT-based TA, its day-ahead per MWh procurement cost is Cjdh = min(Pjdh, 7

  ) show that Pjdh and Cjdh move in tandem because (a) they both use{Xjdh}as their fundamental drivers; and (b) the marginal effects of Xjdh on Cjdh based on in equation (2) mimic those on Pjdh based on  in equation (1). A case in point is that a $1/MMBtu increase in natural gas price is expected to raise Cjdh by $1/MWh and Pjdh by $1/MWh.

  The adjusted R 2 values are between 0.261 and 0.826 for the Northern region and 0.172 and 0.984 for the Southern region. As the low 0.261 and 0.172 values come from the two regional RTM price regressions, the remaining six regressions are deemed to have a reasonable fit. The slope coefficient estimates generally support our expectation of the price and cost effects of the fundamental drivers because only 7 out of the 76 estimates have unexpected signs and are statistically significant. A $1/MMBtu increase in daily natural gas price tends to increase the Northern region's DAM energy price by $6.6/MWh and RTM energy price by $4.9/MWh. Its impact on the Southern region's DAM and RTM energy prices are $7.5/MWh and $7.6/MWh.Reflecting the energy price regressions' empirical plausibility, these estimated effects equal NG's market-based marginal heat rates that in the main are numerically close to a CCGT's engineering-based heat rate of 7 MMBtu/MWh. The responses of per MWh procurement costs to a $1/MMBtu increase in natural gas price are similar to those of DAM and RTM energy prices in both regions. The statistically significant estimated effects of a 1-GW increase in system AS requirement on energy prices and per MWh procurement costs are $0.436/MWh to $3.457/MWh.  Confirming WG's merit order effects, each additional GWh of local wind generation in the Northern region tends to reduce the region's energy prices and per MWh procurement costs by $0.79/MWh to $2.73/MWh, larger than the reductions due to non-local WG's increase of the same amount.  The estimated impacts of a 1-GW increase in the local nuclear generation on energy price are -$0.03/MWh to -$3.70/MWh, larger in size than those of a 1-GW increase in nonlocal nuclear generation.  The estimated marginal effects of 1-GW increases in local and non-local must-run generation on energy prices in both regions are mixed and less than $1.21/MWh in size.  Rising local and total non-local GWh loads tend to have positive impacts of $0.562 to $3.031/MWh on energy prices and $0.012 to $0.129/MWh on per MWh procurement costs in the DAM and RTM of both regions.

  N = 1 if nuclear plant retirement, 0 otherwise; MG = percentage reduction in must-run generation; R = 1 if Southern region, 0 otherwise; and D = 1 if RTM, 0 otherwise. Equation (3)'s slope coefficients untangle the effects of N, MG, R, and D on M, thus revealing what matters in MISO's market-based incremental WED. To see this point, consider the case of N = MG = R = D = 0 so that the expected value of Y is E(Y) = , which is the Northern region's average M value based on the DAM data in the absence of nuclear retirement and must-run generation reduction. Suppose N = 1 that indicates nuclear plant retirement that results in E(Y) = ( + ). Hence, is nuclear plant retirement's effect on the Northern region's average M value. If 's estimate is large, nuclear plant retirement is expected to greatly increase MISO's incremental WED for 2023-2042.

Figure 1 .

 1 Figure 1. MISO's footprint and local resource zones (Source: MISO Tariff Attachment VV)

Table 1

 1 Descriptive statistics of hourly data

Table 2

 2 Regression results by region and market type

Table 3

 3 Results for the extent of incremental wind energy development

Table 4

 4 Response analysis of the extent of incremental wind energy development based on the OLS regression specification

  4

3 https://www.misoenergy.org/about/media-center/corporate-fact-sheet/ 4 According to its 2022 corporate fact sheet (https://www.misoenergy.org/about/media-center/corporate-factsheet/), MISO has a total generation capacity of 205.2 GW, historic summer peak of 127.1 GW set on 07/20/2011, historic winter peak of 109.3 GW set on 01/06/2014, membership of 56 transmission owners and

  ) Nuclear plant retirement. As the nuclear plants in MISO's service territory are over 20 years old, their retirement may occur by the end of the 2023-2042 period. Thus, we Cumulative load growth of 20% for the 20-year period. It is based on the annual forecast of ~1% per year in the EIA's Annual Energy Outlook 2022.(4) Reduction in must-run generation, reflecting the possibility of the rising volume of supply offers with strictly positive price quotes submitted by thermal generators during MISO's market operation hours. This possibility arises due to the retirement of aging power plants with relatively low operational flexibility and high CO2 emissions (e.g., coal-fired power plants) and the construction of new power plants with relatively high operational flexibility and low CO2 emissions (e.g., CCGTs). Since we cannot accurately project the extent of this reduction in the 20-year period, we assume hypothetical reductions of 0%,

	25%, 50%, 75%, and 100%.
	calculate R under two assumptions: (a) without nuclear retirement; and (b) with nuclear
	retirement.

11 https://support.sas.com/documentation/onlinedoc/ets/132/model.pdf 12 https://support.sas.com/kb/40/098.html 13 https://www.eia.gov/outlooks/aeo/pdf/AEO2022_ReleasePresentation.pdf

(3)

Table 1

 1 presents our hourly data's descriptive statistics and correlations, leading to the following remarks. The mean energy prices are between $26 and $28/MWh, higher than the

	LP of a 1-year WPPA of ~$25/MWh recently reported by Wiser et al. (2021). As the
	WPPA's LP presumably fully covers WG's fixed costs, there is adequate investment
	incentive for a windfarm developer and sufficient signing incentive for a LSE.
	The mean per MWh procurement costs are between $19 and $20/MWh. As a result, a
	1-year TA's mean payoff is between $7.55/MWh to $8.07/MWh based on the average

difference between energy prices and per MWh costs. Since a CCGT's per MWh capacity cost is ~$11.41/MWh (PJM, 2018), the TA's amount of missing money is M = ~$4/MWh used in Step 4 of Section 4.2.

Table 1 .

 1 Descriptive statistics of hourly data; sample period = 01/01/2014 -10/31/2020; hourly per MWh procurement cost = min(hourly energy price, HR × daily natural gas price), where HR = 7 MMBtu per MWh for a CCGT-based tolling agreement

	Panel A: Day-ahead hourly data for MISO's Northern region of LRZ 1 to 7
	Variable	Number of non-	Mean	Standard	Minimum	Maximum	Correlation
		missing		deviation			Energy price	Per MWh
		observations					procurement cost
	Energy price ($/MWh)	419160	26.96	13.44	-20.02	463.94	1.000	0.624
	Per MWh procurement cost ($/MWh)	419328	19.28	5.82	-20.02	54.95	0.624	1.000
	Energy price -per MWh procurement cost	419160	7.68	10.81	0.00	425.02	0.907	0.237
	Natural gas price ($/MMBtu)	419328	2.89	0.83	1.36	7.85	0.470	0.865
	System AS requirement (MW)	419328	2442.00	137.76	2310.00	2997.00	0.095	0.060
	Local wind generation (MWh)	418572	817.20	1339.00	0.00	11064.00	-0.260	-0.249
	Total non-local wind generation (MWh)	418572	4903.00	3226.00	21.39	19806.00	-0.139	-0.140
	Local nuclear generation (MWh)	419328	998.47	622.57	0.00	1980.00	-0.021	-0.004
	Total non-local nuclear generation (MWh)	419328	10395.00	1325.00	5609.00	13460.00	0.007	-0.001
	Local must-run generation (MWh)	419328	4292.00	2268.00	0.00	13199.00	0.366	0.285
	Total non-local must-run generation (MWh)	419328	31370.00	7898.00	9508.00	57435.00	0.502	0.468
	Local load (MWh)	419328	7751.00	2954.00	2452.00	20142.00	0.316	0.153
	Total non-local load (MWh)	419328	65604.00	10892.00	40008.00	113227.00	0.501	0.248
	Panel B: Real-time hourly data for MISO's Northern region of LRZ 1 to 7
	Variable	Number of non-	Mean	Standard	Minimum	Maximum	Correlation
		missing		deviation			Energy price	Per MWh
		observations					procurement cost
	Energy price ($/MWh)	419328	26.33	20.23	-172.85	1862.00	1.000	0.447
	Per MWh procurement cost ($/MWh)	419328	18.78	6.39	-172.85	57.05	0.447	1.000
	Energy price -per MWh procurement cost	419328	7.55	18.29	0.00	1830.00	0.950	0.146
	Natural gas price ($/MMBtu)	419328	2.89	0.85	1.33	8.15	0.284	0.744
	System AS requirement (MW)	419307	2441.00	139.43	1135.00	2997.00	0.059	0.059
	Local wind generation (MWh)	419328	814.46	1330.00	0.00	9861.00	-0.189	-0.270
	Total non-local wind generation (MWh)	419328	4887.00	3183.00	0.00	17866.00	-0.122	-0.151
	Local nuclear generation (MWh)	419328	994.44	620.82	0.00	1990.00	-0.008	0.002
	Total non-local nuclear generation (MWh)	419328	10349.00	1353.00	0.00	13447.00	0.011	0.021
	Local must-run generation (MWh)	419328	4112.00	2121.00	0.00	12576.00	0.241	0.302
	Total non-local must-run generation (MWh)	419328	30253.00	7597.00	1948.00	55576.00	0.320	0.447
	Local load (MWh)	419328	7831.00	2966.00	2504.00	19845.00	0.211	0.170
	Total non-local load (MWh)	419328	66245.00	10690.00	41755.00	110784.00	0.325	0.248

Table 2 .

 2 Regression results by region and market type; sample period = 01/01/2014 -10/31/2020; per MWh procurement cost = min(energy price, HR × daily natural gas price), where HR = 7 MMBtu per MWh for a CCGT-based tolling agreement; statistically significant (p-value ≤ 0.01) coefficient estimates based on HAC standard errors[START_REF] Wooldridge | Econometric analysis of cross section and panel data[END_REF] in bold; anomalous coefficient estimates with unexpected signs in italic For brevity, this table omits the voluminous coefficient estimates for the intercept and binary indicators for hour of the day, day of the week, month of the year, and LRZs. Most (~80%) of these estimates are highly significant with p-values ≤ 0.01.

	Variable	Northern region

Table 3 .

 3 Results for M = extent of incremental wind energy development by the end of 2042 without causing missing money for wind and natural-gas-fired generation The other assumptions used to construct this table are the $1/MMBtu increase in natural gas price in the 2023-2042 period and the cumulative load growth of 20% by the end of 2042.

		Scenario	Northern region		Southern region
	Nuclear plant retirement	Reduction in must-run generation (%)	DAM	RTM	DAM	RTM
	No	0	190%	215%	47%	46%
		25	213%	225%	60%	61%
		50	231%	235%	73%	76%
		75	249%	244%		91%
		100	267%	254%	100%	106%
	Yes	0	369%	321%	141%	149%
		25	387%	331%	154%	164%
		50	405%	341%	168%	174%
		75	423%	351%	181%	183%
		100	441%	360%	194%	192%
	Note:

http://www.g20.utoronto.ca/2021/G20ROMELEADERSDECLARATION.pdf

Other decarbonization measures include solar energy development, energy efficiency improvement, conservation, and implementation of smart grid aided by advanced metering infrastructure that enables dynamic pricing and reliability differentiation(Woo and Zarnikau, 2019; Woo et al., 2019).

Without any loss in generality, the 1-year contract period is assumed solely for expositional clarity and simplicity.

During low-demand hours (e.g., 02:00 to 05:00), MISO's system loads at times cannot fully absorb the nondispatchable generation output from nuclear plants and windfarms. To maintain its load-resource balance, MISO uses negative prices to induce generation curtailment by dispatchable NG plants in these hours.

9 Thanks to a diligent referee's insightful comment, we exclude the PJM price for energy at the MISO-PJM border as a regressor because it is highly colinear with the natural gas

price.10 The following reasons support our use of the Henry Hub price(Woo et al., 2006a): (a) Henry Hub is the largest trading hub in the US; (b) the Henry Hub is the delivery point of the US natural gas futures; (c) the Henry Hub price is highly correlated (r > 0.9) with local hub prices in MISO's service territory; and (d) the Henry Hub price is exogenously determined by the national integrated market formed by interconnected markets across the US.

Our conclusions are: (a) hourly generation investment incentives in MISO's DAM and RTM move with their fundamental drivers; (b) incremental WED's negative effects on generation investment incentives for the forward-looking period of 2023-2042 are offset by positive effects of rising natural gas price, nuclear plant retirement, declining must-run generation, and growing electricity demand; (c) incremental WED without causing missing money for WG and NG can be as much as 441% in the extreme case of nuclear plant retirement and zero must-run generation; and (d) the regression results shown in Table 4 delineate what matters in the assessment of MISO's market-based WED. The overall policy implication of these conclusions is that market forces alone may suffice to induce large-scale incremental WED in MISO's DAM and RTM.

We would be remiss had we failed to acknowledge the following caveats of our paper.

First, as our regression-based approach necessarily uses historical hourly data, it cannot determine the changes in generation investment incentives due to significant events that may occur beyond our sample period (e.g., a substantial revision of MISO's wholesale product mix and expansion of transmission interconnection between MISO's Northern and Southern regions). Nevertheless, its empirics can be readily updated with additional data collection.

Second, our regression setup in Section 3 may be considered inadequate when compared to time series modeling of market prices that exhibit long memory and regime switching [START_REF] Haldrup | A regime switching long memory model for electricity prices[END_REF][START_REF] Karakatsani | Intra-day and regime-switching dynamics in electricity price formation[END_REF][START_REF] Janczura | An empirical comparison of alternate regime-switching models for electricity spot prices[END_REF] and engineering simulation of WED's impact on locational marginal prices [START_REF] Morales | Simulating the impact of wind production on locational marginal prices[END_REF]. That said, our regression setup is meritorious in providing a first look at incremental WED without missing money in MISO's DAM and RTM.

Finally, our paper overlooks the numerous thorny issues that can limit the ability of a RTO like MISO to deliver reliable service at stable prices [START_REF] Woo | Electricity market reform failures: UK, Norway, Alberta and California[END_REF][START_REF] Woo | Costs of electricity deregulation[END_REF]) that obey the principles of least-cost resource planning, procurement, and operation (Woo et al., 2019).

A partial list of these issues includes wholesale market's trading inefficiency, transmission congestion, market power abuse, inadequate use of reliability differentiation and dynamic pricing at the retail level, retail pricing's limited pass-through of wholesale price changes, market power abuse by rogue independent power generators, and poor coordination of generation and transmission investments in a competitive market environment [START_REF] Cao | Energy trading efficiency in the US Midcontinent electricity markets[END_REF]. While meaningfully addressing some of these issues is possible through a market design with reliability differentiation and generation dispatch based on marginal fuel costs (Woo et al., 2019), it is well beyond this paper's intent and scope.

Declarations of Interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.


Maroua Bahri
email: maroua.bahri@inria.fr

Nikolaos Georgantas
email: nikolaos.georgantas@inria.fr

AutoClass: AutoML for Data Stream Classification

Keywords: AutoML, stream classification, data stream, algorithm selection, hyper-parameter tuning

Automated Machine Learning (autoML) is a novel topic that aims to tackle the parameter configuration issue using automatic monitoring models and comprises different machine learning tasks, such as feature selection, model selection, and hyper-parameter tuning. It makes easier use of algorithms for non-ML experts as well as ML experts by automating tasks that rely on expert domain knowledge. Nevertheless, autoML is in its infancy stage and not well explored yet in the offline and stream settings. In this paper, we propose automated Classification (auto-Class) method for automated algorithm selection and configuration for data stream classification. AutoClass consists of training an ensemble of different tuned configurations and selecting the best-performing configuration to do the prediction. We present experiments performed on a diverse set of real and artificial datasets and show how our proposed approach can outperform the performance of competitive state-of-the-art ensemble and single-based methods.

I. INTRODUCTION

In recent years, we have witnessed a digital revolution that has dramatically changed the way in which we generate and consume data. This new paradigm of ubiquitous data led to several emerging applications, sensors, and Internetof-Things (IoT) devices that generate massive amounts of data continuously and online as big data streams. In this context, data stream mining tasks have become indispensable in multiple real-world applications that demand real-time -or near real-time -analysis, such as traffic management, smart cities, and predictive maintenance for transport and logistics.

Machine Learning (ML) benefited from tremendous research progress recently in many application areas, particularly for the stream setting. Often, to unravel the full potential of ML, some human expertise and domain knowledge are required in order to select the most effective algorithm, tune its hyper-parameters, as well as to perform pre-processing including, but not limited to, data cleaning, filling missing data, feature selection, feature extraction, etc [START_REF] Bahri | Automl: state of the art with a focus on anomaly detection, challenges, and research directions[END_REF]. This is often a tedious and non-trivial task for researchers with all levels of expertise. To alleviate such a burden for researchers, several techniques have been proposed under the emerging automated Machine Learning (autoML) topic that is still in its infancy stage and not well explored yet in both offline and stream settings.

AutoML tackles mainly the problem of finding ML models with high predictive performance. In the traditional offline setting, existing approaches assume that all data are available at the beginning of the training process. They configure and optimize a pipeline of preprocessing, feature engineering, and model selection by choosing suitable hyper-parameters. With the creation of a vast amount of data from all kinds of sources every day, our capability to process and understand these datasets in a single batch is no longer viable. In fact, the unbounded nature of evolving data streams raises some technical and practical limitations that make traditional stream algorithms fail because of the high use of resources, such as time and memory, to process dynamic data distributions. By training ML models incrementally (i.e. online learning 1 ), the flood of data can be processed sequentially within data streams. However, if one assumes an online learning scenario, where an autoML instance executes on evolving data streams, the question of the best model and its optimal configuration remains open.

Numerous frameworks for data stream processing have emerged that focus on the application of ML algorithms, which mainly pursue online ML requirements. Common frameworks such as MOA [START_REF] Bifet | Moa: Massive online analysis[END_REF], and river [START_REF] Montiel | River: machine learning for streaming data in python[END_REF], aim to foster research for algorithms on data streams. Nevertheless, on the conceptual level for the processing and orchestration of data streams, further frameworks and functionalities are needed to perform autoML. In this paper, we focus on these issues in the classification of data streams using an ensemble of different algorithms and configurations and a hyper-parameters optimization technique. The main focus of our work attempts to propose an autoML solution for classification to deal with massive data by tuning the parameters incrementally and selecting the best-performing classifier at evaluation time. Our main contributions can be summarized as follows:

• autoClass. We propose an automated classification approach for data streams that naturally adapts the best algorithm and its hyper-parameters over time. • Empirical results. We present an experimental study that shows the abilities of our proposal to obtain good results against popular baselines.

The remainder of this work is organized as follows. In Section II, we detail related work where we present the stateof-the-art in autoML. Section III presents the description of our proposed approach for automated stream classification. In Section IV, we show and discuss the experimental results using a diverse set of datasets. We finally end this paper by drawing conclusions in Section V.

II. RELATED WORK

AutoML is a novel topic that has attracted several researchers in recent years due to the importance of its application. It supports researchers and practitioners with the tedious work of manually designing ML pipelines and can also be viewed as the process that makes ML easier by avoiding manual hyper-parameters tuning for both ML experts and nonexperts. Automated approaches often model the autoML task as the Combined Algorithm Selection and Hyper-parameter optimization problem, aka CASH [START_REF] Thornton | Autoweka: Combined selection and hpo of classification algorithms[END_REF].

Most of the autoML solutions have been initially proposed for the batch setting and are discussed in recent surveys on autoML and its open challenges [START_REF] Feurer | Hyperparameter optimization[END_REF], [START_REF] He | Automl: A survey of the state-of-the-art[END_REF]. Examples of established autoML frameworks are Auto-WEKA [START_REF] Kotthoff | Auto-weka 2.0: Automatic model selection and hpo in weka[END_REF] and Auto-Sklearn [START_REF] Feurer | Auto-sklearn 2.0: The next generation[END_REF] which are two automated systems that have been implemented on top of the well-known ML software scikit-learn and WEKA, respectively, GAMA [START_REF] Gijsbers | Gama: genetic automated machine learning assistant[END_REF], TPOT [START_REF] Le | Scaling tree-based automated machine learning to biomedical big data with a feature set selector[END_REF], and H 2 O [START_REF] Ledell | H2O AutoML: Scalable automatic machine learning[END_REF]. These solutions mainly differ in their search space and Hyper-Parameter Optimization (HPO) technique. For instance, Auto-WEKA uses a random forest algorithm as HPO, Auto-Sklearn exploits a Bayesian optimization technique [START_REF] Hutter | Automated machine learning[END_REF], H 2 O, GAMA, and TPOT employs a grid search approach. However, these tools and techniques exclusively work in the batch setting and cannot work with evolving data streams because they allow, among others, several access to data instances and therefore break the requirements of the streaming framework, notably the one-pass constraint.

The first attempts to improve the performance of ML methods, and make them automated, have been proposed using ensemble and meta-learning approaches. The first attempts for automated solutions in the classification task consist of using an ensemble and selecting the best-performing classifier on the last window to be used for the next window of instances as the BLAST [START_REF] Van Rijn | The online performance estimation framework: heterogeneous ensemble learning for data streams[END_REF]. However, this method operates as a heterogeneous ensemble and does not tune the parameters. The few existing automated solutions in classification, such as EvoAutoML [START_REF] Kulbach | Evolutionbased online automated machine learning[END_REF], use an ensemble and precise a diverse set of algorithms with fixed hyper-parameter values from which a solution is randomly selected whenever the current models in the ensemble are evaluated and the worst one has to be changed. Nevertheless, in such solutions, there is no automated tuning because they operate as heterogeneous ensembles where the parameters' values are already fixed.

In the clustering task, a first solution, called Conf-Stream [START_REF] Carnein | confstream: Automated algorithm selection and configuration of stream clustering[END_REF], for automated configuration for clustering evolving data streams has been proposed. It uses an ensemble of a diverse set of stream clustering algorithms with different configurations and the best-performing one will be used to create a new one from it that will replace one of the configurations in the ensemble (if it is good enough). The silhouette width measure has been used to evaluate the clustering quality of every configuration in ConfStream. Therefore, the configuration with the best cluster quality is selected for the next window of the stream. However, ConfStream uses periodically a small part of the stream (window) to select the model and the configuration that obtain the best performance and use them for the next window from the stream, and so on. Albeit it adopts a batch-incremental strategy, ConfStream is a very promising method for unsupervised autoML for data streams that could be extended to serve the supervised setting.

III. AUTOMATED CLASSIFICATION FOR DATA STREAMS

Classification algorithms require usually hyper-parameters to be set a priori, such as the k-nearest neighbors algorithm which has one parameter k, and the stream version of the decision tree, Hoeffding tree [START_REF] Domingos | Mining high-speed data streams[END_REF], which requires to set the grace period (the number of instances a lead should observe between split attempts) and the split confidence. In practice, it is difficult to find appropriate values for these hyperparameters that potentially need to be changed for each dataset and whenever the data distribution changes. Whence the need for a method that naturally adapts its configurations within a small ensemble to enable the adaptation of algorithm hyperparameter search spaces to data streams.

The question of changing and adapting the configuration of an algorithm as well as the orchestration of models while considering the requirements of the stream setting remains open. One viable solution is to propose a supervised version of ConfStream [START_REF] Carnein | confstream: Automated algorithm selection and configuration of stream clustering[END_REF] where the dynamic adaptation of the hyperparameters is mandatory to have a fully automated method and explore the HPO that could be used with classification to improve the accuracy.

As previously mentioned, an autoML method, based on algorithm configuration and selection, comprises (i) a configuration space that consists of a set of algorithms that are going to be selected and the corresponding ranges for their hyper-parameters, and (ii) an experimental setup to evaluate the candidate configurations composed of {algorithms, hyper-parameters} based on a predefined quality measure that enables the autoML method to select the best performing configuration. In this work, we propose an ensemble approach for automated data stream classification, called autoClass, that considers the configuration space and setup and aims at predicting class labels for potentially infinite data streams using the best-performing model. Our main idea is depicted in Fig. 1 and explained in the following sections.

A. Configuration space

The configuration space models a standard ML pipeline that comprises main high-level components which are the learning and prediction. Other components such as preprocessing including feature selection and extraction could be freely added to the pipeline. Definition 1: Data stream CASH, adapted from [START_REF] Feurer | Hyperparameter optimization[END_REF]. Let A = {A (1) , . . . , A (n) } be a set of n different stream classifiers, and let the hyper-parameters of each algorithm A (j) have a domain Λ (j) . Let S be a potentially infinite stream of instances {x i , y i }, and S t = {x 0 , y 0 }, . . . , {x t , y t } be an ordered sequence of past examples, where x i is a tuple of attributes and y i is the true class label that we aim to predict. Let L(M-→ A , -→ λ (S T ), S V ) denote the loss that algorithm combination M (j) achieves on a set of validation examples S V ⊂ S t when trained on S T ⊂ S t with hyperparameters -→ λ . Denote that the algorithm is used for testing using S V before being used for training using S T using the Test-Then-Train validation.

Then the stream CASH problem is to find the joint algorithm combination and hyper-parameter setting that minimize the loss:

- → A * , - → λ * ∈ arg min M (j) ∈M,λ∈Λ (j) ,A∈A L(M-→ A , - → λ (S T ), S V ) (1)
Existing ensemble-based methods for data streams [START_REF] Bifet | Leveraging bagging for evolving data streams[END_REF]- [START_REF] Gomes | Streaming random patches for evolving data stream classification[END_REF] do not fully cover the stream CASH problem since their hyper-parameters are usually set at the start of the stream processing and are not changed as the stream evolves. Moreover, they are mostly composed of homogeneous algorithms while our proposed automated solution addresses the online CASH and considers the configuration space for each algorithm.

In Algorithm 1, we present the training phase of the autoClass algorithm that adapts the algorithm configuration in an online manner. The first step of our approach consists of having a set of classifiers generated from A and their hyper-parameter ranges that constitute the main core of our automated solution. The learning phase of this component consists of learning on incoming data from the stream and is performed on an ensemble M of size s of candidate methods from the configuration space that are composed of different algorithms and different hyper-parameter values. Each instance received from the stream will be used to train all learners in M. Besides, given an unlabeled instance from the stream, the prediction is made using the best-performing algorithm inside the ensemble. In this work, since data are evolving and are not all available in the memory, we use the Test-Then-Train evaluation method where each instance from the stream is firstly used for prediction by testing the best-performing classifier, then using it to train all the classifiers in M.

(x, y) ← N ext(S)
15:

if t mod w == 0 then ▷ Each w instances 16:

M best ← min M ∈M L(M (S T ), S V )
17:

M worst ← max M ∈M L(M (S T ), S V )
18:

M mut ← Mutate(M best )
19:

M ← M ∪ M mut ▷ Add

22:

for M ∈ M do ▷ Update the ensemble

23:

M .fit(x, y)

24:

end for 25:

t ← t + 1
26: end while

B. Configuration setup

The most important factors involved in the configuration setup concern (i) the performance metric used to evaluate the different configurations within M and (ii) the generation of new configurations, i.e., {algorithms, hyper-parameters}.

Performance metric. The choice of the performance metric is related to the task one wants to address. For example, for clustering, typical measures include the Sum of Squares Error (SSE) and silhouette. For regression, one can use Mean Squared Error (MSE). In our classification case, since we deal with balanced datasets, we adopt the traditional accuracy as a quality measure computed as the number of correct predictions over the total number of predictions.

Generation of new configurations. We build an ensemble of candidate algorithms that contains the set of chosen algorithms with their default hyper-parameter values (lines 8-11, Algorithm 1). This solution is automated, i.e., the configuration (the representative algorithm and its hyper-parameter values) may change over time since data are evolving. So, the currently learned classifiers in M may no longer be representative of the next upcoming data, thus classifiers and their hyper-parameters need to change. For this to be achieved, we process the stream in sliding window [START_REF] Bahri | Data stream analysis: Foundations, major tasks and tools[END_REF] of a fixed size w by keeping the most recent instances from the stream at each time (line 15, Algorithm 1). We evaluate the predictive performance of every algorithm and configuration in the ensemble M and use the method with the highest accuracy for prediction.

Periodically, at the end of each window, a regression model is applied to the observed performance to predict the performance for unknown configurations. Besides, the bestperforming configuration in the ensemble is used to create a new configuration (child) from it using the truncated normal distribution [START_REF] Robert | Simulation of truncated normal variables[END_REF]. If the predicted performance is good enough, it replaces the worst configuration in the ensemble (lines 16-20, Algorithm 1). For each w processed instances, we evaluate the predictive performance of the algorithms, as depicted in Fig. 1 (step 1). To obtain new configurations, a configuration is sampled from the ensemble to serve as a parent (step 2) to derive a new configuration from it. The sampling is performed proportionally to the predictive performance of the ensemble members where better-performing algorithms have a higher probability to be selected. For this, we used roulette wheel selection, also called fitness proportionate selection, commonly used in genetic algorithms for selecting potential configurations in the ensemble. In our case, the fitness function f consists of obtaining the predictive performance of each configuration in order to associate a probability of selection with each configuration. Let us suppose that f i is the fitness of configuration i in our ensemble M, its probability of being selected is:

p i = f i s j=1 f j , ( 2
)
where s is the size of the ensemble and j represents the jth configuration. Moreover, we calculate the Cumulative Probability (CP), q i , for each learner within the ensemble as follows:

q i = i j=1 p j . (3)
Then, we generate a uniform random number r ∈ [0, 1] and compare it with the CP. Based on this comparison, we, therefore, make a decision about which of the configuration c i is going to be selected as a parent as follows:

1) if r < q 1 then select the first configuration c 1 , 2) else, q i-1 < r ≤ q i , select c i .

We do not necessarily choose the best-performing configuration as in rank selection in order to increase the ensemble diversity and avoid having the same algorithm with different hyper-parameter values in our ensemble. The sampled algorithm and configuration are therefore used as a parent to create an offspring and allow for mutations, as shown in Fig. 1, step 3. Similarly to confStream [START_REF] Carnein | confstream: Automated algorithm selection and configuration of stream clustering[END_REF], [START_REF] López-Ibáñez | The irace package: Iterated racing for automatic algorithm configuration[END_REF], we consider that every parameter of every configuration maintains an associated probability distribution. Every numerical parameter has a truncated normal numerical distribution N (µ, σ), where µ is the expectation and σ is the standard deviation that is derived from a normal distribution by bounding the random variable within a limited range of values. This truncated distribution is derived from a normal distribution by bounding the random variable within a limited range of values.

For a new configuration, we place µ at the position of the parent. Thereafter, the standard deviation σ of the child is reduced to increase the exploitation of this area using a fading factor λ through a decay function as follows:

σ t+1 = σ t × 2 -λ , (4)
where σ t and σ t+1 are the standard deviation of the parent and child, respectively. For integer parameters, the same process is performed to do the sampling but the result is rounded to the nearest integer. Ordinal parameters are a kind of categorical data with a set order that is processed similarly to integer parameters using an integer sampling strategy where the sampled integer is used as the index in the list of possible ordinal values. An instance example of ordinal parameters from decision trees is the split criterion parameter such as Gini, Information gain, Standard deviation reduction, and Variance reduction to which is assigned a numerical value from 1-4 as the index. For categorical parameters, we maintain a list of probabilities for each outcome and start with equal probabilities. The new configuration is sampled based on the latter list and the probability of its corresponding category is increased to serve exploitation. Similarly to confStream, the probability is increased by the same quantity that we reduce σ with numerical parameters as follows:

p t+1 = p t × (2 -2 -λ ). (5)
To obtain the final probabilities, the values are therefore normalized to sum one.

IV. EXPERIMENTAL STUDY

We present experiments to show the abilities of our proposal in the three axes: (i) the accuracy obtained as the final percentage (%) of instances correctly classified; (ii) the memory in megabytes (MB) used to maintain the learned models or/and statistical information about the models; and (iii) the running time in seconds (s) required to learn and predict from data and also for tuning.

A. Datasets

In our experiments, we used a diverse set of both synthetic and real datasets in different scenarios. We used data generators and real data, most of which have been thoroughly used in the literature [START_REF] Gomes | Adaptive random forests for evolving data stream classification[END_REF], [START_REF] Gomes | Streaming random patches for evolving data stream classification[END_REF], [START_REF] Bahri | Sketch-based naive bayes algorithms for evolving data streams[END_REF]- [START_REF] Bahri | Efficient batchincremental classification using umap for evolving data streams[END_REF] to assess the performance of data stream classification algorithms used notably in the original papers introducing the baseline algorithms [START_REF] Kulbach | Evolutionbased online automated machine learning[END_REF], [START_REF] Domingos | Mining high-speed data streams[END_REF], [START_REF] Gomes | Adaptive random forests for evolving data stream classification[END_REF], [START_REF] Gomes | Streaming random patches for evolving data stream classification[END_REF], [START_REF] Bifet | Efficient data stream classification via probabilistic adaptive windows[END_REF]. Table I shows an overview of the datasets used for experiments and further details are provided in what follows.

AGR. The AGR a and AGR g datasets are based on the AGRAWAL generator [START_REF] Agrawal | Database mining: A performance perspective[END_REF] which creates data streams with 9 attributes and 2 classes. A perturbation factor is used to change the original value of the data by adding a deviation value to it. AGR is used to simulate 3 gradual drifts in the generated stream. AGR g simulates 3 gradual drifts, while AGR a simulates 3 abrupt drifts.

RBF. The RBF f and RBF m datasets were generated using the Radial Basis Function (RBF) generator. The latter creates centroids at random positions, and each one has a standard deviation, a weight and a class label. This dataset simulates drift by moving the centroids at a constant speed. Both RBF m and RBF f were parametrized with 50 centroids and all of them drift. RBF m simulates a "moderate" incremental drift (speed of change set to 0.0001) while RBF f simulates a more challenging "fast" incremental drift (speed of change set to 0.001).

Hyper. The Hyperplane generator [START_REF] Hulten | Mining time-changing data streams[END_REF] is used to generate streams with gradual concept drift by changing the values of its weights. We parameterize HYP with 10 attributes and a magnitude of change equal to 0.001. CNAE. CNAE is the national classification of economic activities dataset, initially used in [START_REF] Ciarelli | Agglomeration and elimination of terms for dimensionality reduction[END_REF]. It contains 1,080 instances, each of 856 attributes, representing descriptions of Brazilian companies categorized into 9 classes. The original texts were preprocessed to obtain the current highly sparse dataset.

Poker. The Poker hand dataset consists of 829,201 instances and 10 attributes. Each instance of the Poker-Hand dataset is an example of a hand consisting of five playing cards.

Covertype. The forest covertype dataset was obtained from US Forest Service (USFS) Region 2 Resource Information System (RIS) data. It contains 54 attributes and 7 classes.

KDD99. KDD cup'992 for network intrusion detection. This dataset contains 41 attributes and 23 classes. It has been often used to evaluate big data stream algorithms' performance.

Airlines. The Airlines dataset was inspired by the regression dataset from Ikonomovska3 . The task is to predict whether a given flight will be delayed given information on the scheduled departure. Thus, it has 2 possible classes: delayed or not delayed. This data set contains 539, 383 records with 7 attributes.

B. Search space

We aim to select the optimal algorithm and its hyperparameterization among the available stream classification algorithms in MOA [START_REF] Bifet | Moa: Massive online analysis[END_REF] for a specific stream. To do so, we consider a representative subset of the baseline classifiers and particularly use state-of-the-art classification algorithms [START_REF] Bahri | Data stream analysis: Foundations, major tasks and tools[END_REF]: Hoeffding Tree (HT) [START_REF] Domingos | Mining high-speed data streams[END_REF], Hoeffding Adaptive Tree (HAT) [START_REF] Bifet | Adaptive learning from evolving data streams[END_REF], kNN, and kNN with ADWIN (kNN a ) [START_REF] Bifet | Efficient data stream classification via probabilistic adaptive windows[END_REF] to handle concept drifts. For each algorithm, we aim to optimize the parameters that influence the classification prediction. For example, the kNN algorithm can be parameterized by the number of neighbors k. The details on the hyper-parameters and their ranges for every algorithm used are explained in Table II. Besides, more algorithms and hyper-parameters can be tuned and considered in this approach.

C. Experimental setup

To evaluate our approach, we implemented it in Java as a classification algorithm under the MOA software. We evaluated our approach using a population size s = |M| = 10 and a window of size w = 1000, where we evaluate our s learners for each w instances. All algorithms are trained in an ensemble, so a high number of learners in the ensemble or big window (w) would lead to expensive computation resource usage. The parameters s and w have been fixed based on an evaluation that has been done during the implementation of this approach and supported by previous empirical studies [START_REF] Gomes | Adaptive random forests for evolving data stream classification[END_REF], [START_REF] Bifet | Efficient data stream classification via probabilistic adaptive windows[END_REF]. We show an extensive evaluation against wellknown state-of-the-art single algorithms that are widely used in comparison for the data stream classification task and also ensemble algorithms, such as Adaptive Random Forest [START_REF] Gomes | Adaptive random forests for evolving data stream classification[END_REF], Stream Random Patches (SRP) [START_REF] Gomes | Streaming random patches for evolving data stream classification[END_REF], and Leveraging Bagging (LB) [START_REF] Bifet | Leveraging bagging for evolving data streams[END_REF]. While current ensemble-based methods are formed of homogeneous algorithms, autoClass is able to handle diverse algorithms and pipelines. We also compare against the unique, as far as we know, automated solution for stream classification, EvoAutoML [START_REF] Kulbach | Evolutionbased online automated machine learning[END_REF].

For a fair comparison, we used 10 learners for the ensemblebased competitors as the population size s of our approach with their default configuration as presented in Table II.

D. Results

In this section, we present the results of our approach and compare them against those of state-of-the-art classifiers. To do so, autoClass is initialized with the same default hyperparameterization of the baselines, as shown in Table II, and optimizes the parameters ahead. In Fig. 2, we depict the performance of autoClass and the baselines for the Poker dataset and show its ability to adapt and react to temporal changes along the stream. One can see that the accuracy of the competitors decreases at ∼ 200000 evaluated instances while autoClass remains steady. AutoClass is more accurate and more efficient in terms of memory than EvoAutoML, LB, ARF, SRP, and HT. However, in terms of running time, single-based algorithms (e.g., HT and kNN) obtain better performance (lower values). Moreover, autoClass takes more time for learning and prediction than ensemble-based learners because, in addition, it does hyper-parameters optimization and adapts them while the data stream evolves. We also point out that the only automated solution for stream classification so far, EvoAutoML, is more costly in terms of resources than our autoClass and other baselines.

resources.

Table III shows that autoClass is very competitive in terms of predictive performance in comparison with competitors. We can see that on the overall average accuracy of 87.43% autoClass outperforms the baselines. In comparison with the top single classifiers, autoClass performs, at least, 10% better on average. It obtains the best accurate performance on almost all the datasets except for Hyper, CNAE, and Covertype data it achieves the second-best result where the difference with the highest accuracy is not huge. It is very competitive and obtains better results than well-known ensemble-based methods such as LB, ARF, and SRP because our approach is as diverse as these aforementioned ensemble methods. AutoClass is effective with datasets that contain different types of concept drift as AGR and RBF thanks to the dynamic and adaptive configuration strategy. The latter changes the best-performing method used for prediction when its accuracy decreases because of the drift. Moreover, through its additional hyper-parameter optimization component, autoClass achieves better predictive performance than its rival EvoAutoML [START_REF] Kulbach | Evolutionbased online automated machine learning[END_REF] on all datasets taking into account that the latter is based on an evolutionary strategy where methods and configurations are fixed at the beginning without tuning. In Fig. 3, we show the time needed to predict and learn from different data streams. It shows that the better predictive performance of ensemble-based approaches is accompanied by higher time consumption than with single algorithms. However comparing the consumption of ensemble classifiers, we can observe that EvoAutoML has undeniably the highest time consumption of all the datasets. AutoClass consumes more time than LB, ARF, and SRP methods on almost all the datasets due to the hyper-parameter tuning component where we need to optimize the configurations to obtain the optimal method with the current concept.

In terms of memory consumption, as highlighted in Table IV, single classifiers, mainly kNN and kNN a , are more efficient because they only maintain a fixed size sliding window over the whole stream. Similar to the time behavior, autoClass is more efficient than EvoAutoML, LB, ARF, and SRP for almost all the datasets because it uses neighbor-based approaches in its ensemble M which turns out to be memory efficient since the size of the window limits them. It performs even better than the single HT and HAT which may be the result of (i) the existence of drifts as revealed AGR and RBF datasets which implies that the trees continue to grow up while with autoClass, a new learn starts empty; and/or (ii) our ensemble of size s is composed of multiple kNN-based methods which consume few AutoClass meets the data stream resource requirements by providing high predictive performance and consuming less memory and time than EvoAutoML while consuming relatively more in comparison with the non-automated single and ensemble state-of-the-art algorithms. Nevertheless, autoClass outperforms these baselines, in terms of predictive performance, in a common test-then-train evaluation which shows its ability to adapt to changes in data distributions thanks to the employed hyper-parameters tuning mechanism.

Fig. 1 :

 1 Fig. 1: AutoClass.

Algorithm 1 autoClass training 1 :

 1 Input: 2: Data stream S, Ensemble size s, sampling rate w, loss function L, configuration space A, Λ 3: Output: 4: Set of suited algorithms configurations: 5: M = {M (1) , . . . , M (s) } 6: 7: M ← ∅ ▷ Initialization 8: while |M| < s AND M is ∅ do 9: M ← Add(A, Λ) ▷ Add the algorithms in A with the default parameters 10: M ← M ∪ M 11: end while 12: t ← 0 13: while HasN ext(S) do ▷ Start the data stream 14:

Fig. 2 :

 2 Fig. 2: Accuracy, time, and memory performance for autoClass and baselines.

Fig. 3 :

 3 Fig. 3: Running time performance for autoClass and baseline approaches.

TABLE I :

 I Overview of the datasets.

	Dataset	#Inst	#Att	#Classes	Type
	AGRa	1,000,000	9	2	Synthetic
	AGRg	1,000,000	9	2	Synthetic
	RBF f	1,000,000	10	5	Synthetic
	RBFm	1,000,000	10	5	Synthetic
	Hyper	1,000,000	10	2	Synthetic
	CNAE	1,080	856	9	Real
	Poker	829,201	10	10	Real
	Covtype	581,012	54	7	Real
	KDD	4,898,431	41	23	Real
	Airlines	539,383	7	2	Real

TABLE II :

 II The algorithms and their hyper-parameter ranges.

	Algorithm	Parameter	Type	default	range
	HT	g	Integer	200	[10,200]
		c	Numeric	0.01	[0,1]
	HAT	g	Integer	200	[10,200]
		c	Numeric	0.01	[0,1]
	kNN	k	Integer	10	[2,30]
	kNN a	k	Integer	10	[2,30]

TABLE III :

 III Accuracy comparison.Bold and underlined values indicate the best and the second best results per dataset, respectively.

	Dataset	autoClass	EvoAutoML	LB	ARF	SRP	HT	HAT	kNN	kNN a
	AGRa	92.21	86.17	89.65	85.51	90.50	88.80	91.06	84.71	85.83
	AGRg	88.46	85.91	83.86	80.48	86.71	81.06	87.74	82.43	83.48
	RBF f	86.37	85.34	55.43	71.09	70.83	33.79	48.68	84.03	80.70
	RBFm	90.39	85.59	83.88	84.69	83.26	53.53	73.12	89.35	89.95
	Hyper	86.59	85.19	87.56	84.54	81.85	76.93	84.97	83.32	83.87
	CNAE	73.33	39.62	55.92	55.46	76.20	55.92	55.83	71.75	69.07
	Poker	95.21	91.49	87.59	87.11	86.89	94.91	77.85	76.57	76.48
	Covertype	94.46	94.29	91.69	94.39	95.04	83.62	86.99	92.59	92.10
	KDD	99.98	99.97	99.96	99.96	99.97	99.91	99.91	99.97	99.97
	Airlines	67.34	61.80	63.14	61.34	61.29	61.02	61.56	59.54	60.16
	Overall ∅	87.43	81.54	79.87	80.46	83.25	72.95	76.77	82.43	82.16

TABLE IV :

 IV Memory comparison.

	Dataset	autoClass	EvoAutoML	LB	ARF	SRP	HT	HAT	kNN	kNN a
	AGRa	0.025	248.63	0.066	0.129	0.127	0.637	0.703	0.002	0.005
	AGRg	0.024	289.80	0.065	0.130	0.124	0.679	0.661	0.002	0.005
	RBFf	0.025	126.35	0.052	0.118	0.106	0.103	0.015	0.002	0.005
	RBFm	0.025	127.17	0.059	0.116	0.122	0.200	0.127	0.002	0.006
	Hyper	0.025	296.52	0.064	0.110	0.119	0.612	0.004	0.002	0.005
	CNAE	0.096	242.42	0.105	0.177	0.239	0.001	0.002	0.002	0.054
	Poker	0.024	125.27	0.060	0.142	0.123	0.248	0.006	0.002	0.004
	Covertype	0.025	129.22	0.062	0.125	0.109	0.246	0.008	0.008	0.011
	KDD	0.027	129.06	0.067	0.195	0.176	0.035	0.003	0.003	0.006
	Airlines	0.025	150.96	0.053	0.148	0.107	0.623	0.112	0.002	0.004
	Overall ∅	0.032	186.54	0.065	0.139	0.135	0.338	0.364	0.002	0.010

In compliance with literature terminology, we use the terms online and stream learning interchangeably.

http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html

https://www.openml.org/search?type=data&sort=runs&id=42493&status= active

ACKNOWLEDGEMENTS

This work has been partially supported and funded by the EU Horizon Europe SEDIMARK (SEcure Decentralised Intelligent Data MARKetplace) project (Grant no. 101070074).

V. CONCLUDING REMARKS AND FUTURE DIRECTIONS

In this paper, we proposed an automated approach for data stream classification that automatically tunes the hyperparameters of stream classifiers and dynamically adapts to the eventual changes in data distributions. AutoClass outperforms the only existing automated solution to the best of our knowledge for the classification of data streams, EvoAutoML, in terms of accuracy and resources. Moreover, it obtains better predictive performance than state-of-the-art algorithms on almost all datasets while providing a good tradeoff accuracyresources. Other stream classifiers can be easily added to this automated solution.