Studzinski Filipo

Perotto

Mathieu Bourgais
email: mathieu.bourgais@insa-rouen.fr

Laurent Vercouter
email: laurent.vercouter@

Bruno Castro Da Silva

Alina Beygelzimer

Daniel Hsu

F S Perotto
email: filipo.perotto@litislab.fr

B C Silva
email: bsilva@inf.ufrgs.br

32nd Annual Conference on Learning Theory Open Problem: Risk of Ruin in Multiarmed Bandits

Keywords: Budgeted Multiarmed Bandits, Ruin Theory, Risk-Averse Decision Making

We formalize a particular class of problems called survival multiarmed bandits (S-MAB), which constitutes a modified version of budgeted multiarmed bandits (B-MAB) where a true risk of ruin must be considered, bringing it closer to risk-averse multiarmed bandits (RA-MAB). In a S-MAB, pulling an arm can result in both positive and negative rewards. The agent has an initial budget that evolves in time with the received rewards. The goal is finding a good exploration-exploitationsafety trade-off, maximizing rewards while minimizing the probability of getting ruined (i.e. hitting a negative budget). Such simple and until now neglected modification in the MAB statement changes the way to approach the problem, asking for adapted algorithms and specific analytical tools, and also make it more likely related to some important real-world applications. We are interested in the following open problems which stem from such new MAB definition: (a) how can the regret be meaningfully defined in formal terms for a S-MAB given its multiobjective optimization nature? (b) can a S-MAB be reduced to a RA-MAB or a B-MAB, transferring their theoretical guarantees? (c) what kind of method or strategy must an agent follow to optimally solve a S-MAB?

Introduction

Multiarmed Bandits (MAB) constitute a framework to model online sequential decision-making while facing the exploration-exploitation dilemma. A MAB is typically represented by an agent interacting with a process by choosing some action i to perform among k possible actions (or "arms"), then receiving a corresponding reward r. Because the complete information about the reward functions is not available, the agent must estimate their parameters by sampling them (i.e. by pulling the arms and observing the received rewards). Solving a MAB means maximizing the expected rewards to be received by the agent over time, finding the best trade-off between exploration (sampling arms) and exploitation (pulling the estimated best arm). Different methods and guarantees have been proposed in the literature depending on the available information and on the assumptions made about the reward functions [START_REF] Bubeck | X-armed bandits[END_REF]. However, most of the existing bandit-related research ignores any notion of cost and the possible finitude of the resources for pulling arms.

In this paper we are interested in a class of problems that we call survival multiarmed bandits (S-MAB), which constitutes a version of MAB where a true risk of ruin must be considered. In a S-MAB, an initial budget b 0 evolves in time following the received rewards, so as b t+1 = b t + r t . The returned reward can just as well be positive as negative, and affects the budget, which corresponds to the agent's "capital" or "fortune". Therefore, solving a S-MAB means estimating the best arm, maximizing the budget over time, but managing the costs of exploratory actions to avoid ruin. To do it, the agent must consider the uncertainty on the expected rewards, which can be a consequence of both high distribution variances and low confidence on the estimations.

If we see the budget as an essential variable, e.g. like an "energy level", a S-MAB becomes a survival problem, and can be associated to the cybernetic notion of homeostasis: the agent wants to learn the optimal action but remaining alive, i.e. preserving a positive budget during all its lifetime. The MAB problem defined as a S-MAB becomes more closely related to some general real-world problems: while trading-off between exploration and exploitation, trying to maximize their winnings, gamblers in a casino have some money and must avoid ruin, investors in the stock market have a bankroll and must avoid bankruptcy, organisms in the nature have essential variables, inherent to their structure, and must avoid death. The exploration-exploitation dilemma becomes more complex when such risk of ruin is taken into account.

Survival MAB

Considering a stochastic approach, a S-MAB can be formally defined as M = {I, F, b 0 }, where I = {1, ..., k} is the set of possible actions, F = {f 1 , ..., f k } is the set of reward distribution functions, and {b 0 ∈ R | b 0 ≥ 0} is the initial budget. The process evolves discretely over time. Each observation of a given arm is independent and identically distributed. Let a t = i be the action chosen (the arm pulled) and r t be the immediate reward received at time t (drawn from f i ). A stationary reward distribution function f i ∈ F defines the probability of having reward r t ∈ R after pulling arm i at time t, i.e. {r t ∼ f i | a t = i}. Received rewards impact the budget, increasing or decreasing it in the form b h = b 0 + h t=1 r t , where b h is the budget in time h, and r t is the reward received in time t.

In a S-MAB, the reward support D = [r min , r max ] is given, where {r min < 0 < r max }. At least one arm presents a negative mean reward and at least one presents a positive mean, i.e. {∃i, j ∈ I | µ i < 0 < µ j }. In such scenario, depending on the initial budget, on the arms' parameters, and on the chosen strategy, and supposing an infinite time-horizon (h → ∞), the agent can either increase the probability of running the process indefinitely, becoming infinitely rich (i.e. τ → ∞, b τ → ∞), or inversely, can increase the probability of ruin, until eventually getting broke (i.e. τ < ∞, b τ < 0), with τ the time where the agent is ruined.

Classically, solving a MAB means finding a policy that can minimize the reward regret λ (i.e. the cumulated difference between the rewards expected by following the given strategy and the rewards that could be obtained by always pulling the arm with highest true mean reward, which is unknown to the agent) [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]. Considering a given time-horizon h (which can be infinite), the theoretical cumulative regret is defined as

λ h = h t=1 [µ * -µ at ], where µ * = max i∈I [µ i ].
However, in a S-MAB, the single reward regret minimization is not sufficient for evaluating the quality of an algorithm; strategies with very small regret can lead frequently to ruin. In fact, reducing the probability of ruin may imply an increasing regret, especially when considering a finite time-horizon. Thus, some other criterion must be also taken into account for optimizing survival. From a theoretical point of view, solving a S-MAB can be defined as the problem of finding an optimal sequence of actions A * h = {a 1 , ..., a h } that minimize both the expected reward regret and the probability of being ruined, which can be formally stated as a multiobjective optimization:

A * h = arg min A h [λ h , P † h ]
, where λ h is the theoretical cumulative regret expected on the (poten-tially infinite) time-horizon h and P † h is the probability of being ruined before the time-horizon h, while executing the sequence of actions A h . Note that, such kind of optimization produces a Paretooptimal frontier. This issue gives raise to our first question: (a) how can the regret be meaningfully defined in formal terms for a S-MAB given its multiobjective optimization nature?

Related Works: Budget and Risk

In the budgeted multiarmed bandit (B-MAB) setting, the player receives a reward but needs to pay a cost after pulling an arm. There are two independent functions associated to each action: cost and reward. The costs of pulling arms are taken from a given initial budget which limits the time-horizon of the process. No more arms can be pulled after running out of budget. When the budget is over, the game is over. The optimal arm can vary on time depending on the remaining budget. The agent needs to explore and estimate not only the reward function of an arm but also its cost function. Key results and algorithms developed for B-MAB are [START_REF] Xia | Finite budget analysis of multi-armed bandit problems[END_REF][START_REF] Ding | Multi-armed bandit with budget constraint and variable costs[END_REF]. In such references, the common strategy for evaluating arms considering these two different functions (rewards f i and costs c i ) is based on a coefficient (called density or reward-to-cost ratio) which divides the expected reward μr i of arm i by its expected cost μc i , i.e. ŵi,t = μr i/ μc i .

In a similar setting called bandits with knapsacks, each arm is associated to d different costs that stochastically consume d different budgets [START_REF] Badanidiyuru | Bandits with knapsacks[END_REF]. Other variations of the budgeted problem considering specific scenarios can be found in [START_REF] Zhou | Budget-constrained multi-armed bandits with multiple plays[END_REF][START_REF] Koren | Multi-armed bandits with metric movement costs[END_REF][START_REF] Agrawal | An efficient algorithm for contextual bandits with knapsacks, and an extension to concave objectives[END_REF]. Another version of the budgeted problem called "best arm identification" [START_REF] Audibert | Best arm identification in multi-armed bandits[END_REF] considers that pulling an arm has a cost but the budget is only imposed on a preliminary exploration phase. The subsequent exploitation phase is not associated with costs nor constrained by budgets, which constitutes a "pure exploration" problem.

Alternatively, in the risk-averse multiarmed bandit (RA-MAB) setting, the agent must take into account the expected variability on the expected rewards in order to identify (and avoid) less predictable (then considered risky) actions, but without worrying about ruin since no notion of budget is considered. The notion of "safety" is then understood as "stability" or "predictability" of the rewards. In this sense, the risk-reward trade-off can be tackled by using some mean-variance metric like ŵi,t = μi -σ2 i , where { ∈ R | 0 < < ∞} represents the risk aversion, μi is the estimated mean reward of arm i, and σ2

i is its estimated variance [START_REF] Sani | Risk-aversion in multi-armed bandits[END_REF][START_REF] Vakili | Risk-averse multi-armed bandit problems under mean-variance measure[END_REF]. Another criterion for evaluating the riskiness of an arm is its conditional value at risk, based on the quantiles (or lower-tails) of the reward distribution. In such proposition, the utility of an arm i is not based on its estimated mean reward, but on Φ i (α), its estimated cumulative distribution function, where {α ∈ R | 0 ≤ α ≤ 1} represents the risk tolerance, and indicates the reference quantile [START_REF] Galichet | Exploration vs exploitation vs safety: Risk-aware multiarmed bandits[END_REF][START_REF] Maillard | Robust risk-averse stochastic multi-armed bandits[END_REF][START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF]. In another variation of the problem called conservative bandits [START_REF] Wu | Conservative bandits[END_REF], the agent also have a budget determined by the obtained rewards, but the constraint is also related to the regret. The agent must minimize regret in the long-term, but such regret must not exceed some relative amount. The idea is to limit wild exploration by defining the notion of immediate acceptable loss.

What makes S-MAB different of B-MAB is that the same single reward function impacts the budget and defines the objective (cumulated reward maximization). There is no separated cost function, and for such reason, there is no cost-reward ratio. Inverselly, RA-MAB does not consider any notion of budget. The risk is understood as unpredictability, whereas in S-MAB the risk is the ruin. Such differences motivate the last two questions: (b) can a S-MAB be reduced to a RA-MAB or a B-MAB, transferring their theoretical guarantees? and finally (c) what kind of method or strategy must an agent follow to optimally solve a S-MAB?


M Forget

V Bos

S Carrizo

With social science methods (direct and semi-direct interviews) we analyse the public policies and the exploitation processes of the three main salt-lake brines (Bolivia, Argentina, Chile)

Main extractive stakeholders public-private partnership public

Regulated by

The total lithium demand of a constant current lithium demand combined with growth of electric vehicles according to IEA's blue map scenario (Vikström et al., 2013)

*

  The desire to preserve the environment and the idealised view of mountains in the global North induces extractive impacts, "denaturing" of indigenous landscapes and possible irreversible environmental damage, in the mountain territories of the South where metals needed for new energy technologies are mostly mined * Energy transition means the use of intermitent energy and induce growth of lithium production destinated to lithium-ion batteries Map of resource avalaibility and production. Adaped from Grosjean and al., 2012 Results indicate di erent perceptions of the lithium resource : -mineral of the future, territorial development -economic oportunity, commodity -Relocalisation of Li-ion batteries chain productions in the extractive territories? -Developing the extractive processes in Global North rich lithium territories? * Behaviour-based regulations rather that technological choices (without another extractive pattern in Global South countries to mitigate industrial impacts?) * The energy transition, which, beyond its justi cation as a vital part of the response to global change, reveals enormous contrasts between the logics operating in raw materialextraction territories and territories transitioning toward new ways of producing energy EDYTEM Lab., Savoie Mont Blanc University, France marie.forget@univ-smb.fr, vincent.bos@univ-smb.fr CONICET, La Plata University, Argentina scarrizo@conicet.gouv.ar * Energy transition polices intent to mitigate the e ects of global change and preserve vulnerable mountains territories * Goals and impacts di er between global North and South -In global North development of lithium based devices (electric vehicles, solar energy storage, etc.) as an opportunity for «green» economy -In global South energy transition is more an opportunity to deliver some cheap electricity to deconected mountains territories * Spatial concentration of the extractive and battery production processes gathered in diferent poles


Filipo Studzinski Perotto
email: filipo.perotto@litislab.fr

Laurent Vercouter
email: laurent.vercouter@litislab.fr

Tuning the discount factor in order to reach average optimality on deterministic MDPs

Keywords: Average Dynamic Programming, Sensitive Dynamic Programming, Reinforcement Learning, Markovian Decision Process

Considering Markovian Decision Processes (MDPs), the meaning of an optimal policy depends on the optimality criterion chosen. The most common approach is to define the optimal policy as the one that maximizes the sum of discounted rewards. The intuitive alternative is to maximize the average reward per step. The former has strong convergence guarantees but suffers from the dependency on a discount factor. The latter has the additional inconvenience of being insensitive to different policies with equivalent average. This paper analyzes the impact of such different criteria on a series of experiments, and then provides a threshold for the discount factor in order to ensure average optimality for discounted-optimal policies in the deterministic case.

Introduction

Dynamic Programming (DP) refers to a set of algorithms that can efficiently compute optimal policies for Markovian Decision Processes (MDPs), providing essential foundations for Reinforcement Learning (RL) methods [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF][START_REF] Puterman | Dynamic programming[END_REF]. DP and RL algorithms are fundamentally based on discounted-optimality. In this setting, an optimal policy maximizes the sum of discounted rewards over time using a discount factor γ.

When considering infinite time-horizon, the use of discounted rewards constitutes an important key on guarantying polynomial time convergence for such methods [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF]. However, in many domains, the use of a discount factor does not present any relation to the optimization problem itself. Typically, when facing recurrent MDPs (where terminal states do not exist), discounting future rewards in favor of immediate rewards can introduce a kind of "distortion" on the real utility of a policy of actions [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF][START_REF] Mahadevan | Average reward reinforcement learning: Foundations, algorithms, and empirical results[END_REF][START_REF] Tadepalli | Average-reward reinforcement learning[END_REF].

The crawling robot problem [START_REF] Tokic | The crawler, a class room demonstrator for reinforcement learning[END_REF] offers an illustrative example of such issue. The robot is endowed with a single articulated arm, and some of its movements cause the displacement of the robot. The objective is finding the optimal cyclical sequence of actions in order to make the robot walk forward as fast as possible. Rewards correspond to immediate progressions. However, depending on the discount factor, a discounted-optimal policy can be unable to reach the maximum velocity. In other words, an intuitively optimal behavior can be seen as sub-optimal under the discounted framework. In fact, the robot reaches its maximum speed when it enters in the recurrent cycle of states that offers the highest average displacement per step. Other examples of this issue are given in [START_REF] Mahadevan | Average reward reinforcement learning: Foundations, algorithms, and empirical results[END_REF]. For such scenarios, maximizing the average reward per step is, in some sense, more appropriate, but a key limitation of such approach is that average-optimality cannot distinguish among policies which have the same recurrent average reward per step, but which are not necessarily equivalent in terms of transient rewards [START_REF] Mahadevan | Learning representation and control in markov decision processes: New frontiers[END_REF].

The discussion about optimality is not new [START_REF] Kallenberg | Finite state and action mdps[END_REF][START_REF] Cao | The n th -order bias optimality for multichain markov decision processes[END_REF][START_REF] Feinberg | Reduction of total-cost and average-cost mdps with weakly continuous transition probabilities to discounted mdps[END_REF], and is summarized in section 2. In practice, the discounted framework had been largely preferred. Such algorithms are easier to implement, and the polynomial convergence bounds are guaranteed for the general case [START_REF] Ye | The simplex and policy-iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate[END_REF][START_REF] Kalyanakrishnan | Batch-switching policy iteration[END_REF].

In fact, if the discount factor γ is sufficiently high, discounted-optimal policies become also average-optimal. How high γ needs to be depends on each particular setting (topology and rewards), and cannot be calculated beforehand. For that reason, it is a hard-to-tune parameter. Without any other information, such average optimality is only guaranteed in the limit when γ → 1 [START_REF] Blackwell | Discrete dynamic programming[END_REF][START_REF] Cao | The n th -order bias optimality for multichain markov decision processes[END_REF][START_REF] Hordijk | Blackwell optimality[END_REF]. However, the higher the discount factor, the slower the convergence of iterative methods. When γ approaches 1, the necessary time for convergence approaches ∞ [START_REF] Ye | The simplex and policy-iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate[END_REF].

How often are discounted-optimal policies not average-optimal? The first contribution of this paper is an analysis on the difference between discountedoptimality and average-optimality in terms of total reward loss on the long run depending on how γ is tuned. Using a set of experiments with random MDPs we show that the impact of the use of too low γ values is not negligible.

How can the discount factor be optimally tuned in practice? The second contribution of this paper is a method for calculating a threshold for γ in order to ensure average-optimality to discounted-optimal policies. In this paper, as it consists on a first approach to the problem, only deterministic MDPs will be considered.

The rest of the paper is organized as follows. Section 2 reviews related concepts and methods on computational sequential decision-making. Section 3 presents our contributions: an analysis about the impact in terms of reward loss on choosing either discounted or average optimality, and the deduction of a formula for identifying average-optimal discount factors. Section 4 concludes the paper.

Background: Markovian Decision Processes

Markovian Decision Processes (MDPs) are in the center of a widely-used framework for approaching automated control, sequential decision-making, planning, and computational reinforcement learning problems [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF][START_REF] Puterman | Dynamic programming[END_REF][START_REF] Sutton | Introduction to Reinforcement Learning[END_REF][START_REF]Markov Decision Processes in Artificial Intelligence[END_REF][START_REF] Wiering | Reinforcement learning and markov decision processes[END_REF][START_REF] Uther | Markov decision processes[END_REF].

An MDP works like a discrete stochastic finite state machine: at each time step the machine is in some state s, the agent observes that state and interacts with the process by choosing some action a to perform, then the machine changes into a new state s and gives the agent a corresponding reward r.

An MDP can be defined as a set M = {S, A, T, R} in the form: The transition function T defines the system dynamics by determining the next state s given the current state s and the executed action a. The reward function R defines the immediate reward r ∈ R after moving from state s to s with action a. Deterministic MDPs (D-MDPs) constitute the particular set of MDPs where the transitions are deterministic, in the form T : S × A → S.

M =          S =
Solving an MDP means finding a policy of actions that maximizes the rewards received by the agent, according to a given optimality criterion and a given timehorizon. The optimality criterion is defined by an utility function U . An optimal policy π * is a policy that cannot be improved:

U (s, π * ) ≥ U (s, π), ∀s ∈ S, ∀π ∈ Π (1)
where U (s, π) is the utility of following the policy π from the state s.

A deterministic stationary policy π is a mapping between states and actions in the form π : S → A. The number of such policies contained in Π, the set of possible policies, is exponential, and corresponds to |Π| = m n .

Discounted Optimality

When the stopping time h is finite and known, a simple solution consists in evaluating policies by estimating their total rewards. The utility function U is then equivalent to Z, the (undiscounted) sum of expected rewards:

Z h (s, π) = h t=1 R t (s, π) (2)
where R t (s, π) corresponds to the expected reward in time t starting from state s and following policy π. In that case an exact optimal policy can be found through beckward recursion [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF] in polynomial time, O(nmh). However, such solution cannot be applied when the time-horizon is infinite, unbounded or unknown. The standard approach to the infinite horizon setting consists in applying a discount factor {γ ∈ R | 0 < γ < 1} that reduces the weight of future rewards compared to immediate rewards in the sum. Such sum is always finite, which guarantees the convergence of iterative methods to an optimal solution [START_REF] Puterman | Dynamic programming[END_REF][START_REF] Wiering | Reinforcement learning and markov decision processes[END_REF]. The sum of discounted rewards V γ (s, π), starting in a given state s, following a given policy π, and for a given discount factor γ, is:

V γ (s, π) = lim h→∞ h t=1 γ t-1 R t (s, π) (3)
Tuning the discount factor implies a trade-off: the higher γ is (closer to 1), the better the chances of ensuring average optimality for discounted-optimal policies, but the bigger the computational costs for calculating the solution. The convergence time bound to compute discounted-optimal policies using iterative methods increases with rate O( 11-γ log 1 1-γ ) [START_REF] Ye | The simplex and policy-iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate[END_REF][START_REF] Scherrer | Improved and generalized upper bounds on the complexity of policy iteration[END_REF] Typical values of γ in the literature are 0.9 and 0.99. The inconvenience of using such generic suggestions is that, in certain circumstances, when γ is too low, discounted-optimality can lead the agent to a sub-optimal behavior in terms of average reward. We would like to call such phenomenon a "discount trap". a up to the infinity. So, considering an unbounded time-horizon (where the stopping time h of the process is likely to be greater than 100), the policy b would be preferred. However, given that V0.99(y, a) = +50 and V0.99(y, b) = +49, the discounted-optimal policy for any discount factor γ ≤ 0.99 is a. In (ii ), the average of the policy starting on the middle state y and choosing the action a is G(y, a) = +0.45 (the average per step on the cycle {y, z}, on the right). The action b presents G(y, b) = +0.5 (the average per step on the cycle {y, x}, on the left) and is gain-optimal. However, given that V0.9(y, a) = V0.9(y, b) ≈ 4.737, the discounted-optimal policy for any discount factor γ < 0.9 is a. In fact b becomes definitely better (i.e. get better total rewards) than a after 20 execution steps and up to the infinity.

Average (or Gain) Optimality

In many domains, there is no grounded interpretation for the discount factor γ. In addition, the value corresponding to the sum of discounted rewards is less human readable (i.e. harder to interpret) than the average reward per step. Moreover, in recurrent domains (where later rewards are as important as earlier rewards) the use of low values of γ can "distort" the utility of some sequence of actions. For such reasons, maximizing the average reward received per time step can be preferable [START_REF] Tadepalli | Model-based average reward reinforcement learning[END_REF][START_REF] Gosavi | A reinforcement learning algorithm based on policy iteration for average reward: Empirical results with yield management and convergence analysis[END_REF]. The average reward over an infinite time-horizon, called gain, of a policy π starting on state s, is:

G(s, π) = lim h→∞ 1 h h t=1 R t (s, π) (4)
The convenience of average-optimality compared to discounted-optimality can be observed regarding the MDPs shown in figure 1. On both problems, depending on the discount factor, the discounted-best policy can correspond to a clearly worse solution on the long run.

Considering unichain MDPs running over an infinite time-horizon, the average reward (or gain) of a given policy π converges to a single value g independently of the starting state [START_REF] Puterman | Dynamic programming[END_REF], i.e. G(s, π) = g, ∀s ∈ S. Considering multichain MDPs, there is a convergent gain for each communicant subset of states (i.e. for each recurrent class within the process). The major drawback of averageoptimal methods is that they have weaker convergence guarantees compared to discounted methods, even with the strong constraint of unichainess [START_REF] Hansen | Lower bounds for howard's algorithm for finding minimum mean-cost cycles[END_REF]. Worse yet, they are insensitive for distinguishing different policies with same average reward per step [START_REF] Lewis | Bias optimality[END_REF].

Sensitive (or Blackwell) Optimality

For a same MDP there may be several average-optimal policies which are not necessarily equivalent. That is the case regarding the examples in figure 2. On both cases, two possible policies converge to a common average reward as time approaches infinity. They are, for that reason, indistinguishable from an average reward point of view.

The problem is that rewards obtained in the transient path toward the recurrent states disappear on the infinite averaging. In the same way, the position of each reward inside a sequence of cyclical rewards also disappears. However, such differences are important when considering an unbounded (but finite) timehorizon. Even though the gain of a policy π is mathematically independent of the starting state s on the infinite, the total expected reward in a given time h is not, i.e. G(s, π) = 1 h Z h (s, π). Such differences are generally called bias [START_REF] Lewis | Bias optimality[END_REF][START_REF] Mahadevan | Learning representation and control in markov decision processes: New frontiers[END_REF][START_REF] Mahadevan | Average reward reinforcement learning: Foundations, algorithms, and empirical results[END_REF][START_REF] Tadepalli | Model-based average reward reinforcement learning[END_REF]. In fact, for a given MDP, there is a discount factor γ * from which the optimal policies do not change [START_REF] Blackwell | Discrete dynamic programming[END_REF]. Such common "unanimous" optimal policies correspond to a sensitive-optimality [START_REF] Mahadevan | Sensitive discount optimality: Unifying discounted and average reward reinforcement learning[END_REF][START_REF] Veinott | Discrete dynamic programming with sensitive discount optimality criteria[END_REF].

Dynamic Programming

Dynamic Programming (DP) refers to iterative optimization methods which can be used to efficiently compute optimal policies of Markovian Decision Processes (MDPs) when a model is given [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF][START_REF] Puterman | Dynamic programming[END_REF]. Value-Iteration (VI) [START_REF] Bellman | Dynamic Programming[END_REF] and Policy-Iteration (PI) [START_REF] Howard | Dynamic Programming and Markov Processes[END_REF] are the two fundamental and widely used DP algorithms for infinite time-horizon MDPs. It had been demonstrated that PI converges at least as quickly as VI [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF], and, in practice, PI has been shown to be most effective [START_REF] Littman | On the complexity of solving markov decision problems[END_REF].

There is a significant research effort for understanding the complexity of PI. The demonstration of its tight upper and lower bounds is still an open problem. Considering stochastic MDPs under discounted optimality, with a fixed discount rate 0 ≤ γ < 1, PI is proved to be strongly polynomial [START_REF] Ye | The simplex and policy-iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate[END_REF][START_REF] Hansen | Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor[END_REF][START_REF] Kalyanakrishnan | Batch-switching policy iteration[END_REF], i.e. the number of operations required to compute an optimal policy has an upper bound that is polynomial in the number of state-action pairs. Unfortunately, the convergence time increases with rate O( 1 1-γ log 1 1-γ ) [START_REF] Ye | The simplex and policy-iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate[END_REF][START_REF] Scherrer | Improved and generalized upper bounds on the complexity of policy iteration[END_REF]. It constitutes a major impediment for using high discount factors (γ → 1) in practice.

Typically, average optimization is a more difficult problem than discounted optimization [START_REF] Feinberg | Strong polynomiality of policy iterations for average-cost mdps modeling replacement and maintenance problems[END_REF]. PI can need an exponential number of iterations under averageoptimality for stochastic MDPs in the general case [START_REF] Fearnley | Exponential lower bounds for policy iteration[END_REF]. In contrast, a deterministic MDP under average-optimality can be solved in strongly polynomial-time, O(n 2 m) [START_REF] Papadimitriou | The complexity of markov decision processes[END_REF][START_REF] Hansen | Lower bounds for howard's algorithm for finding minimum mean-cost cycles[END_REF] as the well-known Minimum-Mean-Cost-Cycle problem. Experimental studies suggest that PI works very efficiently in this context [START_REF] Dasdan | Experimental analysis of the fastest optimum cycle ratio and mean algorithms[END_REF]. Recent advances in average optimisation have been proposed in [START_REF] Yang | Efficient average reward reinforcement learning using constant shifting values[END_REF].

Contribution: Average-Optimal Discount Factor

Given that low discount factor values can lead to "discount traps", and that high discount factor values imply exponentially high computational costs, suggestions about how to tune such factor are needed. To the best of our knowledge, there is no method published in the literature for doing so.

The main contribution of this paper consists then in proposing a first threshold for γ in order to ensure average optimality to discounted-optimal policies, considering the case of deterministic MDPs, and based on simple characteristics of the target process. It means deducing a value for γ ensuring that discountedoptimal policies will correspond to average-optimal policies for any D-MDP which fits the given characteristics.

Tuning the Discount Factor

Let a discount trap be characterized by the situation where, for a given MDP M, and for a given discount factor γ, there is a state s from where the gain of discounted-optimal policies is smaller than the gain of average-optimal policies. Formally, a discount trap exists if:

∃π ∈ Π, ∃s ∈ S V γ (s, π * ) ≥ V γ (s, π) G(s, π * ) < G(s, π) (5)
Let a family F of deterministic MDPs be the set of all possible D-MDPs presenting the following identical characteristics:

F =                    n = |S| number of states m = |A| number of actions r min ∈ R worst immediate reward value r max ∈ R best immediate reward value δ ∈ R +
the smallest non-zero immediate reward difference ∆ = r max -r min the range of the reward support In order to avoid discount traps, the discount factor must be tuned over a certain threshold. It is known that there exists an optimal discount factor γ * from where the optimal policies do not change [START_REF] Blackwell | Discrete dynamic programming[END_REF][START_REF] Hordijk | Blackwell optimality[END_REF]. Such value can be called the sensitive-optimal discount factor. γ is guaranteed to be over such threshold on the limit when it approaches 1, i.e. lim γ → 1 =⇒ γ > γ * . Another threshold, generally smaller than γ * , ensures that discounted-optimal policies are also average-optimal. We would like to call it the average-optimal discount factor, and denote it γ .

When looking for the average-optimal discount factor γ for a given family of D-MDPs F, we must look for the worst case within the family, i.e. the process M ∈ F which requires the highest value of γ to ensure average-optimality for discounted-optimal policies.

Deterministic Worst Case

Let an optimal loop L within a given D-MDP M be a cycle over a single state s, with some action a, presenting the maximum possible reward r max . Let an almost optimal cycle C be a cycle containing a subset of states, having a period |C| that can vary from 1 (a single state) to n -1, and presenting a sequence of |C| -1 maximum rewards r max followed by an almost maximum reward r max -δ. Let a maximally penalizing path W be a path having a length |W | that can vary from 1 (a single step) to n -1, and presenting the worst possible reward r min on every step. When such 3 structures appear connected within a D-MDP, we discover a graph topology similar to the cases presented in figure 3. However, a long and hardly penalizing transient path must be traversed in order to get from x to y. Such path counts the maximum possible distance between x and y, and is rewarded with the worst possible reward rmin at each step. The period of the almost optimal cycle is 1 (a loop), and the size of the maximally penalizing path is n -1. In (ii ), the almost optimal cycle constitutes the biggest possible cycle disjoint from the optimal loop in the MDP. Its period is n -1, which reduces the length of the path to 1. However, the gain difference is also smaller, G(x, b) = rmax > G(x, a) = rmax -δ n-1 . Such are the two intuitively possible worst situations within a given MDP family for defining an average-optimal discount factor.

The infinite discounted sum of rewards on the almost optimal cycle C, denoted V γ (C), can be calculated by the difference between the infinite discounted sum of r max and the infinite discounted sum of δ discounted by its position in the cycle:

V γ (C) = r max 1 -γ - γ |C|-1 δ 1 -γ |C| (6)
The infinite discounted sum of rewards on the path W followed by the loop L, denoted V γ (W ), corresponds to the finite discounted sum of r min on the max-imally penalizing path plus the infinite discounted sum of r max on the optimal loop discounted by the size of the path:

V γ (W ) = r min (1 -γ |W | ) 1 -γ + r max γ |W | 1 -γ (7)
When the discount factor is average-optimal, there is no discount trap.

Theorem 1. Let F be a family of D-MDPs, corresponding to the set of processes presenting identical characteristics n, m, r min , r max , δ, ∆. The set of D-MDPs within such family which requires the highest average-optimal discount factor is characterized by: (i) an almost optimal cycle C disjoint from (ii) an optimal loop L, both separated by a unique (iii) maximally penalizing path W .

The theorem 1 necessarily holds because:

1. Shortening W increases V γ (W ) without changing V γ (C). 2. Increasing r min increases V γ (W ) without changing V γ (C). 3. Increasing δ decreases V γ (C) without changing V γ (W ). 4. Decreasing r max makes V γ (C) decrease faster than V γ (W ).
Developing the theorem 1 results in n possible worst cases, from where the two extremes are illustrated in figure 3.

Deterministic average-optimal discount factor

In order to avoid a "discount trap", the utility of staying in an almost optimal cycle must be worse than the utility of traveling across all the states of the maximally penalizing path to the optimal loop. In the precedent section, two candidate worst cases have been presented. In this section the formula for an optimal discount factor is deduced for both examples. Such procedure allows to confirm what is effectively the worst case. An optimal discount factor for such worst case must necessarily be an optimal discount factor for any other case within the same MDP family. Such formulas can be deduced by simply developing the statement:

γ > γ =⇒ V γ (W ) > V γ (C) (8)
Firstly, we consider the case presented in figure 3 (i), which contains a long penalizing path. The value of the average-optimal discount factor γ from which the discounted-optimal policies are also average-optimal for such D-MDP is:

V γ (x, b) > V γ (x, a) =⇒ n-2 i=0 γ i r min + ∞ i=n-1 γ i r max > ∞ i=0 γ i (r max -δ) =⇒ (1 -γ n-1 )r min 1 -γ + γ n-1 r max 1 -γ > r max -δ 1 -γ =⇒ r min -γ n-1 r min + γ n-1 r max > r max -δ =⇒ γ n-1 r max -γ n-1 r min > r max -r min -δ =⇒ γ n-1 ∆ > ∆ -δ =⇒ γ > n-1 1 - δ ∆
Then we consider the case presented in figure 3 (ii), which contains a short penalizing path. The value of the average-optimal discount factor γ from which the discounted-optimal policies are also average-optimal for such D-MDP is:

V γ (x, b) > V γ (x, a) =⇒ r min + γV γ (y) > V γ (y) - γ n-2 δ 1 -γ n-1 =⇒ V γ (y) -r max + r min > V γ (y) - γ n-2 δ 1 -γ n-1 =⇒ -r max + r min > - γ n-2 δ 1 -γ n-1 =⇒ γ n-2 δ 1 -γ n-1 > ∆ =⇒ γ n-2 δ > ∆(1 -γ n-1 ) =⇒ γ n-2 > -δ ± √ δ 2 + 4∆ 2 2∆ =⇒ γ > n-2 -δ + √ δ 2 + 4∆ 2 2∆
Effectively, the second case is the worst case, which can be algebraically confirmed:

n-2 -δ + √ δ 2 + 4∆ 2 2∆ > n-1 1 - δ ∆ =⇒ -δ + √ δ 2 + 4∆ 2 2∆ 2 > 1 - δ ∆ =⇒ 2δ 2 -2δ √ δ 2 + 4∆ 2 + 4∆ 2 4∆ 2 > 1 - δ ∆ =⇒ 2δ 2 -2δ √ δ 2 + 4∆ 2 4∆ 2 > - δ ∆ =⇒ 2δ 2 -2δ δ 2 + 4∆ 2 > -4∆δ =⇒ δ -δ 2 + 4∆ 2 > -2∆ =⇒ δ -δ 2 + 4k 2 δ 2 > -2kδ =⇒ k > 0 =⇒ ∆ δ > 0
Hence, the formula for calculating the optimal discount factor is:

γ = n-2 -δ + √ δ 2 + 4∆ 2 2∆ ( 9

Discount Trap Frequency

The choice of the optimality criterion has an impact on the calculated optimal policies. When using the discounted framework, if the discount factor γ is not sufficiently high, discounted-optimal policies could not correspond to gainoptimal policies, and it means a worst performance in terms of total rewards in the long run. In order to be able to measure the impact of such choice, we made a series of experiments, verifying how often a "discount trap" is detected.

Each experiment consists in generating 10000 random D-MDPs for a given setting (or family) i, varying the number of states n = i and the reward granularity δ = 1 /i. It means that the MDP size and the reward granularity are both gradually incremented. The number of actions is fixed to m = 2, as well as the minimum reward r min = 0 and the maximum reward r max = 1. We make the parameter i vary from 2 to 50. Fig. 5. The frequency of discount traps, when the discounted-optimal policy is not gain-optimal, considering γ = 0.9 and γ = 0.99. The parameter i indicates the number of states n = i and the reward granularity δ = 1 /i.

The results presented in figure 5 confirms that, for standard "naive" values of γ, like 0.9 and 0.99, the frequency of "discount traps" is not negligible, even for such small MDPs.

Conclusion

Using a set of experiences with randomly generated MDPs, we demonstrated that the occurrence of discount traps, inherent to all mechanisms that calculate utility functions using a discount factor, can cause sub-optimal behaviors on several recurrent MDPs in terms of total (undiscounted) rewards, and can be observed more often than usually suspected. In our experiments, we show that the use of "naive" but classical values for γ can result in discounted-optimal policies which are not average-optimal in 40% of the simulations when γ = 0.9, and almost 4% when γ = 0.99, which is far from being negligible.

In this paper, a formula for calculating an average-optimal discount factor is deduced, given the target family of deterministic MDPs characterized by n (the number of states), r max and r min (the reward bounds), and δ (the "reward granularity", equivalent to the smallest difference between any two rewards into the reward function). It represents an upper bound that could be improved by taking other characteristics into account. This paper was limited to deterministic MDPs. The next step of the work is understanding how such γ threshold can be defined on the stochastic case.

Fig. 1 .

 1 Fig. 1. In (i ), there is a unique decision to be taken, on the middle state y. The gain of choosing the action a is G(y, a) = 0. It corresponds to the reward on the loop of the recurrent left state x. The gain of choosing the action b is G(y, b) = +1. It corresponds to the reward on the loop of the recurrent right state z. Because G(y, b) > G(y, a), the action b constitutes the average-optimal policy. The action a earns a unique and immediate positive reward of 50 in time t = 1. At the same time, the action b loses 50, but then earns an additional reward of +1 per each subsequent time-step. In time t = 101, the policy b "reaches" the policy a, both having accumulated the same total rewards Z101(y, a) = Z101(y, b) = +50. Then, after 101 execution steps, b becomes better thana up to the infinity. So, considering an unbounded time-horizon (where the stopping time h of the process is likely to be greater than 100), the policy b would be preferred. However, given that V0.99(y, a) = +50 and V0.99(y, b) = +49, the discounted-optimal policy for any discount factor γ ≤ 0.99 is a. In (ii ), the average of the policy starting on the middle state y and choosing the action a is G(y, a) = +0.45 (the average per step on the cycle {y, z}, on the right). The action b presents G(y, b) = +0.5 (the average per step on the cycle {y, x}, on the left) and is gain-optimal. However, given that V0.9(y, a) = V0.9(y, b) ≈ 4.737, the discounted-optimal policy for any discount factor γ < 0.9 is a. In fact b becomes definitely better (i.e. get better total rewards) than a after 20 execution steps and up to the infinity.

Fig. 2 .

 2 Fig. 2. In (i ), the gain of both possible policies is equivalent, G(x, a) = G(x, b) = G(y) = 0, but their initial steps (in t = 1) are not equivalently rewarded, R1(x, a) = -50 and R1(x, b) = +50. In this case, both policies a and b are average-optimal, but b offers a better transient reward and would be preferred over a. In fact, the total rewards accumulated by the policy b are greater than the rewards accumulated by a after the first step, i.e. Z h (x, b) > Z h (x, a), ∀h ∈ Z | h ≥ 1. In (ii ), the gain of both possible policies is equivalent, G(x, a) = G(x, b) = 0, but the policy b presents a bigger total reward compared to a every time when t is odd. Then Z h (x, b) ≥ Z h (x, a), ∀h ∈ Z + , and for such reason, b would be the preferred policy.

Fig. 3 .

 3 Fig. 3. In (i ), the recurrent optimal loop on the state y can be reached by choosing the action b on state x. It offers a better gain than the almost optimal cycle reached with the action a in x, i.e. G(x, b) = G(y) = R(y) = rmax > G(x, a) = R(x, a) = rmax -δ.However, a long and hardly penalizing transient path must be traversed in order to get from x to y. Such path counts the maximum possible distance between x and y, and is rewarded with the worst possible reward rmin at each step. The period of the almost optimal cycle is 1 (a loop), and the size of the maximally penalizing path is n -1. In (ii ), the almost optimal cycle constitutes the biggest possible cycle disjoint from the optimal loop in the MDP. Its period is n -1, which reduces the length of the path to 1. However, the gain difference is also smaller, G(x, b) = rmax > G(x, a) = rmax -δ n-1 . Such are the two intuitively possible worst situations within a given MDP family for defining an average-optimal discount factor.

)Figure 4

 4 Figure 4 plots the function γ for two different settings. Parameter i indicates the number of states, n = i, and the two different series represent reward granularity δ = 1/i and δ = 1 (binary rewards).

Fig. 4 .

 4 Fig. 4. The average-optimal discount factor for n = i and reward support in [0, 1]. The two different series represent reward granularity δ = 1/i and δ = 1 (Bernoulli).

  {s 1 , s 2 , ..., s n } is the finite set of states A = {a 1 , a 2 , ..., a m } is the finite set of actions |S| is the number of states, and m = |A| is the number of actions.

	T = Pr(s |s, a)	is the transition function
	R = Pr(r|s, a, s )	is the reward function
	where n =


Deepa Senapathi
email: g.d.senapathi@reading.ac.uk

Jochen Fründ

Matthias Albrecht

Michael P D Garratt

David Kleijn

Brian J Pickles

Simon G Potts

Jiandong An

Georg K S Andersson

Svenja Bänsch

Parthiba Basu

Faye Benjamin

Antonio Diego

M Bezerra

Ritam Bhattacharya

Jacobus C Biesmeijer

Brett Blaauw

Eleanor J Blitzer

Claire A Brittain

Luísa G Carvalheiro

Daniel P Cariveau

Pushan Chakraborty

Arnob Chatterjee

Soumik Chatterjee

Sarah Cusser

Bryan N Danforth

Erika Degani

Breno M Freitas

Lucas A Garibaldi

Benoit Geslin

G Arjen De Groot

Tina Harrison

Brad Howlett

Rufus Isaacs

Shalene Jha

Kristian Björn

Klatt

Kristin Krewenka

Samuel Leigh

Sandra A M Lindström

Yael Mandelik

Megan Mckerchar

Mia Park

Gideon Pisanty

Romina Rader

Menno Reemer

Maj Rundlöf

Barbara Smith

Henrik G Smith

Patrícia Nunes

Ingolf Steffan-Dewenter

Teja Tscharntke

Sean Webber

Duncan B Westbury

Catrin Westphal

Jennifer B Wickens

Victoria J Wickens

Rachael Winfree

Hong Zhang

Alexandra-Maria Klein

Wild insect diversity increases inter-annual stability in global crop pollinator communities

Keywords: Ecology Subject Areas: ecology, environmental science pollinators, temporal stability, crops, insect diversity, inter-annual variation, dominant species

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

While an increasing number of studies indicate that the range, diversity and abundance of many wild pollinators has declined, the global area of pollinator-dependent crops has significantly increased over the last few decades. Crop pollination studies to date have mainly focused on either identifying different guilds pollinating various crops, or on factors driving spatial changes and turnover observed in these communities. The mechanisms driving temporal stability for ecosystem functioning and services, however, remain poorly understood. Our study quantifies temporal variability observed in crop pollinators in 21 different crops across multiple years at a global scale. Using data from 43 studies from six continents, we show that (i) higher pollinator diversity confers greater inter-annual stability in pollinator communities, (ii) temporal variation observed in pollinator abundance is primarily driven by the three-most dominant species, and (iii) crops in tropical regions demonstrate higher inter-annual variability in pollinator species richness than crops in temperate regions. We highlight the importance of recognizing wild pollinator diversity in agricultural landscapes to stabilize pollinator persistence across years to protect both biodiversity and crop pollination services. Shortterm agricultural management practices aimed at dominant species for stabilizing pollination services need to be considered alongside longer term conservation goals focussed on maintaining and facilitating biodiversity to confer ecological stability.

Introduction

The crucial role played by pollinators in the reproduction of flowering plants is well-established [START_REF]The assessment report on pollinators, pollination and food production[END_REF]. Biotic pollination is important for the reproduction of at least 78% of wild plants [START_REF] Ollerton | How many flowering plants are pollinated by animals?[END_REF] and insects contribute to the pollination of 75% of leading global crops [START_REF] Klein | Importance of pollinators in changing landscapes for world crops[END_REF]. Crop systems have also recently become more pollinator dependent because of a disproportionate increase in the area cultivated with entomophilous flowering crops [START_REF] Aizen | Long-term global trends in crop yield and production reveal no current pollination shortage but increasing pollinator dependency[END_REF]. Given the documented declines of wild insect pollinators in some NW European and North American landscapes where these crops are grown [START_REF]The assessment report on pollinators, pollination and food production[END_REF][START_REF] Carvalheiro | Species richness declines and biotic homogenisation have slowed down for NW-European pollinators and plants[END_REF][START_REF] Powney | Widespread losses of pollinating insects in Britain[END_REF], understanding temporal variation in assemblages is important to maintain ongoing food security.

Higher pollinator diversity can lead to increases in fruit and seed set in plants and is an important predictor of crop yields worldwide [START_REF] Garibaldi | Mutually beneficial pollinator diversity and crop yield outcomes in small and large farms[END_REF][START_REF] Dainese | A global synthesis reveals biodiversity-mediated benefits for crop production[END_REF]. Conversely, pollinator communities with lower diversity and fewer species have been linked to lower fruit set or seed production, and decreased temporal and spatial stability within seasons [START_REF] Bommarco | Drastic historic shifts in bumble-bee community composition in Sweden[END_REF][START_REF] Klein | Advances in pollination ecology from tropical plantation crops[END_REF][START_REF] Albrecht | Diverse pollinator communities enhance plant reproductive success[END_REF], and may be one reason for lower inter-annual stability of yields in pollinator-dependent crops [START_REF]The assessment report on pollinators, pollination and food production[END_REF]. While biologically diverse communities can enhance ecological resilience [START_REF] Winfree | Are ecosystem services stabilized by differences among species? A test using crop pollination[END_REF][START_REF] Isbell | Biodiversity increases the resistance of ecosystem productivity to climate extremes[END_REF], and diversity is a key factor affecting system stability [START_REF] Ives | Stability and diversity of ecosystems[END_REF], most ecological communities are generally made up of a few species that are numerically abundant and may have many rarer species with very few individuals [START_REF] Mcgill | Species abundance distributions: moving beyond single prediction theories to integration within an ecological framework[END_REF].

Evidence suggests that numerically dominant species may provide most of the ecosystem services [START_REF] Vázquez | Interaction frequency as a surrogate for the total effect of animal mutualists on plants[END_REF], with Kleijn et al. [START_REF] Kleijn | Delivery of crop pollination services is an insufficient argument for wild pollinator conservation[END_REF] finding that approximately 80% of biotic crop pollination in Europe and North America are fulfilled by approximately 2% of the pollinator species in a community. In addition, the scale of spatial assessment, is also important, with Winfree et al. [START_REF] Winfree | Species turnover promotes the importance of bee diversity for crop pollination at regional scales[END_REF] showing that the number of wild bee species required for reaching a minimum pollination service threshold rapidly increased with spatial scale, indicating that maintaining pollination services across large areas requires many species, including rare ones. Providing stable pollination services for crop systems across several years is needed for sustainable crop production, but the mechanisms driving temporal stability for ecosystem functioning and services still remains an important but poorly understood phenomenon [START_REF] Grman | Mechanisms contributing to stability in ecosystem function depend on the environmental context[END_REF].

Previous studies aimed at disentangling the mechanisms of temporal stability highlight the role of both diversity and royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 20210212 dominance. Lehman & Tilman [START_REF] Lehman | Biodiversity, stability, and productivity in competitive communities[END_REF] showed that greater diversity increases the temporal stability of the entire community but decreases the temporal stability of individual populations. The counterview is that dominant species, rather than diversity itself, might regulate temporal stability-for e.g. Sasaki & Lauenroth [START_REF] Sasaki | Dominant species, rather than diversity, regulates temporal stability of plant communities[END_REF] found that temporal stability in a shortgrass steppe plant community was controlled by dominant species rather than by community diversity. In addition, species asynchrony has also been considered an important mechanism of diversity-stability relationships and may lead to higher stability on the community level even when the stability of individual populations decreases with diversity. However, the majority of such studies have used long-term observations of the same plant communities over time (for e.g. [START_REF] Allan | More diverse plant communities have higher functioning over time due to turnover in complementary dominant species[END_REF]), while such equivalent information on pollinators in general or even crop pollinator communities in particular are lacking.

A few multi-year, single-crop studies exist showing that pollinator communities can vary over longer time periods [START_REF] Bommarco | Drastic historic shifts in bumble-bee community composition in Sweden[END_REF][START_REF] Bartomeus | Biodiversity ensures plantpollinator phenological synchrony against climate change[END_REF][START_REF] Rader | Spatial and temporal variation in pollinator effectiveness: do unmanaged insects provide consistent pollination services to mass flowering crops?[END_REF]. What implications this may have for stability remains unknown due to a lack of synthesized knowledge on temporal dynamics of crop pollinator communities and underlying driving factors. For example, evidence for the contribution of managed pollinators to the temporal stability of the overall crop pollinator community is largely lacking. Such knowledge gaps, if addressed, could lead to a better understanding of the stability and long-term resilience of global crop systems that rely on insect pollination. Temporal stability of ecosystem functioning increases predictability and reliability of ecosystem services and understanding the drivers of stability across spatial scales is important for land management and policy decisions [START_REF] Wilcox | Asynchrony among local communities stabilises ecosystem function of metacommunities[END_REF].

Here, we synthesize data from multiple studies to examine factors that affect the temporal stability of crop pollinator communities, which in turn has implications for the stability of pollination services provided. Using data from 43 studies across six continents, we characterize the annual variation observed in crop pollinators and explore the following questions: (1) is temporal stability of crop pollinator communities primarily driven by the diversity of pollinator communities or by inter-annual stability of dominant species? (2) What crop characteristics if any (e.g. annual/perennial, flower morphology, mass flowering/non-mass flowering crops) influence inter-annual stability of crop pollinator communities? (3) Does inter-annual variation observed in pollinator communities differ between climatic regions (i.e. tropics and temperate study areas)?

Material and methods (a) Data collection

We collated datasets from 12 countries across six continents on 21 crop species to examine the variations in richness and abundance of insect pollinators in crop systems. The criteria for inclusion in the analyses were as follows: data on crop pollinator species/ morpho-species were required (a) from the same crop for two or more years, (b) with consistent sampling methods used across years, (c) focused on flower visitation data and (d) in the case of annual crops, field sites were required to be within 500 m of the crop field used for recording in previous years to make sure they could be visited by the same pollinator communities. Our final dataset included information on 375 crop fields (hereafter referred to as sites) from 43 studies (see electronic supplementary material, table S1 for additional information).

Data were standardized to ensure that species names and taxonomic groups were consistent across all studies prior to analyses.

Each dataset was classified on the basis of climatic region (tropical/temperate), crop type (annual/perennial), plant family and flower type (open/not open)-based on nectar accessibility criteria in Garibaldi et al. [START_REF] Garibaldi | Trait matching of flower visitors and crops predicts fruit set better than trait diversity[END_REF]. In addition, we distinguished crop species that exhibit mass flowering (MFC)-i.e. short duration intense bloom with high floral density, from those with extended flowering periods with lower density and more sparse blooms. Some crops are clearly defined as mass flowering in the literature [START_REF] Westphal | Mass flowering crops enhance pollinator densities at a landscape scale[END_REF][START_REF] Klein | Wild pollination services to California almond rely on semi-natural habitat[END_REF][START_REF] Hanley | Increased bumblebee abundance along the margins of a mass flowering crop: evidence for pollinator spill-over[END_REF][START_REF] Rader | Alternative pollinator taxa are equally efficient but not as effective as the honeybee in a mass flowering crop[END_REF][START_REF] Kallioniemi | Local resources, linear elements and mass-flowering crops determine bumblebee occurrences in moderately intensified farmlands[END_REF], while others remain ambiguous. To overcome this uncertainty, we requested the original authors to indicate if their crop was considered as MFC in their study and that is reflected in the dataset and subsequent analyses (see electronic supplementary material, table S2). Almonds (Prunus dulcis), apples (Malus domestica), highbush blueberry (Vaccinium corymbosum), cranberry (Vaccinium angustifolium), red clover (Trifolium pratense), field beans (Vicia faba), oilseed rape or canola (Brassica napus), pears (Pyrus communis), pak choi (Brassica rapa subsp. chinensis) and turnips (Brassica rapa subsp. rapa) were classified as MFC. Non-mass flowering crops in our analyses include avocado (Persea americana), bitter gourd (Momordica charantia) and brinjal (Solanum melongena)-also known as eggplant or aubergine, cashew (Anacardium occidentale), cotton (Gossypium hirsutum), kiwifruit (Actinidia deliciosa), mango (Mangifera indica), mustard (Brassica napus), onion (Allium cepa), pumpkin (Cucurbita pepo), ridge gourd (Luffa acutangula), spine gourd (Momordica dioica), strawberry (Fragaria x ananassa) and watermelon (Citrullus lanatus). Note: Brassica napus includes oilseed rape (OSR)-a MFC in Europe and North America but a different subspecies considered as a type of mustard in India which is not grown as MFC.

(b) Characterizing year to year variation in crop pollinators

Initially, crop pollinators recorded were classified into taxonomic groups which included the following: (i) honeybees (including Apis mellifera, Apis cerana, Apis dorsata and other recorded as Apis sp.); (ii) bumblebees (all Bombus sp.); (iii) other bees (wild solitary and social bees including stingless bees but excluding bumblebees and honeybees); (iv) butterflies and moths; (v) hoverflies; (vi) other Diptera (flies excluding hoverflies); (vii) wasps; and (viii) beetles. The single most dominant taxonomic group and species were identified at all study sites (figure 1) based on recorded abundance and a binary (change/no change) analysis was used to determine whether the most dominant group and species remained constant across all years of sampling.

To characterize between year variation in crop pollinators, (i) a coefficient of variation (CV) of total pollinator abundance and (ii) a CV of pollinator species richness was calculated for each site across all years of the study. The CV (which incorporates a bias correction) is defined as the ratio of the sample standard deviation 's' to the sample mean x ¯-i.e. CV = s/x ¯and shows the extent of variability in relation to the mean of the population. These measures were calculated using species level data for each study site and the mean and standard deviation of these two measures were also calculated for each individual study (electronic supplementary material, figure S1). In addition, the CV of abundance and CV of richness were calculated for each site for every pairwise year comparisons (i.e. Y1&Y2; Y2&Y3; Y3&Y4 etc.) to account for studies having a different number of years of data.

Other calculated indices included (a) CV of honeybee abundance, (b) CV of the proportion of honeybees, (c) CV of the most dominant pollinator species across all years and (d) the mean Shannon index of pollinator diversity (H') were calculated across years. The Shannon diversity index was chosen as it accounts for the evenness of the species present, thus reflecting effective diversity, and is less sensitive to sampling effects than royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 20210212 species richness [START_REF] Krebs | Ecological methodology[END_REF][START_REF] Jost | Entropy and diversity[END_REF]. Since a subset of studies (28 out of 43) also recorded temperature at the study sites, a standard deviation (s.d.) of temperature was also calculated as a measure of variation in local climatic condition across years.

(c) Factors influencing the observed variation

In order to examine the potential drivers of inter-annual variation in crop pollinator communities, linear mixed-effects models were constructed using (i) CV of total pollinator abundance and (ii) CV of pollinator species richness. These two indices were calculated across all years of each study and for every pairwise year in each study to account for studies with different numbers of years of observations and ensure checks for sensitivity and robustness. The models included descriptors of pollinator communities such as Shannon diversity (H') of pollinators, CV of dominant species, and change in dominant pollinator species between years (Y/N) as fixed effects. External predictors including climatic region (tropical/temperate), crop type (annual/perennial), crop family, flower type, MFC (Y/N) and SD of site temperature were also included as other explanatory variables. Study ID was included in all models as a random effect and for models where the response variables were calculated for every two years of the study, site ID nested within the study ID were used as random effects (and identified as relevant indicated by positive variance estimates).

The calculated indices were tested for collinearity and correlated variables were not used within the same models (see correlation matrix in electronic supplementary material, table S3). Similarly, categorical predictors which exhibited significant collinearity were not used as variables within the same models. A series of candidate models were constructed for each response variable. Each candidate model was 'dredged' to obtain a series of plausible intermediate models. Intermediate models with Δ AICc value less than 7 of the model with lowest AICc were averaged (using the default zero average method) to obtain the final outputs. Residual plots for final models were used to check for heteroscedasticity. Models were fitted using maximum likelihood (ML) and analysed using the 'lme4' [START_REF] Bates | Fitting linear mixed-effects models using lme4[END_REF] and 'MuMIn' [START_REF] Burnham | Model selection and multimodel inference: a practical informationtheoretic approach[END_REF] packages. All statistical analyses were carried out in R v. 4.0.3 statistical software [START_REF]R: a language and environment for statistical computing[END_REF].

(d) Influence of dominant pollinator species

To test whether variation in total crop pollinator abundance was driven primarily by variation of the most dominant pollinator species, a paired t-test was used to determine whether the CV of total pollinator abundance was significantly different from the CV of the abundance of the single most dominant pollinator species. The same test was repeated using the combined CV of the abundance of the two-most, three-most and four-most dominant species to determine how many dominant pollinator species were required to influence the overall variation in total abundance observed. While abundance of dominant species will always be a subset of the total pollinator abundance, these tests were conducted to determine how many dominant species it took to match the change in overall pollinator abundance across years and determine the minimum number of species that drive the temporal variation in overall pollinator abundance. A Welch two-sample t-test was used to determine if inter-annual variation in pollinator abundance differed between sites dominated by honeybees versus other pollinator species. Sites where there was mixed dominance between honeybees and other pollinators were excluded from this analysis.

(e) Species dominance and stability effect

To further understand mechanisms of stability and particularly the relationship of the dominant species to the whole community, we calculated the correlation between the changes in abundance of the dominant species and the changes in abundance of the rest of the community. Negative correlation (negative covariance) suggests asynchrony, which is considered a key driver of stability and the main mechanism of diversity-stability relationships [START_REF] Blüthgen | Land use imperils plant and animal community stability through changes in asynchrony rather than diversity[END_REF]. Negative correlations could indicate density compensation or different responses to environmental variation [START_REF] Winfree | Are ecosystem services stabilized by differences among species? A test using crop pollination[END_REF]. In general, the higher the asynchrony (i.e. more negative the correlation), 2%

2% ( 1 royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 20210212

the stronger the contribution to stability. With our short time series, many correlations are -1 or +1, without an even continuous gradient in the degree of asynchrony. Therefore, we separated sites by asynchronous (r ≤ 0) or synchronous (r > 0) fluctuations of the dominant pollinator species in comparison to the rest of the pollinator community and, for each group separately, repeated the paired t-test of the CV of the dominant species versus the whole community.

Results

(a) Characterizing year to year variation in crop pollinators

Honeybees were dominant across 41.9% of studies with other wild bees (32.6%) representing the next most dominant group (figure 1a). At the site level, other wild bees were the most dominant group at 41.6%, with honeybees (38.0%) the second most dominant (figure 1b). The dominant taxonomic group did not change between years in most of the studies or the sites, whereas the dominant species varied between years in approximately half the studies and half the sites (table 1). The mean (±s.d.) of the CV of total pollinator abundance, and the CV of total pollinator richness for all sites within each study are provided in electronic supplementary material, figure S1.

(b) Factors influencing the observed variation

The relative variability of total pollinator abundance across all years was significantly related to the Shannon diversity (table 2, estimate = -0.16, z = 3.96, p < 0.0001, figure 2a). It was also significant whether the most dominant species varied between years: systems where dominant species stayed the same showed less inter-annual variation in overall pollinator abundance (table 2, estimate = -0.08, z = 2.23, p = 0.03, figure 2b). However, in models using CV of abundance for every two years, the variability in dominant species showed no significant relationship (table 2, estimate = -0.05, z = 1.42, p = 0.16) Having a diverse pollinator community also reduced the inter-annual variation in pollinator species richness (table 2, estimate = -0.16, z = 5.61, p < 0.0001, figure 3a), and this was true for indices calculated across all years of the studies as well as every two years of the studies (table 2). The relative change in species richness between years was related to the change in the abundance of the most dominant species, with study systems showing larger changes in species richness if there was increased inter-annual variation in dominant species abundance across all years (table 2, estimate = 0.09, z = 3.31, p < 0.001, figure 3b). This was also significant in models accounting for change in species richness for every two years (table 2, estimate = 0.12, z = 3.77, p < 0.001). However, any change in dominant species across years showed no significant relationship with the relative change in species richness. The change in pollinator species richness also varied between climatic regions with crops grown in temperate systems showing less inter-annual variability in pollinator species richness than crops in tropical areas (table 2, figure 3c).

Other factors tested including crop family, flower type, annual versus perennial crop type, mass flowering or site temperature did not show any significant relationship with variability observed in the abundance or richness of species across years.

(c) Influence of dominant pollinator species

It took the pooled abundance of the three-most dominant pollinator species to match the relative variability of total pollinator abundance (respective mean CVs: 0.58 versus 0.55, t = 1.09, d.f. = 362, p = 0.2, difference in means = 0.03). The relative variability of total pollinator abundance at the site level was found to be significantly lower than that of the single (t = 9.56, d.f. = 362, p-value < 0.001, difference in means = 0.17) and top two-most dominant species (t = 6.34, d.f. = 362, p-value < 0.001, difference in means = 0.07). Sites where honeybees were dominant species (mean CV = 0.46) were found to have significantly lower variability (t = 3.25, d.f. = 295.26, p = 0.001) than sites where other bees were dominant species (mean CV = 0.60).

Where the dominant species changed asynchronously to the rest of the community, the difference between the CV of the dominant species and CV of total abundance was strong, with less than half the variability in the whole community than in the dominant species (t = -11.02, d.f. = 125, p-value less than 0.0001, mean of total 0.31, mean of single most dominant species 0.67; difference in means = 0.36). By contrast, CV of total abundance was only slightly lower than the CV of the dominant species where the dominant species changed synchronously with the rest of the community (t = -3.48, d.f. = 219, p-value = <0.001, mean of total 0.65, mean of single most dominant species 0.71, difference in means = 0.06, figure 4). In simple terms, the stability of the whole pollinator community only increased to a considerable degree when other species buffered changes by asynchronous fluctuations.

Discussion

This study is the first to use a global dataset to explore interannual variation in crop pollinator communities and has revealed several important features of community stability. Our findings indicate that: (i) crop pollinator communities with higher pollinator diversity are more stable between years, and (ii) the variation observed in pollinator communities is driven by dominant species changes across years. The importance of other species in addition to the stability of the dominant species was in line with mechanisms of royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 20210212 diversity-stability relationships: while the stability of the dominant species was similar to the total community where the dominant species fluctuated synchronously with the rest of the community, community abundance was much more stable than abundance of the dominant species where these fluctuations were asynchronous. Neither the variation in abundance nor the variation in species richness was significantly affected by any crop characteristics.

Our results show that sites with higher pollinator species diversity experience less variation in total crop pollinator abundance and less change in pollinator species richness between years. These results concur with studies from royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 20210212

individual cropping systems which have shown that diversity provides greater spatial and temporal stability and resilience [START_REF] Winfree | Are ecosystem services stabilized by differences among species? A test using crop pollination[END_REF][START_REF] Bartomeus | Biodiversity ensures plantpollinator phenological synchrony against climate change[END_REF], and support the theory that ecological systems with higher species diversity are better buffered against interannual variation in species abundance, and possibly more resilient to changes in the longer term [START_REF] Ives | Stability and diversity of ecosystems[END_REF]. This has implications beyond ecological resilience, as stable pollination services could help mitigate risks and uncertainties for farmers growing pollinator-dependent crops, providing economic resilience.

In addition to diversity, our results demonstrate that dominant species play a significant role in the inter-annual stability of crop pollinator communities. Honeybees were found to be the single most dominant species in 18 out of 43 datasets and in 140 out of 375 sites which concurs with the findings of Kleijn et al. [START_REF] Kleijn | Delivery of crop pollination services is an insufficient argument for wild pollinator conservation[END_REF]. Sites where honeybees were the dominant species across all years also showed greater inter-annual stability in abundance when compared to sites dominated by other species. Unlike wild pollinators, managed pollinators are often placed near crops and due to hive management practices may show less variability in abundance between years. Managed pollinators are considered to supplement rather than substitute pollination by wild insects in most crop pollination systems [START_REF] Garibaldi | Wild pollinators enhance fruit set of crops regardless of honey bee abundance[END_REF], but there is experimental evidence to suggest that managed bees in high numbers could displace wild pollinators from crop fields [START_REF] Lindström | Experimental evidence that honeybees depress wild insect densities in a flowering crop[END_REF]. Our study systems from Argentina, for instance, were entirely reliant on managed Apis mellifera and no other species were recorded. The management of bees could therefore be an important contributor to the inter-annual variability observed in the crop pollinator community depending on the placement of hives, stocking densities and how much these vary from one year to the next. Careful targeting of managed pollinators could be used to increase the stability of pollination [START_REF] Garratt | Integrated crop pollination to buffer spatial and temporal variability in pollinator activity[END_REF][START_REF] Isaacs | Integrated crop pollination: combining strategies to ensure stable and sustainable yields of pollination-dependent crops[END_REF][START_REF] Rollin | Impacts of honeybee density on crop yield: a meta-analysis[END_REF], particularly in those crops for which inter-annual variation is high due to fluctuations in populations of the dominant wild pollinators.

While we can say with a high level of certainty that most honeybees recorded in the USA and European studies were from managed hives, it is difficult to distinguish between managed and wild honeybees in other studies. For example, in China and India, while almost all Apis mellifera were managed and all Apis dorsata wild, it is difficult to distinguish between wild and managed Apis cerana with any degree of certainty. In addition, certain areas-particularly in Western Europeuse Bombus terrestris as a managed pollinator, and managed and wild individuals of this species are indistinguishable from each other. Therefore, we cannot draw specific conclusions on the effect of managed pollinators on the changes in richness and turnover of wild pollinator communities but raise this as a possible question to be explored in future studies.

From our results, we also infer that a significant part of the year to year variation in crop pollinator abundance is driven by as few as three of the most dominant species within each system (see list of dominant species by study in electronic supplementary material, table S4). This is consistent with the findings of Kleijn et al. [START_REF] Kleijn | Delivery of crop pollination services is an insufficient argument for wild pollinator conservation[END_REF] who showed that the three-most dominant pollinator species account for twothirds of flower visits recorded. Even if only a few species are quantitatively important in crop pollination systems, enhancing stability by managing for diversity effects delivered through asynchrony among species could be really effective as our results above have indicated. It is worth noting that while the delivery of crop pollination services may be predominantly driven by a few key functional pollinator species [START_REF] Kleijn | Delivery of crop pollination services is an insufficient argument for wild pollinator conservation[END_REF], depending on the context, the diversity and abundance of other pollinators may complement or largely replace the functional role of dominant species [START_REF] Fijen | Insect pollination is at least as important for marketable crop yield as plant quality in a seed crop[END_REF].

The Winfree et al. [START_REF] Winfree | Species turnover promotes the importance of bee diversity for crop pollination at regional scales[END_REF] study-which explored functional consequences of spatial turnover in crop pollinator communities-indicated that more species would be required to fulfil the minimum pollination service threshold if dominance effects were to be removed or lost, but that is based on the assumption that another species would be unable to take over the dominant role through increased abundance. This raises questions of which systems would remain resilient in the event these specific dominant species are lost due to future environmental conditions. For example, field beans flower morphology excludes small solitary bees and depends predominantly on effective flower visits from long-tongued bumblebees [START_REF] Garratt | The identity of crop pollinators helps target conservation for improved ecosystem services[END_REF][START_REF] Marzinzig | Bee pollinators of faba bean (Vicia faba L.) differ in their foraging behaviour and pollination efficiency[END_REF], may be less resilient to the loss of dominant pollinators when compared to crops like oilseed rape dependent on a diverse suite of pollinators [START_REF] Garratt | The identity of crop pollinators helps target conservation for improved ecosystem services[END_REF].

While no effect of the climatic region was observed on the inter-annual change in pollinator abundance, there was less variation in pollinator species richness in temperate crops than in crops grown in the tropics. Studies from temperate regions (n = 29) showed a higher average Shannon diversity (H' = 1.21) than studies from the tropics (n = 13, H' = 1.19) but the difference was not statistically significant (electronic supplementary material, figure S2, t = 0.26, d.f. = 356, p = 0.74), and it is difficult to disentangle whether this result may be due to differences in sampling effort. The difference between the temperate and tropical studies could not be attributed to contrasting temperature regimes in the different climatic regions as we did not detect a significant effect of temperature on inter-annual stability of crop pollinators in any of the models. Pollinator populations are known to be sensitive to weather conditions [START_REF] Kallioniemi | Local resources, linear elements and mass-flowering crops determine bumblebee occurrences in moderately intensified farmlands[END_REF] with temperature influencing pollinator phenology [START_REF] Duchenne | Phenological shifts alter the seasonal structure of pollinator assemblages in Europe[END_REF] as well as plant-pollinator interactions [START_REF] Classen | Specialization of plant-pollinator interactions increases with temperature at Mt Kilimanjaro[END_REF]. Our analyses indicated that the crops in the tropics experienced significantly less variation in temperature than those in temperate regions (t = 6.71; d.f. = 34.74; p < 0.001, electronic supplementary material, figure S3) but insufficient climate data across all the datasets (only 28 studies of the 43 recorded temperature), meant this aspect could not be fully explored within this study.

Of the 43 studies used, 25 studies had two years of data, 14 studies three years of data and four studies with four or more years of repeated sampling. With these differences in number of years of sampling, our global synthesis has only provided a first step to looking at temporal dynamics. Estimates of temporal dynamics may vary with the number of years sampled and every effort has been made to account for these differences by analysing changes in observed in every two years of each study. It is to be noted that results of the models with the pairwise year calculations were consistent with the model using data across all the years, but further measures to account for any differences caused by a varying number of sampling years, and are beyond the scope of this manuscript. Also, the diversity-stability effect identified, may be linked to sampling effort with lower sampling leading to high CV values and low diversity between years. As this is a collated dataset consisting of various studies that have taken place across several geographic regions across multiple years and we cannot retrospectively change the sampling effort, we acknowledge that the CV may be sensitive to these underlying effects and raise this as a point to be considered in future studies.

Many studies to date have focused on spatial variations observed between crops, fields and across different landscapes [START_REF] Hanley | Increased bumblebee abundance along the margins of a mass flowering crop: evidence for pollinator spill-over[END_REF][START_REF] Beduschi | Spatial community turnover of pollinators is relaxed by semi-natural habitats, but not by mass-flowering crops in agricultural landscapes[END_REF][START_REF] Palma | Ecological traits affect the sensitivity of bees to land-use pressures in European agricultural landscapes[END_REF], while relatively few studies have considered temporal variation caused by differences in crop flowering times [START_REF] Kallioniemi | Local resources, linear elements and mass-flowering crops determine bumblebee occurrences in moderately intensified farmlands[END_REF][START_REF] Garratt | Integrated crop pollination to buffer spatial and temporal variability in pollinator activity[END_REF][START_REF] Grab | Temporally dependent pollinator competition and facilitation with mass flowering crops affects yield in co-blooming crops[END_REF] and even these focussed only on within-season variation. To the best of our knowledge, our study is the first to explore temporal variation in pollinator communities across different crops. Our results highlight the importance of considering both wider pollinator diversity as well as the abundance of dominant species in understanding the inter-annual stability of crop pollinators. Temporal stability of ecosystem functioning increases the predictability and reliability of ecosystem services and understanding the drivers of stability across spatial scales is important for land management and policy decisions [START_REF] Wilcox | Asynchrony among local communities stabilises ecosystem function of metacommunities[END_REF]. Stability in the availability of pollinators is also important from an agro-ecological resilience perspective as increased variation in animal pollination could reduce average yield and yield stability [START_REF] Garibaldi | Global growth and stability of agricultural yield decrease with pollinator dependence[END_REF]. We further propose that the stability and ecological resilience brought about by enhancing the diversity of pollinator communities will contribute beyond agriculture and should be considered alongside longer term conservation targets focussed on maintaining and enhancing wider biodiversity.

the R code for the initial data analyses, and along with M.A., M.P.D.G., D.K., B.J.P., S.G.P. and A.M.K. was involved in helping structure subsequent data analyses and in commenting on several early drafts of the manuscripts. B.B. produced the insect illustrations used in figure 1 in addition to contributing data. All other authors provided the data used in the analyses and contributed to revisions of the manuscript.

33%Figure 1 .

 1 Figure 1. Most dominant taxonomic group of crop pollinators across years at (a) study and (b) site levels with number of studies and number of sites in parentheses.

1 Figure 2 .

 12 Figure 2. The relative change in total abundance of crop pollinators between years are driven by (a) species diversity (Shannon index) with 95% CI and (b) the change in dominant species.

Figure 3 .

 3 Figure 3. Inter-annual variability of crop pollinator species richness is driven by (a) relative change in the abundance of the most dominant species (showing 95% CI), (b) average species diversity (Shannon index) and (c) climatic region.

Figure 4 .

 4 Figure 4. Relative change in single most dominant species (grey) compared to relative change in overall abundance (white) when split into asynchronous (left side) and synchronous (right side) pollinator communities.

Table 1 .

 1 The proportion of studies and sites showing inter-annual changes in the dominant taxonomic groups and species of crop pollinators; actual no. of studies and sites shown within parentheses. Note: one study with five sites (Pisa01) had only morpho-species level data.

		study level		site level
			no		no
		change	change	change	change
	taxonomic	27.9%	72.1%	31.2%	68.8%
	group	(12)	(31)	(117)	(258)
	species (excl	48.1%	51.2%	50.8%	49.2%
	Pisa01)	(20)	(22)	(188)	(182)

Table 2 .

 2 Results of model averaging of candidate models that were within AICc Δ7 of the model with the lowest AICc value. CV of most dominant species remained significant when it was the single most dominant, two-most dominant as well as three-most dominant species.

		fixed effects remaining in the		adjusted
	response variable	averaged model	estimate	SE	z-value	p-value
	CV total pollinator	models with CV calculated across all years of the studies
	abundance	conditional R 2 = 0.33; marginal R 2 = 0.09
		same dominant species	-0.08482	0.03802	2.231	0.0257*
		H' index	-0.15584	0.03932	3.964	7.38 × 10 -5 ***
		climatic region	0.08302	0.09064	0.916	0.3598
		MFC	-0.08627	0.08326	1.036	0.3001
		models with CV calculated for every two years of the studies
		conditional R 2 = 0.35; marginal R 2 = 0.06
		same dominant species	-0.05286	0.03726	1.418	0.15607
		H' index	-0.10368	0.03792	2.734	0.00626**
		climatic region	0.11703	0.08691	1.347	0.17812
		MFC	-0.10889	0.03726	1.322	0.18609
	CV of pollinator species	models with CV calculated across all years of the studies
	richness	conditional R 2 = 0.56; marginal R 2 = 0.19
		climatic region	0.16877	0.08576	1.968	0.049096*
		CV of most dominant species a	0.09774	0.02957	3.305	0.000951***
		H' index	-0.16173	0.02879	5.616	<2 × 10 -16 ***
		MFC	0.00435	0.11645	0.037	0.970190
		models with CV calculated for every two years of the studies
		conditional R 2 = 0.37; marginal R 2 = 0.09
		climatic region	0.111412	0.079390	2.138	0.032545*
		CV of most dominant species a	0.121180	0.032136	3.771	0.000163***
		H' index	-0.048424	0.037559	2.242	0.024961*
		MFC	0.002177	0.051874	0.073	0.942094

a

Downloaded from https://royalsocietypublishing.org/ on 17 March 2021

Acknowledgements. The authors wish to thank Riccardo Bommarco for his data contribution.

Funding. This study was supported EU COST Action Super-B project (STSM-FA1307-150416-070296) and D.S. by the University of Reading Research Endowment Trust Fund (E3530600) and NERC KE Fellowship NE/S006400/1. J.F. by DFG grant FR 3364/4-1; L.G.C. funded by FCT and EU project EUCLIPO-028360 and by CNPq 421668/2018-0; P.Q. 305157/2018-3; M.P.D.G. by an Insect Pollinators Initiative grant no. BB/I000348/1; D.K. by the Dutch Ministry of Agriculture, Nature and Food Quality (BO-11-011.01-011); A.J. and H.Z. by the Bee resources research funds (CAAS-ASTIP-IAR; NSFC31672500) in China; B.M.F. by a Productivity in Research Sponsorship (no. 308948/16-5), Brasilia-Brazil; M.M. and D.W. funded by Waitrose & Partners, Fruition PO and the University of Worcester; and C.W. funded by the DFG grant no. 405945293.

Data accessibility. The data supporting the analyses are available from the University of Reading Research Data Archive http://dx.doi. org/10.17864/1947.291 [START_REF] Senapathi | Temporal variation in global crop pollinator communities[END_REF].