Elisa M Tartaglia

Celine Boucly

Guillaume Tatur

Angelo Arleo

Eye-movements as a signature of age-related differences in global planning strategies for spatial navigation

The ability to efficiently find alternatives routes when faced with unexpected obstacles along our path is among the most compelling evidence of the flexibility of human behaviour. Although a plethora of plausible computations have been put forward to elucidate how the brain accomplishes efficient goal-oriented navigation, the mechanisms that guide an effective re-planning when facing obstructions are still largely undetermined. There is a fair consensus in postulating that possible alternatives routes are internally replayed sampling from past experiences, however, there is currently no account of the criterion according to which those memories are replayed. Here, we posit that paths, which are expected to be more rewarding are replayed more often and that eye movements are the explicit manifestation of this re-planning strategy. In other words, the visual sampling statistics reflects the retrieval of available routes on a mental representation of the environment. To test our hypothesis, we measured the ability of both young and old human subjects to solve a virtual version of the Tolman maze, while we recorded their eye movements. We used reinforcement learning (RL) to corroborate that eye movements statistics was crucially subtending the decision making process involved in re-planning and that the incorporation of this additional information to the algorithm was necessary to reproduce the behavioral performance of both screened populations.

Introduction

The ability to efficiently find alternatives routes when faced with unexpected obstacles along our path is among the most compelling evidence of the flexibility of human behaviour. If, for example, our usual way to the bakery is blocked by some construction work, we would still be able to reach our destination by making -hopefully-a small detour. Although a plethora of plausible computations have been put forward to elucidate how 1 the brain accomplishes efficient goal-oriented navigation (for a review see [START_REF] Madl | Computational cognitive models of spatial memory in navigation space: A review[END_REF][START_REF] Chersi | The cognitive architecture of spatial navigation: Hippocampal and striatal contributions[END_REF], the mechanisms that guide an effective re-planning when facing obstructions are still largely undetermined. There is a fair consensus in postulating that possible alternatives routes are internally replayed sampling from past experiences via network interactions involving hippocampus and prefrontal cortex [START_REF] Boucly | Solving the detour problem in navigation: a model of prefrontal and hippocampal interactions[END_REF]. However, there is currently no account of the criterion according to which those memories are replayed. Here, we posit that paths which are expected to be more rewarding are replayed more often and that eye movements are the explicit manifestation of this re-planning strategy. In other words, the visual sampling statistics reflects the retrieval of available routes on a mental representation of the environment (i.e. on a cognitive map). To test our hypothesis, we measured the ability of human subjects to solve a virtual version of the Tolman maze, while we recorded their eye movements. In his original study, Tolman found that rodents were capable to take detours and to select the shortest path to food pellets. Consequently, he put forward that an internal representation of the physical space is a necessary condition to be able to re-plan one's way in a modified environment [START_REF] Tolman | Cognitive maps in rats and men[END_REF]). In the light of these results, we expected young adults not only to be able to successfully re-plan their way through the maze, but also to be able to sketch a faithful representation of the maze, as further evidence of a cognitive map formation (one that cannot be accounted for in non-human subjects). According to our hypothesis, then, the eye movements statistics collected when subjects are required to find alternative routes, should reflect planning over the cognitive map; if such map is correct, the correct alternative route will be chosen. A much more challenging instance to test our hypothesis upon concerns older adults performance. Aside from general spatial navigation deficits [START_REF] Barrash | Age-related decline in route learning ability[END_REF][START_REF] Wilkniss | Age-related differences in an ecologically based study of route learning[END_REF][START_REF] Burns | Navigation and the mobility of older drivers[END_REF][START_REF] Newman | Spatial Memory and Aging: Performance on a Human Analog of the Morris Water Maze[END_REF][START_REF] Moffat | Effects of age on virtual environment place navigation and allocentric cognitive mapping[END_REF][START_REF] Boucly | Virtual navigation in humans: the impact of age, sex, and hormones on place learning[END_REF][START_REF] Head | Age effects on wayfinding and route learning skills[END_REF][START_REF] Boucly | Spatial knowledge acquisition in younger and elderly adults: A study in a virtual environment[END_REF][START_REF] Lester | The aging navigational system[END_REF], aging has been specifically linked to a delay in cognitive map formation as well as to an impediment in using it [START_REF] Iaria | Age differences in the formation and use of cognitive maps[END_REF]. Moreover, elderly have been shown to be defective in allocentric navigation [START_REF] Moffat | Age differences in the neural systems supporting human allocentric spatial navigation[END_REF][START_REF] Antonova | Age-related neural activity during allocentric spatial memory[END_REF][START_REF] Rodgers | Effects of age on navigation strategy[END_REF][START_REF] Bohbot | Virtual navigation strategies from childhood to senescence: evidence for changes across the life span[END_REF][START_REF] Gazova | Spatial navigation in young versus older adults[END_REF], which would suggest that they rely more on a stimulus-response association type of navigation strategy, rather than on planning. According to our hypothesis, instead, older adults, as their younger counterpart, are capable to do mental planning; likewise, their eye movements reflect planning over a cognitive map. However, we conjectured, older adults re-plan their alternative route over a degraded cognitive map; this might explain why older subjects have been found to be impeded in solving a detour task and/or in taking shortcuts [START_REF] Harris | How age-related strategy switching deficits affect wayfinding in complex environments[END_REF]. To corroborate that eye movements statistics was crucially subtending the decision making process involved in re-planning, we used reinforcement learning (RL) models. Although a phenomenological model, both neuroimaging and neurophysiological studies have provided strong indication that specific anatomically identified brain regions are indeed performing RL based computations [START_REF] Schultz | A neural substrate of prediction and reward[END_REF][START_REF] Berns | Predictability modulates human brain response to reward[END_REF][START_REF] Boucly | Reward representations and reward-related learning in the human brain: insights from neuroimaging[END_REF][START_REF] Pagnoni | Activity in human ventral striatum locked to errors of reward prediction[END_REF]. Moreover, the use of RL to fit behavioural data of (young) observers performing a navigation task has provided a clear indication of the strategies at use. In particular, contrasting model-based and model-free RL algorithms have led to identify whether navigation strategies involve or not the ability of planning November 28, 2018 2 future path selection [START_REF] Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Simon | Neural correlates of forward planning in a spatial decision task in humans[END_REF][START_REF] Keramati | Speed/accuracy trade-off between the habitual and the goal-directed processes[END_REF][START_REF] Gershman | Retrospective revaluation in sequential decision making: A tale of two systems[END_REF][START_REF] Tartaglia | Where to go next? A paradigm for testing human sequential decision making[END_REF]).

Here, we show that the incorporation of EM statistics as additional information to RL algorithms was necessary to reproduce the behavioral performance of both screened populations. In the classical RL theory, the agent (e.g. the observer navigating through the maze) learns from reinforcement the future value of the expected reward at any given state of the environment (e.g. a location in the maze). The value of a state is updated only if the agent has visited it [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. Here, we made a much stronger assumption: the states that observers looked at are used for planning and consequently updated even if they have not been visited. The rationale behind this assumption goes as follow: often, in a natural setting, visual exploration of the environment alone provides enough information about which direction should be taken next. Imagine you are at a crossroad, looking to the right you realize the street is dead-ended, hence you plan to turn left instead. In classical RL, the state values on the right would not get updated since the agent did not visit them. However, we argued, a more efficient and biologically plausible algorithm should be able to assign (in this case negative) values to the right path. More into details, using the recorded data, we fit our RL algorithm to the behavioural choices (i.e. the discretised sequence of maze locations visited by each observer), as well as to the number of fixations performed during the test trials by both young and old observers. In particular, as a first step, we fed to the model (with two free parameters) the sequence of states visited during the exploration phase by each individual subject. This method provided individual (and more realistic) initial conditions to be used, in a second step, in the test trials, during which we compared observers' performance to the model performance. Importantly, the model was enriched with a crucial information: the statistics of eye movements made by each individual subject when planning their next move towards the goal. As a final step, we determined the region of parameters which yielded the closest performance to the behavioral data. We seek to provide, for the first time, a plausible mechanistic account of the process of planning alternative routes when facing unexpected obstacles. Furthermore, we expect these results to lead to a quantification of the extent to which such process differ from young to older observers.

Methods

Participants and set-up

We tested n = 39 participants. Exit points of both right and left paths were hidden by virtual doors, to prevent participants to get a glimpse of the maze structure from the very starting point; virtual doors opened only when subjects were in their close proximity and were not identifiable otherwise.

Subjects were generally allowed to take any path, from any direction, within the maze (except when specified otherwise).

The virtual maze was presented on a projection screen (0.9m * 1.8m). Subjects were sitting at a distance of 1.5m from the screen and moved within the environment -from a first person perspective-using a joystick. Turning rate and walking speed were kept constant i.e. all participants navigated within the environment at the same speed. This allowed to minimize differences in performance that might arise from age-related difficulties in using the joystick. We synchronized the virtual walking with footsteps sound, to render the virtual experience more realistic. Binocular eye movements were recorded throughout the experiment with a headmounted eye tracking system (Eyelink II, SR Research). The calibration of the eye tracking was done before the experiment and was regularly checked upon during the whole experimental session.

Procedure

Before giving their consent, subjects were informed that they were going to participate, for about 1.5 hours, in a virtual navigation study, in which their eye movements would have been recorded. The experiment was divided into three main phases: training, exploration and testing.

During training, instructions were verbally given and assistance with joystick manipulation and task understanding was constantly provided. During both exploration and testing phase, instructions appeared on the screen to minimize any experimenter bias.

Training phase. To reduce motor skills differences, we asked subjects to practice the use of the joystick while navigating in a virtual environment -which was different from the Tolman maze. Several repetitions of this training task were run, depending on how good the experimenter judged the individual ability to handle the joystick. Once subjects were judged at ease with the set-up, they were asked to practice a pointing task in yet another rendering of the virtual environment. In this case navigation was guided, i.e. subjects were asked to follow a given trajectory until a compass appeared on the screen. The task consisted in using the joystick to point the compass arrow towards a previously visited location, which was out of sight from the current position. The rationale of this phase was to get the subjects acquainted to the use of the compass in the virtual environment, before the proper pointing task in the Tolman maze was carried out. Being able to point to the correct direction provided an indication of the observers ability to locate themselves in the environment, as well as an indirect November 28, 2018 4 measure of their ability to form an internal representation of the environment. Eye movements were not recorded during the training phase.

Exploration phase. Subjects were first instructed to freely navigate in the virtual Tolman maze, of whom they ignored the structure; we informed them that a chest was located somewhere in the maze, although they were not explicitly instructed to look for it, or to memorize its location. Subjects were also aware that the chest acted like a portal: once reached it, they would be instantaneously brought back to the maze starting point. In other words, the chest allowed to resume navigation from a fixed reference point, providing a useful direction cue. This first stage of exploration lasted six minutes. Next, we instructed subjects to follow popping out arrows guiding them from the starting state to the Goal trough the three main maze paths, i.e the left, the central and the right path. This "guided exploration" phase lasted not more than two minutes and was designed to ensure that all subject went through each of the three paths at least once before the test phase. A second session of free exploration, identical to the first one, followed. The whole exploration phase lasted twelve minutes (such duration was determined based on a previous pilot study). Eye movements were recorded all along.

Testing phase. We used three different kind of post-exploration tests to evaluate subjects performance, i.e. their ability to grasp the Tolman maze structure. Each test phase will be described in details in the following section.

Performance measurements

Detour task

Participants were instructed to reach the chest from the starting point by traveling the shortest possible distance. The structure of the maze as well as the chest location were left unchanged with respect to the exploration phase. Observers performed a sequence of 7 trials, which we labelled as: 0 -A -B -A -B -A -B. In condition 0, all paths were available, hence the central corridor provided the shortest path to the chest; in condition A, a portion of the central corridor was blocked, therefore the left path became the shortest; in condition B, a portion of both the central corridor and the left path were blocked and the right path became the shortest way to the Goal (Fig 1A). Eye movements were recorded during the whole session. To quantify performance we measured for each subject i, each trial repetition j (j = 1, 2, 3) and each trial type k (k = 0, A, B), the length of the distance travelled from Start to Goal, D ijk , to which we subtracted the shortest path length min(D k ), i.e. the length of the shortest possible path leading from Start to Goal:

∆d ijk = D ijk -min(D k ) (1)
Given the above definition, ∆d ijk ∈ [0, ∞[; performance was optimal when ∆d ijk = 0.

Sketched map task

After a quick break, participants were instructed to draw on an electronic tablet a bird's eye view of the maze. They were asked to label the hallways, the obstacles November 28, 2018 5

A.

B.

C. November 28, 2018 6 -Boucly C., Tatur G., Arleo A., Tartaglia E.M.they encountered and the chest location. They could take as long as they needed to complete the task. Eye movements were not recorded.

To quantify performance we designed a grading system and asked four unbiased raters to grade the subjects' maps according to several criteria (see Table in Appendix and [START_REF] Billinghurst | The use of sketch maps to measure cognitive maps of virtual environments[END_REF]. A maximum of 20 points was assigned if the maze was correctly drawn. Negative points were assigned when subjects drawn extra-elements, i.e. hallways or intersections that did not exist (see Fig. 4C).

Pointing task

After sketching the map, participants were presented with the virtual Tolman maze again and were asked to point toward the chest in four different locations along the left and the right path (Fig. 3). We measured performance by calculating the angular deviation of each subject choice from the actual Goal location, i.e. the error subjects made in pointing towards the chest from their current position.

Eye-movements analysis

Eye movements were recorded binocularly at a rate of 250Hz during both the exploratory and the testing phase. The EyeLink II system allowed to detect saccades and blinks by setting threshold parameters for saccade's velocity (30 • /s), acceleration (9000 • /s 2 ) and motion (0.15 • ). Periods between saccades in which blinks were absent, were identified as fixations. A fixation was quantified in terms of gaze direction, i.e. the angle between the centre of the virtual camera (which coincides with the position of the subject in the virtual space) and the 2D fixation point on the screen, whose coordinates were directly provided by the eye tracking device. From the point of view of a subject moving in the virtual space, the horizontal field of view spanned approximately 106 • .

For the analysis, we focused on the subset of gaze directions recorded while subjects navigated from the maze starting point to the first decision point (FDP, see Fig. 1A), i.e. the portion of space in which subjects were expected to make a decision about which of the three paths to choose. This portion of the maze, which we referred to as region of interest, spanned 5.5m. We analyzed, separately, gaze directions recorded during the exploration phase and those recorded during each test trial, pooling together gaze directions for each population and test trials of the same kind. We were interested in testing two hypothesis concerning eye movements dynamics: first, as a manifestation of learning, the pattern of gaze directions changed significantly when comparing the exploration phase to the test trials; second, as a manifestation of a different decision making dynamics, the pattern of gaze directions within test trials changed significantly when comparing younger to older adults. To test our hypothesis, we partitioned the region of interest in several subregions, for each of which we built a gaze direction distribution polling together the gaze direction data of the whole population. The number of subregions was determined by a sliding window of size 0.5m and step 0.05m. Such a representation allowed us to assess and to compare the evolution in time, from start to FDP, of the gaze direction distribution of each population.

To be able to incorporate eye movements during re-planning of an alternative route in the model, we analysed eye movements statistics for each individual observer during test trials. In particular, we counted the proportion of fixations made straight ahead November 28, 2018 7

towards the central corridor, towards the right and towards the left arm while the observer moved from start to FDP. Note that we discarded the number of times in which observer looked behind towards the starting point simply because none of the observers went back to it before reaching the FDP. Subsequently, we fed the proportion of eye movements to the model.

Model

We used hierarchical reinforcement learning (HRL) to model subjects'performance in the Tolman maze task. HRL, unlike standard RL algorithms, allows the learning agent to take actions extended in space and to keep track of the distance travelled while executing them. We referred to this class of actions as options. Formally, each option is labelled by its initial and terminal state, as well as by its length z (Fig. 6A).

In a given state s, the agent chooses an option o according to the following probability rule (soft-max policy):

P (o|s) = e Q(s,o) τ e o ∈O Q(s,o ) τ (2)
where Q(s, o) is the value associated to option o while in s and τ is a "noise" parameter.

The normalisation takes into account all available options o in state s. If τ → 0, the option with the highest probability to be selected is the one associated with the largest Q value. In this case, the agent is said to be greedy. To the contrary, the larger the τ the more the agent is explorative and is willing to choose options whose Q value is not necessarily the largest. In the limit of τ → ∞, all available options in s become equiprobable.

Once the option o has been chosen, the Q value associated to the option initial state s is updated according to:

Q(s, o) = Q(s, o) + αδ (3)
where α controls the learning rate and δ adjust the previous estimation of the reward given the reward r collected when executing o from s:

δ = r + γ z max o Q(s , o ) -Q(s, o). (4)
As in standard Q-learning algorithms, γ is the discount factor, which weights the rewards according to how far from s they will be obtained (0 ≤ γ ≤ 1). Unlike standard Q-learning, here γ is affected by the option length z, i.e. by the number of steps elapsed since o was chosen, so that longer options are discounted more.

The execution of option o brings the agent to the option terminal state s . The task of the agent is to maximise the total reward across the environment through the update of Q values estimation. If the algorithm has converged, choosing the option associated with the largest Q value at each s, ensures the highest total reward.

In the model-free version of HRL (MF-HRL), Q(s, o) gets updated online, i.e. only after the agent visits s and chooses o. In the model-based version of HRL (DQ-HRL), Q values are updated also offline, through planning. Planning works in the following way: upon visiting s, the transition s → o → s and its associated reward r are stored in a matrix M to build up a "model" of the environment. Before choosing the next option, the agent uses M to randomly sample n pl states and options visited in the past November 28, 2018 8 -Boucly C., Tatur G., Arleo A., Tartaglia E.M.and update their Q values offline. This planning procedure provides faster convergence to the optimal Q values.

In the following, we used DQ-HRL to reproduce the behavioural data. The model has four free parameters, τ, α, γ, n pl ; we fixed γ = 0.9 for both young and old subjects; we assumed the remaining three parameters to differ between the two populations, but not within individual subjects.

Results

Behavioral Results

As expected, we found that young observers significantly outperformed old ones in the detour task. The histogram of ∆d ijk shows that the whole young population performed optimally, or nearly optimally, in all trials, i.e. choosing almost always the shortest possible path to the Goal (Fig. 2A.). A much higher variability and sub-optimality was observed for older subjects, with some trajectories deviating up to 250m from the shortest path length (Fig. 2B.). The difference in performance between the two populations appears even more pronounced when plotting ∆d i averaged across subjects, with young subjects performing significantly better than old ones in all but one trial (Fig. 2C.,F.). Averaging further across repetitions of the same trial type (Fig. 2D.,G.), or across all trials confounded (Fig. 2E.,H.) confirmed these results. Furthermore, we found that young observers were significantly more accurate in pointing towards the Goal location in all tested sites (Fig. 3). Old subjects appear to be less and less able to locate the Goal at each change of direction in the maze (Fig. 3B). Analogous results were found when testing pointing performance across four locations spread on the left arm, or when pointing towards the starting point rather than the Goal (results not shown). Old and young participants map grades were also significantly different. Three examples are shown in Fig. 4. When averaging maps scores across graders, we found a mean score of 17/20 ± 0.1 for young, versus 5/20 ± 1 for old subjects (Wilcoxon rank sum test, p < 10 -4 ), indicating a much higher accuracy in the maze rendering of the young population. The standard deviation of the scores indicates that the evaluation of old subjects maps was, as expected, much noisier. Finally, we checked and found that the three performance measurements were significantly correlated. (Spearman test; correlation ∆d ijk -pointing: p < 10 -6 ; correlation ∆d ijk -map: p = 0.005; correlation pointing-map: p = 0.002; note that the correlation is calculated using ∆d i1B , i.e. only the first repetition of trial B).

Taken together, theses results consistently point to a significant and detrimental effect of age on the ability to build up a coherent representation of the environment.

Eye-movements results

Having conducted the experiment in a virtual simulator provides the advantage of isolating the visual contribution to the navigation task; in the absence of proprioceptive and vestibular signals, observers could rely exclusively on their sight to collect information about the environment. Given that, we expected to find a manifestation of learning and/or of the ongoing decision making process in the dynamics of the ob-November 28, 2018 9 Average grades were, from left to right, 20, 14.5 and -6.

1 1 2 2 3 3 Y O Y O Y O 0 All A All B Δdijk >i < Δd>ij < Δd>ijk < Δd>i < Δd>ij < Δd>ijk < n.s.
servers' eye movements.

In particular, we anticipated significant changes in gaze direction distributions when comparing the exploration phase to the test trials within the same population. During exploration we expected gaze to be more dispersed, with participants trying to maximize the range of their eye movements to probe the unknown environment. During test trials, instead, the uncertainty should be reduced since participants had gained some insight about the environment, had a precise task to accomplish and were not expected to visually explore the environment anymore. Consequently, gaze directions distribution during test trials should be more peaked.

In accordance with our hypothesis, by comparing the gaze distibution's mean, standard deviation and kurtosis in each sliding window of the region of interest (see Procedure), we found that the gaze direction distributions of the young population was flatter during exploration and more picked during test trials. Interestingly, the opposite was observed for the elderly population (a three way ANOVA was performed with three factors: age (old/young), correctness (correct/incorrect), epoch (exploration/tests). Significant differences between exploration and trials were found for the standard deviation and the kurtosis of horizontal gaze distributions for young vs old participants; results not shown). A possible interpretation of these results could be that young participants were more exploratory at first (flat gaze distribution), while they rather exploited once learning had taken place (picked gaze distribution). To the contrary, old people appeared to be less exploratory in the beginning (picked gaze distribution). This behaviour could have led to some sort of confusion during test trials and, consequently, to a flatter gaze distribution. It is worth mentioning that, while these effects were significant for age, they were not when considering correctness as a factor. Subsequently, we analyzed gaze direction distributions of young and old populations virtually walking trough the maze region of interest while performing the test trials. In agreement with the spontaneous anticipation of locomotor trajectory by gaze direction [START_REF] Bernardin | Gaze anticipation during human locomotion[END_REF], we found that also virtual trajectories, or, in other words, joy-November 28, 2018 12 stick turns, were anticipated by gaze direction in both populations. More importantly, we hypothesised that such a gaze anticipatory effect could be delayed in older adults, as a manifestation of a lingering onset of the decision making process. Three principal gaze directional patterns were expected close to the decision point: left, forward and right oriented gaze directions, corresponding to the three possible paths to the goal. However, if we choose to analyze, for example, B trials alone, we should expect a bias of gaze direction towards the right, at least for the population who succeeded in performing the task (i.e. who chose more often the right path). While still considering young and old data categories separately, we analyzed two datasets of gaze directions, pooling together B trials repetitions: 'All' and 'Correct' data. 'All' data refers to the use of gaze directions drawn from all subjects in the considered category, disregarding whether they succeed or not the trial, whereas 'Correct' data refers to the selection of successfully completed trials.

In order to evaluate the onset of the gaze anticipatory effect in the two populations, we evaluated the location within the region of interest (DM * Y,O ) at which the gaze direction distribution significantly departed from unimodality (Hartigan test). As a matter of fact, as soon as subjects got closer to the FDP, their gaze direction was no longer peaked at 0 • , i.e. towards the central corridor, but started to shift towards the right or the left, i.e. towards the two only available paths in B trials, depending on which direction of motion was subsequently chosen. When considering All data populations, we found, as expected, that DM * Y < DM * O , suggesting that young observers were faster in the decision making process. Note that we constrained the speed of the virtual navigation, which was constant and the same for all subjects, hence the result did not depend on a faster motor response in young adults. As evidence that the delayed onset of gaze anticipation was not due to the uncertainty associated to unsuccessful trials but was rather a true age effect, we found that DM * Y < DM * O even when we exclusively considered successful trials (see Fig. 12 in Appendix). Our main hypothesis on eye movements pertained to their statistics, rather than to their dynamics. In particular, we expected the frequency of gaze directions towards a given path to be a proxy of the amount of reward attributed to it; in other words, we hypothesised that while planning an alternative route, observers look more often in the direction from which they expect to attain the highest reward, which, in our case, coincides with the direction that they believe leads to the Goal faster. To test our hypothesis, we included in a model-based hiererchical RL algorithm (DQ-HRL) the updating of the value of the states observers looked at while moving towards the F DP ; we assumed that those are the states observers are relying on to mentally plan their detour to the Goal. In the following we describe in detail how the model works.

Model Results

We first tested the ability of HRL to learn the shortest -and hence most rewardingpath to the Goal in the Tolman maze. As expected, we found that DQ-HRL learned faster and more accurately than both MF-HRL and a standard MB algorithm, which used single-step actions instead of options (i.e. Dyna-Q; Fig. 7B). In DQ-HRL, Q values associated to the first decision point attained their optimal values already after the completion of 15 episodes (Fig. 7C., right panel). Q values in Dyna-Q also reached their asymptote quite fast, i.e. starting from 20 episodes, however, the algorithm was less accurate than HRL, since it could not disentangle most actions values (i.e.

November 28, 2018 13

Egocentric gaze direction (deg) labels the location at which the distribution switches from unimodal to multimodal, i.e. the location at which the eyes anticipated the direction of motion. On the right plot, the gaze direction distribution obtained in the last bin, i.e. closest to the F DP , is represented on a semi-polar plot; the superimposed line labels the median of the distribution. Note how the gaze distribution of Y observers is much more biased towards the right direction. B.

To estimate the variability associated to the decision making point DM * Y,O , we applied for each of the four datasets Young correct (YC)/old correct (OC), Young All data (YAD) / old All data (OAD) the following procedure. For each sliding window, we generated a kernel probability density estimate for the current gaze direction distribution, using a normal kernel smoothing function. We then randomly sampled 20 synthetic datasets from the pdf of the kernel distribution. For each dataset, we performed the Hartigan test. Parametric t-tests showed that 'All' Y and 'All' O data exhibited a significant (p < 0.001) difference between onsets of the multimodal gaze direction pattern. Within each of these groups, changes in onset value when considering 'All' or only 'Correct' trials were not significant.

November 28, 2018 14 7C., left panel). The model free version of HRL was the slowest and less accurate (Fig. 7C., central panel). Importantly, the way in which we defined options in HRL algorithms impedes the agent to repeatedly go back and forth in small subregions of the maze, i.e. to get jammed in unrealistic loops, which we never observed in the subjects behaviour. Those unrealistic loops are hardly avoidable in standard MB algorithms, which use primitive actions. Subsequently, we used DQ-HRL in a two-steps procedure to reproduce the performance of young (Y ) and old (O) subjects during the execution of the test trials (Fig. 6B). First, we fed the model with the actual sequence of states and options taken by each subject during the exploration phase. Initial Q values were set to zero for all subjects; the values of α and n pl were fixed and assumed to be equal for all subjects within a population; the noise parameter τ was unnecessary since state transitions in the model were not governed by any stochastic policy, but were deterministically defined by the actual trajectory followed by the subjects. The reward r associated with the very last transition bringing the agent to the Goal was set to one; all the other rewards were set to zero. The feeding procedure provided, for each subject, the Q values at the end of the exploration phase, i.e. Q 0,i (Fig. 6B). The following step of the procedure consisted in i) setting individual Q 0,i as initial conditions for running the model with individual test trials, and ii) using subjects gaze directions during the execution of test trials as a tool to re-plan a path within the maze. The parameters α and n pl were assumed to remain unchanged for both populations with respect to the exploration phase; furthermore, we set τ 0 since, contrary to the exploration phase, during the tests trials we asked subjects to be "greedy" , i.e. to reach the Goal via the shortest possible path. We run the model for each individual subject in the seven test conditions 0

Q(L) = Q(D) = Q(R) in Fig.
-A -B - A -B -A -B.
To clarify the role of eye movements (EM ) in the model, let us describe the procedure we followed for a given subject i engaged, for example, in a B trial. Q values were initialised at Q 0,i . Starting from s = S, the agent chose an option according to the policy in Eq.2, where Q(s, o) = Q 0,i (s, o), and landed in s = F DP , i.e. in the maze first decision point. The value of Q 0,i (s, o) was updated according to Eq. 3 and the state transition was stored in M together with its associated reward r: M (s, o) ← (F DP, r = 0);

(5) we assumed that, before choosing any other option, the agent used subject i eye movements for planning his next move. From the eye-tracking data, we extracted the number and the direction of fixations made by subject i while navigating from S to F DP (Fig. 5). If, for example, 70% of the fixations were made straight ahead towards the corridor, 20% towards the right arm and, 10% towards the left arm, we stored in M the following transitions:

M (F DP, o U ) ← (s U , r = r U ) (6) M (F DP, o R ) ← (s R , r = r R ) (7) M (F DP, o L ) ← (s L , r = r L ) (8)
where o U labelled the option straight ahead (or UP) and analogously, o R (o L ) the option right (left). The state s U labelled the terminal state of o U coming from F DP November 28, 2018 15 (analogously for s R and s L ; Fig. 6C). Importantly, the value of the reward associated to each transition was set equal to the proportion of EM made in the corresponding direction; in our example, r U = -0.7, r R = 0.2 and r L = 0.1. Note that we set r U negative for all trials in which a block was used (i.e. A and B trials); we reasoned that, since the block was made clearly visible, subjects were well aware of the impossibility to choose o U in those cases (at the expenses of bumping straight into the block) and internally associated it with a negative reward. While still in F DP , the agent randomly sampled n pl states and options from M and updated the corresponding Q values. Crucially, in our procedure EM allow the agent to do planning using a larger sample of transitions (i.e. those associated to visited states, as well as those associated to states which were only looked at) and, eventually, end up with a better and faster estimation of Q values; without EM, instead, the agent would rely only on a single transition (Eq. 5) for planning while in F DP . When the n pl number of planning steps were over, the agent chose almost always (since τ 0) the option associated with the highest Q value, o R in our example, ending up in s R . Once the agent went past F DP , we reasoned that EM would no longer reflect a planning strategy to reach the Goal and hence we did not consider subjects EM (i.e. beyond F DP , only those transitions which were actually executed were stored in M and used for planning, as in standard MB frameworks). The test trial ended as soon as the agent reached the Goal state. The model performance was quantified as in Eq. 1 and compared with the behavioral data: consistently with the data, we plotted the histogram of ∆d ijk for both populations and found that the model behaved qualitatively very similarly to the subjects (Fig. 8A, for a given set of parameters). Box-plots, showing ∆d ijk medians across observers and trials, confirmed this result also quantitatively (see inset in Fig. 8B).

Importantly, when EM were discarded, the model was unable to reproduce observers performance with the same accuracy; in the case of young subjects, while the algorithm performed optimally in 0 and A trials, its performance was severely suboptimal in B trials (left panel in Fig. 8C). The reason is fairly straightforward: at the end of the feeding procedure (i.e. at the end of the exploration phase), Q values of individual young subjects had, on average, converged to their optimal value. For the values associated with F DP , for example, 9C). Accordingly, in the upcoming trial 0, the agent most likely chose option U while in F DP (since τ 0), and successfully reached the Goal travelling trough the central corridor. In A and B trials, instead, U was negatively rewarded because of the block, and the agent chose the L option as the most rewarding one. The L choice successfully led to the Goal in A, but not in B trials. To correctly perform in B trials, standard RL algorithms should re-learn the new configuration of the environment, once blocks are introduced. The same reasoning holds true for old subjects; in this case, however, the variability of the algorithm performance was higher, given that Q values did not yet converge to their optimal values at the end of the exploration phase (right panel in Fig. 8C).

Q(U ) > Q(L) > Q(D) > Q(R) (Fig.
A similar suboptimal behaviour was found when EM were used to identify available paths to plan upon, but not to assign their corresponding rewards (Fig. 8D). In this November 28, 2018 case, Eq. 8 become:

M (F DP, o U ) ← (s U , r = 0) (9) M (F DP, o R ) ← (s R , r = 0) (10) M (F DP, o L ) ← (s L , r = 0) (11)
and, as before, the environment should be re-learned for the algorithm to successfully perform on B trials. Up to know we showed the model results for a single set of parameters. Next, we studied in which region of parameters these results hold. The relevant parameters were α, i.e. the learning rate, and n pl , i.e. the number of planning steps. To compare the model performance to the data we considered the following quantity, for the young and the old population separately:

S = ∆d M (α, n pl ) -∆d D ijk (12)
where k labelled the trial type (k = 0, A, B), j the number of repetitions of each k trial (j = 1, 2, 3), i the subject; the superscripts M and D indicated, respectively, model and data. In other words, we calculated the median of the deviation from the minimum path over subjects and trials for any given pair of parameters (α, n pl ) and compared it to the data; accordingly, the best parameters were those minimized S. We found that, young subjects data were best described by both large α and large n pl ; old subjects data instead were best described by smaller values of both parameters (blue regions in Fig. 9A). Consistently, plotting the model performance, ∆d M ijk , for the best parameters yielded results which were very close to the data for both populations (Fig. 9B). Note that, for the young population, the 0 < α < 0.1 region in the parameter space for which S to be small is actually a local minimum (see Fig. 10 in Appendix).

We then looked at the dynamics of Q values in F DP obtained from the feeding procedure; we found that the parameters which best described the data were also those for which the Q values of the young population were optimal well before the end of the exploration phase, while those of the old population never reached optimality (Fig. 9C). This partly accounts for the flawless performance of the model in reproducing young in the test trials and it provides an indication that young observers learned the structure of the environment already before the end of the exploration phase. Such a result was corroborated by verbal reports of young subjects at the end of the exploration phase, who spontaneously stated that 12min of exploration were unnecessary since they understood the maze structure already after the first couple of minutes.

The ensemble of these results quite naturally leads to the following interpretation: old subjects are slower learners (small α) and they plan their way through the maze less extensively (smaller n pl ). As a consequence, their behaviour is much closer to that of a model free algorithm rather than model based, as exemplified by their Q values dynamics (compare Fig. 9C. with Fig. 7C). It is worth noticing that the performance of the best observer among the elderly was best described by a much bigger n pl than that describing the performance of the worst young subject. It follows that an optimal subject is one who plans his way through the maze more exhaustively (Fig. 10B in the Appendix).

November 28, 2018

Discussion

Despite the vast literature on spatial navigation, not much is known about how the brain adapts to unexpected changes in the environment (for a recent review see Spiers and Gilbert 2015); even less is known about how this ability is effected by a nonpathological aging process. Here, we relied on a RL framework to provide a mechanistic interpretation of how a detour task is solved. The model quite successfully reproduced the age related differences we found in the human ability to solve a Tolman detour task, in particular the impediment of older observers to grasp which alternative route would bring them faster to the Goal. The model supports two pivotal assertions on the role of eye movements in human planning strategies: i) the pattern of eye movements reflects the memory replay of experienced paths, which serves the planning of alternative routes; ii) the statistics of eye movements, i.e. the frequency with which observers look towards a given path, is a measure of the reward they expected when following it. The model's parameters which best fit the data indicated that older adults are slower learners but, indeed, they plan alternative paths through the maze like younger ones, although to a slightly less extent.

the light of these results we put forward that global planning strategies do not radically change with age and that the impediment in solving a detour task found in older adults might rather be due to the building up of the wrong internal representation of the environment. In other words, older adults do plan, but over a wrong cognitive map. Although it remains speculative, this hypothesis gets support from the maze sketched maps drawn by our older subjects, as well as from evidence that it takes a longer for older adults to create an internal representation of the environment while navigating [START_REF] Iaria | Age differences in the formation and use of cognitive maps[END_REF]. Consistently, the time we allocated for exploration in our paradigm might have been insufficient for older observers to build up a correct map of the maze. Moreover, older participants' pattern of eye movements points to a suboptimal strategy of acquisition of information about the environment during the exploration phase, which could also contribute to a defected map formation. Crucially, we checked that the inability we found in older observers was not exclusively due to the lack of both proprioceptive and vestibular signals, which typically provide a reliable support to spatial navigation; when we asked observers to solve the Tolman detour task while actually walking through it, we found just about the same results obtained in the virtual setting (results not yet published).

On eye movements

An abundance of studies have shown that older adults are significantly impeded in spatial navigation tasks (for reviews see [START_REF] Moffat | Aging and spatial navigation: what do we know and where do we go?[END_REF][START_REF] Lester | The aging navigational system[END_REF]. A bunch of these contributions measured performance differences between younger and older adults in tasks which supposedly relied on the ability to use and/or to form a cognitive map, finding, as we did, a detrimental effect of aging on spatial navigation performance [START_REF] Newman | Spatial Memory and Aging: Performance on a Human Analog of the Morris Water Maze[END_REF][START_REF] Moffat | Effects of age on virtual environment place navigation and allocentric cognitive mapping[END_REF][START_REF] Iaria | Age differences in the formation and use of cognitive maps[END_REF]. There is, however, to the best of our knowledge, no study which measured performance in a Tolman detour task in both younger and older observers and, more importantly, which analyzed the role of eye movements during a route re-planning strategy. In the literature, eye movements have been strongly implicated in memory encoding 28, 2018 18 and retrieval; for example, limiting the number of fixations appears to be detrimental for both, when compared to free-viewing conditions [START_REF] Henderson | Eye movements are functional during face learning[END_REF][START_REF] Johansson | Look here, eye movements play a functional role in memory retrieval[END_REF]. Moreover, altered fixation patterns in older adults have been proposed to be functionally linked to memory deficits [START_REF] Chan | Can changes in eye movement scanning alter the age-related deficit in recognition memory? Frontiers in psychology 2[END_REF][START_REF] Hannula | Rapid onset relational memory effects are evident in eye movement behavior, but not in hippocampal amnesia[END_REF][START_REF] Olsen | The role of relational binding in item memory: evidence from face recognition in a case of developmental amnesia[END_REF][START_REF] Ryan | Amnesia is a deficit in relational memory[END_REF][START_REF] Rondina | The organisation of spatial and temporal relations in memory[END_REF][START_REF] Shih | Aging, eye movements, and object-location memory[END_REF][START_REF] Voss | Spontaneous revisitation during visual exploration as a link among strategic behavior, learning, and the hippocampus[END_REF]. Consistently, we found that gaze direction distributions differed significantly between young and old observers during both the exploration and the probe trials (although we did not find any difference in the sheer number of fixations made; results not shown). In particular, the pattern of fixation of older observers exploring the maze, points to an inefficient encoding of stable information about the environment; likewise, the pattern of fixations recorded during the execution of the test trials is consistent with a suboptimal retrieval process, which, in older adults, leads up to a delayed decision about which path to choose (Fig. 5).

If the link between eye movements and memory is fairly well documented, much less is known about eye movements in the context of the re-planning of alternative routes. Two important pieces of evidence related eye movements and activity in the hippocamone of the brain structure most strongly implicated in the replay of past experiences as well as in the planning of future events [START_REF] Buckner | The role of the hippocampus in prediction and imagination[END_REF][START_REF] Foster | Reverse replay of behavioural sequences in hippocampal place cells during the awake state[END_REF][START_REF] Gelbard-Sagiv | Internally generated reactivation of single neurons in human hippocampus during free recall[END_REF][START_REF] Javadi | Hippocampal and prefrontal processing of network topology to simulate the future[END_REF]Johnson and Redish 2007;[START_REF] Pastalkova | Internally generated cell assembly sequences in the rat hippocampus[END_REF][START_REF] Pfeiffer | Hippocampal place-cell sequences depict future paths to remembered goals[END_REF][START_REF] Schacter | The future of memory: remembering, imagining, and the brain[END_REF][START_REF] Wu | Hippocampal replay captures the unique topological structure of a novel environment[END_REF]. First, during the encoding of novel stimuli, the number of fixations correlate with hippocampus activity, suggesting that visual sampling might be directly related to the formation of representations in the hippocampus [START_REF] Liu | Visual sampling predicts hippocampal activity[END_REF]. Second, the hippocampus guides where to look during memory retrieval [START_REF] Hannula | Rapid onset relational memory effects are evident in eye movement behavior, but not in hippocampal amnesia[END_REF][START_REF] Hannula | The hippocampus reevaluated in unconscious learning and memory: at a tipping point? Frontiers in human neuroscience[END_REF][START_REF] Ryals | Hippocampal contribution to implicit configuration memory expressed via eye movements during scene exploration[END_REF][START_REF] Ryan | Amnesia is a deficit in relational memory[END_REF]. These findings appear to be fully consistent with what we propose: eye movements subserve the retrieval of experienced paths from a mental representation of the environment, upon which future routes are planned. In other words, eye movements are the explicit manifestation of hippocampus-based computations implicated in planning. Our theory pushes the link between eye movements and planning even one step further: gaze directions associated with planning reflect the expectation of future rewards, meaning that we plan more often upon routes which we believe to be more rewarding and which, consequently, we deem more valuable. Furthermore, solving the detour task imply a re-evaluation of the value of the options available after the alteration of the environment (i.e. the introduction of the block in the maze); such an updating of reward assignment has been proposed to implicate hippocampal-striatal connectivity [START_REF] Pennartz | The hippocampal-striatal axis in learning, prediction and goal-directed behavior[END_REF][START_REF] Wimmer | Preference by association: how memory mechanisms in the hippocampus bias decisions[END_REF]. Note that, here, we implied that planning consists in two concurrent processes i) the memory replay of experienced paths; ii) the re-evaluation, crucial for future decisions, the benefits associated with any of these paths. Nonetheless, we acknowledge that planning does not necessarily rely on the recall of past experiences, e.g. in some circumstances, hippocampal cells are able to generate firing sequences representing paths never taken by the animal [START_REF] Gupta | Hippocampal replay is not a simple function of experience[END_REF][START_REF] Ólafsdóttir | Hippocampal place cells construct reward related sequences through unexplored space[END_REF]. Moreover, it is important to stress that planning, in particular in the context of detours, is supported not only by the hippocampus but also by prefrontal cortices, although the relative contribution of the two is still unclear [START_REF] Martinet | Spatial learning and action planning in a prefrontal cortical network model[END_REF][START_REF] Spiers | Thoughts, behaviour, and brain dynamics during navigation in the real world[END_REF][START_REF] Boucly | Solving the detour problem in navigation: a model of prefrontal and hippocampal interactions[END_REF][START_REF] Javadi | Hippocampal and prefrontal processing of network topology to simulate the future[END_REF]. Furthermore, both ventromedial November 28, 2018 19

prefrontal and striatal areas have been suggested to code for the value of the reward associated to mentally planned scenarios [START_REF] Benoit | Ventromedial prefrontal cortex supports affective future simulation by integrating distributed knowledge[END_REF][START_REF] Lin | Medial prefrontal cortex: adding value to imagined scenarios[END_REF].

On modelling

Reinforcement learning has been widely used as a learning rule in spatial navigation, not only for its efficiency, but also on the account of its biological plausibility. Midbrain dopaminergic neurons in the substantia nigra pars compacta (SNc) and ventral tegmental area (VTA) supply the striatum with dopamine, signalling a reward prediction error [START_REF] Schultz | A neural substrate of prediction and reward[END_REF], which represents the core of temporal-difference (TD) RL algorithms. The striatum itself is thought to be responsible for the computation of stimulus-response associations typical of MF versions of RL [START_REF] Gläscher | States versus rewards: dissociable neural prediction error signals underlying model-based and modelfree reinforcement learning[END_REF]Johnson et al. 2007;[START_REF] Van Der Meer | Ventral striatum: a critical look at models of learning and evaluation[END_REF]. The more flexible learning rule implemented by MB algorithms, which relies on an explicit internal model of the environment, is instead carried by the hippocampus [START_REF] Gustafson | Grid cells, place cells, and geodesic generalization for spatial reinforcement learning[END_REF][START_REF] Hasselmo | A model of prefrontal cortical mechanisms for goal-directed behavior[END_REF][START_REF] Hirel | The hippocampo-cortical loop: spatio-temporal learning and goal-oriented planning in navigation[END_REF][START_REF] Martinet | Spatial learning and action planning in a prefrontal cortical network model[END_REF][START_REF] Simon | Neural correlates of forward planning in a spatial decision task in humans[END_REF].

Recently, the literature has converged on the idea that a combination of both MB and MF algorithms is most suitable to describe human navigation strategies, or, more generally human sequential decision making [START_REF] Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Daw | Modelbased influences on humans' choices and striatal prediction errors[END_REF][START_REF] Wunderlich | Dopamine enhances modelbased over model-free choice behavior[END_REF][START_REF] Gershman | Retrospective revaluation in sequential decision making: A tale of two systems[END_REF][START_REF] Lee | Neural computations underlying arbitration between model-based and model-free learning[END_REF][START_REF] Otto | Cognitive control predicts use of model-based reinforcement learning[END_REF][START_REF] Lee | Flexibility to contingency changes distinguishes habitual and goal-directed strategies in humans[END_REF]. In particular, the Dyna-Q algorithm [START_REF] Sutton | Integrated architectures for learning, planning, and reacting based on approximating dynamic programming[END_REF]) has been shown to be an optimal candidate, given that it relies by construction on a mixture of the the MB system, using an internal model of the environment, simulates offline state action transitions whose Q values are then updated through the MF system via a TD algorithm [START_REF] Gershman | Retrospective revaluation in sequential decision making: A tale of two systems[END_REF]).

Here, seeking computational efficiency, we implemented Dyna-Q in the framework of (DQ-HRL). The advantage of partitioning the environment-as well as the actions therein-in larger chunks is well known computationally [START_REF] Badre | Frontal cortex and the discovery of abstract action rules[END_REF][START_REF] Botvinick | Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective[END_REF][START_REF] Sutton | Reinforcement learning: An introduction[END_REF] and it has recently been corroborated, behaviourally, by the finding that the elaboration of plans in human observers is indeed hierarchical [START_REF] Balaguer | Neural mechanisms of hierarchical planning in a virtual subway network[END_REF]. We showed that DQ-HRL is faster in learning the shortest path to the Goal in a (block-free) Tolman maze, than both classical Dyna-Q and the model free version of HRL (MF-HRL; Fig. 7). This is mainly because classical algorithms need to learn the value of each state and action within each arm of the maze, which would entail wasting time in going back and forth through sub-portions of a selected arm. To the contrary, an algorithm like HRL would learn the value of an arm as a whole, resulting in much faster learning. However, although extremely efficient, even HRL would need to re-learn the new strucof the environment once the blocks are introduced and, consequently, it would always perform poorly with respect to a young observer, who is capable of taking the optimal detour without further training. In other words, in any RL algorithm the agent would need to visit several time the state immediately close to the block to learn to attribute the appropriate value to it and to be able to choose a suitable alternative (see for example [START_REF] Russek | Predictive representations can link model-based reinforcement learning to model-free mechanisms[END_REF]. However, none of our subjects during the test trials needed to do so, i.e. none of our subjects ever walked straight to the block: looking at the obstacle from a distance was enough to grasp that the central corridor was occluded and that one of the other available paths had to be chosen instead. Inspired by this observation, we gave the agent in the DQ-HRL algorithm the ability to plan over the 28, 2018 20 states our observers only looked at, ending up with a RL model which, for the first time, can learn the Tolman detour task in one-shot. Upon fitting the subjects' trajectories and eye-movements, our model predicts that older adults are not only slower learners, but also less efficient planners compared to younger adults. Plausibly, ageing has an effect on the quality of the internal representation of the environment on which planning is based. Future work might focus on pushing the eye-movements analysis to a higher level of sophistication, e.g. to understand in which respect a cognitive map might result inaccurate and whether ageing affects specifically the binding of the different structural elements of the environment together or rather other aspects of the map formation.

November 28, 2018 21

Sketched map evaluation criteria Score

A main alley (north-south axis) exists +1

The Goal location is facing the Start location +1

The southernmost intersection with the main alley is cross-shaped (intersection 1/point 1) +2

An intersection located to the left of the main alley exists (intersection 2/point 2) +2

An intersection located to the left of the main alley exists (intersection 3/point 3). Intersection 2 and 3 can be at the same location forming a cross-shaped intersection

+2

Intersection 2 is located south of intersection 3 +2.5

Points 1 and 2 are connected through a path which is not the main alley +2

Points 1 and 3 are connected through a path which is not the main alley +2

Points 1 and 2 are connected via 3 segments forming a 90 deg with each other +1

Points 1 and 3 are connected with 3 segments forming a 90 deg with each other +1

The path connecting point 1 and 2 is shorter than the path connecting point 1 and 3 +2.5

No-entry signs exist between point 1 and point 2 and between point 2 and point 3 +0.

Figure 1 :

 1 Figure 1: A. Shortest paths from Start (labelled as S) to Goal (labelled as G) in the three conditions of the test trials: from left to right, no block (0), block A, block B respectively. F DP labels the first decision point in the maze. The actual dimensions of the maze are also indicated. B. A picture of the chest located in G. C. Simulator view of the maze from the starting point S in the subjects' prospective.

Figure 2 :

 2 Figure 2: Behavioral results -Detour task. Histogram of the distance travelled from Start to Goal across the 7 trials (0-A-B-A-B-A-B) for the young (A.) and the old population (B.) An optimal performance implies ∆d i,j,k = 0 (see text for more details). Travelled distance averaged across observers (C.), across observers and trials of the same type (D.), across observers and all trial types (E.) The only non significant result is labelled as n.s.. All p-values have been calculated via the Wilcoxon rank sum test. No subject was excluded from the analysis. Panels F., G., and H. show the box-plots corresponding respectively to panels C., D., E. The green points label the outliers.

Figure 3 :

 3 Figure 3: Behavioral results -Pointing task. A. The red circles label the four locations along the right path in which observers were required to stop and point to the Goal position.B. Average across subjects of the deviation from the Goal position measured in each of the four locations in A. While the error in the young population tends to remain constant (and very small), it increase with the number of changes in direction for the old population. Results are significant in any of the tested location (location 1, p = 0.04; location 2, p = 0.001; location 3, p = 0.0005; location 4, p < 10 -5 ; Wilcoxon rank sum test). Following the same procedure, we also tested four locations along the left path and found similar results. C. Same as in B., but for individual young observers. D. Same as in C., but for individual old observers.

Figure 4 :

 4 Figure 4: Behavioral results -Sketched map task. One young (A.) and two old subjects (B., C.) rendering of the Tolman maze. Drawing were done right after the detour task. Observers were instructed to label in green the Goal location and in red the blocks. Average grades were, from left to right, 20, 14.5 and -6.

Figure 5 :

 5 Figure ??

-Figure 6 :Figure 7 :Figure 8 :Figure 9 :Figure 10 :

 678910 Figure 6: A. For modelling purposes, the Tolman maze is decomposed in a set of 52 states (black squares in the cartoon) and 6 non-overlapping options (grey arrows). Each option is bidirectional, although, for clarity, only one direction is depicted in the figure. The length k of an option is given by the number of underlying states it covers. For example, the first option initiates at the starting point of the maze (S) and terminates at the first decision point (F DP ), hence k = 6. B. The model was first "fed" with the individual subjects trajectories measured during the exploration phase, and produced the Q values of each individual subject as output. Those values, together with individual subjects eye-movements, were used to probe the model with the sequence of behavioural test trials 0-A-B-A-B-A-B. In both the exploration and the test phase the relevant parameters were the number of planning steps n pl and the learning rate α. C. Subjects eye-movements were used in the model to modulate the reward associated with the options available at F DP during the test trials. In the cartoon a B trial is depicted. If a subject, navigating from S to F DP , fixated 20% of the time towards the right, the reward associated to the transition F DP → s R was set to r = 0.2. Likewise, if the proportion of fixation to the left was 10%, the reward for the transition F DP → s L was set to r = 0.1.

Figure 12 :

 12 Figure S??

  Age and gender of the sample are reported in the following table. was used as Goal(Fig 1B). Any visual cue within the environment, which could help subjects to reach the Goal faster, was avoided. Three main paths allowed to reach the chest: a central corridor, a left and a right path, rejoining the central corridor at different locations (exit points). The central corridor provided the shortest path to the Goal, while the right one provided the longest (Fig1A).

	Gender Young (< 45 years) Old (>65 years)
	Female	9	10
	Male	10	10

The mean age for young and old observers was, respectively, 27.3y and 78.5y. Experiments were run on a simulation module developed in Unity 3D, which allowed to create the virtual maze and to record subjects trajectories. A view from above of the maze, together with its dimensions is shown in Fig

1A

. A chest at the end of the November 28, 2018 3 -Boucly C., Tatur G., Arleo A., Tartaglia E.M.maze


Sofia B S D Castro
email: sdcastro@fep.up.ptsami.dakhlia@gmail.com

Sami Dakhlia

Peter B Gothen
email: pbgothen@fc.up.pt

Jean-Marc Bonnisseau

Direct perturbations of aggregate excess demand

Keywords: Arrow-Debreu, transversality, exchange economy JEL classification: D50, C62, C02

Nous démontrons qu'une économie d'échange (définie par ses préférences et ses dotations) qui génère une fonction de demande excédentaire aggrégée (DEA) z est proche de l'économie associée à la DEA z , perturbation arbitraire de z.

Introduction

Genericity results -such as establishing that the set of equilibrium prices constitutes a manifold of a certain dimension or that the number of regular equilibria is finite and stable -are obtained by perturbation techniques, where the underlying primitives (e.g., preferences and endowments) are subjected to infinitesimal changes. [START_REF] Debreu | Economies with a finite set of equilibria[END_REF] proof of local isolation of regular economies, for example, requires a perturbation of endowments. Extending the proof to critical economies requires additional effort: since the perturbation must also affect the curvature of the equilibrium manifold, a linear perturbation such as the one at the heart of Debreu's proof no longer suffices. An example of a higher-order perturbation is found in Mas-Colell (1985, Proposition 8.8.3), who shows that for a one-dimensional parameterization of economies, a "flat" AED is not generic in the first consumer's utility. It is instructive to note that Mas-Colell must resort not only to a perturbation of endowments, but also to a quadratic perturbation of (indirect) utility. [START_REF] Allen | On the finiteness of the equilibrium price set[END_REF], on the other hand, establishes much more, namely, finiteness for multi-dimensional parameterizations of economies by employing a theorem by [START_REF] Tougeron | Idéaux de Fonctions Différentiables[END_REF], according to which local finiteness of the number of pre-images is a generic property of smooth functions. The result comes at a cost, however, in that Allen works directly with AED functions rather than with the agents' underlying preferences and endowments. In this note, we show that Allen's approach is fully justified.

Specifically, we establish that the economy, i.e., preferences and endowments, that generates a given AED function is close to the economy generating the AED obtained by an arbitrary perturbation the initial one. 1We show that the Sonnenschein-Mantel-Debreu results (roughly, if it "looks" like an AED, it is an AED for some economy) are stable in the following sense: the economy underlying a perturbed AED function is necessarily close to the economy behind the original AED function. Moreover, the two economies differ (slightly) in the preferences and endowments of the first consumer only.

Part of the challenge, of course, is to show that perturbations of AED functions correspond to "legal" perturbations of preferences, i.e, perturbed utility functions must continue to satisfy the canonical properties of utility functions.

This opens wider the door to further genericity and determinacy research based on transversality arguments since one may now perturb the AED function directly thus sparing one the more tedious exercise of perturbing economic primitives.

In a companion paper, Castro and Dakhlia (2008) make full use of this result to establish Thom-Boardman stratification of AED, which requires high-order perturbations that would have been more difficult to obtain by conventional means. The stratification result, in turn, provides an alternative proof of finiteness of all equilibria, including critical ones.

Notation

Consider an economy with L commodities ( = 1, . . . , L) and I traders (i = 1, . . . , I). Let Ω be the non-negative orthant of R L and let each trader i be defined by her endowment ω i ∈ R L and her preferences i , a partial order on Ω with the following properties:

1. rationality (completeness, reflexivity, and transitivity) 2. continuity ({x : y i x} and {y : y i x} are open); 3. non-satiation (x ≥ y (x ≥ y , ∀ = 1, . . . , L) and x = y ⇒ x i y); 4. strict convexity (x ∼ i y and x = y ⇒ ∀α ∈ (0, 1), αx + (1α)y i x).

Let Ξ denote the space of all such preferences. Following [START_REF] Kannai | Continuity properties of the core of a market[END_REF], every i ∈ Ξ may be represented by a continuous utility function u i : Ω → R defined as follows: for any x ∈ Ω, there exists a unique x in the principal diagonal of Ω such that x ∼ i x (i.e., agent i is indifferent between x and x). Then, let u i (x) ≡ x , where . is the Euclidean norm. Denote C * the class of utility functions thus defined.

Let u 1 , u 2 ∈ C * represent the preferences 1 , 2 ∈ Ξ of agents 1 and 2. The metric

ρ( 1 , 2 ) = max x∈Ω |u 1 (x) -u 2 (x)| 1 + x 2
induces a minimal topology on Ξ with A ≡ {(x, y,

) : x y} open in Ω × Ω × Ξ. Intuitively speaking, preference order 1 is close to preference order 2 if ρ( 1 , 2 ) is small. Similarly, u 1 is close to u 2 if the Euclidean distance |u 1 (x)-u 2 (x)
| is bounded by a small number for all x in any compact set. 2 Finally, we define a perturbation of preferences as preferences that are distinct but arbitrarily close to the original ones. Similarly, a perturbation of a utility function is defined as a distinct, but arbitrarily close, utility function.

Result

Any preference ordering can be represented by exactly one utility function in C * . Thus, perturbations of a C * -utility necessarily imply perturbations of the underlying preferences and vice-versa. Moreover, direct perturbations of AED correspond to perturbations of preferences and endowments.

The following lemmas show that a perturbation of preferences corresponds to a perturbation of utility, and vice-versa. Proof. Because u is close to u 0 , we know that for any compact subset K ⊂ Ω,

∃ε > 0 : |u(x) -u 0 (x)| < ε, ∀ x ∈ K. We have |u(x) -u 0 (x)| 1 + x 2 ≤ |u(x) -u 0 (x)| < ε, ∀ x ∈ K.
For u ∈ C * there exists a constant C, depending only on L = dimΩ, such that 0 ≤ u(x) < C x . (See Kannai and, in particular, his proof of Theorem 3.2.) We have

∀ x ∈ Ω, |u(x) -u 0 (x)| 1 + x 2 ≤ |u(x)| + |u 0 (x)| 1 + x 2 ≤ 2C x 1 + x 2 ,
which converges to zero as x approaches infinity, that is,

∀ ε > 0, ∃R > 0 : x > R ⇒ 2C x 1 + x 2 < ε.
Take K to be the compact set Cl(B R (0)), the closure of the ball of radius R about the origin. Therefore by definition,

ρ( , 0 ) = max x∈K |u(x) -u 0 (x)| 1 + x 2 < ε. Lemma 3.2. Consider preferences 0 ∈ Ξ represented by a utility function u 0 ∈ C * . Let ∈ Ξ, represented by a utility u ∈ C * , be a perturbation of 0 .
Then u is a perturbation of u 0 .

Proof. We need to establish that for any compact subset K ⊂ Ω,

∀ ε > 0, ∃δ > 0 : ρ( , 0 ) < δ ⇒ max x∈K |u(x) -u 0 (x)| < ε.
From Kannai's equation 3.2 (p. 799), we have

{ : ρ( , 0 ) < δ} = : max x≤R |u(x) -u 0 (x)| 1 + x 2 < δ . Since K is compact, we can choose R such that K ⊂ B R (0) so that x 2 ≤ R 2 . Then |u(x) -u 0 (x)| < δ(1 + x 2 ) ≤ δ(1 + R 2 ).
In order to have |u(x) -

u 0 (x)| < ε, it suffices to choose δ such that δ < ε/(1 + R 2 ).
In light of these two lemmas, we can henceforth interchangeably work with preference and C * -utility perturbations, depending on which is more convenient.

We now turn to the excess demand of an agent endowed with ω i ∈ R L . Trader i solves Utility Maximization Problem (UMP)

max x i ∈Ω u i (x i ) such that p • x i ≤ p • ω i . ( 1
)
Strict convexity of preferences ensures that the solution, x i (p, p•ω i ), is unique and a continuous function of both price vector p and endowment ω i . Nonsatiation guarantees that the budget constraint is binding and can thus be written as p

• x i = p • ω i ⇔ p • (x i -ω i ) = 0. (2)
Geometrically, the constraint is the hyperplane through ω i orthogonal to p, while the solution to the UMP corresponds to the point of tangency between the level curves of u i and the hyperplane. For smooth level curves, the point of tangency is located where ∇u i , the gradient of u i , is parallel to p. The excess demand for trader i is defined as

z i (p) = x i (p, p • ω i ) -ω i ,
and the AED for the economy is given by z(p) = n i=1 z i (p). The main result of this section, Theorem 3.2, establishes that a perturbation of AED is equivalent to a perturbation of economic primitives, that is, preferences and endowments. Note that, from the definition of AED, it suffices to consider the perturbation of the excess demand of a single consumer. We will thus drop the agent-specific superscript, when no confusion is possible.

We proceed by first establishing Theorem 3.1, which states that the perturbation of an agent's utility and endowment is equivalent to a perturbation of her excess demand. This will require a sequence of lemmas. Our main result (Theorem 3.2) then follows as a consequence of considering utilities in C * . Theorem 3.1. Consider a consumer with endowment ω ∈ Ω and preferences satisfying ( 1)-( 4) that are represented by a C 2 (twice differentiable) utility function (not necessarily in C * ). Let demand x 0 (p) be the unique solution to the UMP [START_REF] Allen | On the finiteness of the equilibrium price set[END_REF]. Then, (I) A solution to the UMP defined by a perturbation (ū, ω) of (u, ω) is a perturbation x(p) of x 0 (p).

(II) Conversely, a perturbation x(p) of x 0 (p) is the unique solution of the UMP defined by a perturbation (ū, ω) of (u, ω).

Proof. The proof of part (I) is a consequence of the continuous dependence of demand on price, endowments and utility as proved in Lemma 3.3. Since strict convexity of preferences guarantees that x 0 (p) ≡ x 0 (p, p • ω) is a continuous function of ω, any perturbation of ω will correspond to a perturbation of x 0 (p). Lemma 3.3 shows that the same applies to perturbations of utility.

To prove part (II), we show that a perturbation of the solution must originate in a perturbation of utility and/or endowments. It is here that we need the assumption of a twice-differentiable utility function. At the solution x 0 (p), the budget constraint ( 2) is tangent to the indifference curve containing x 0 , which means that the gradient of u at x 0 , ∇u(x 0 ), is orthogonal to p • (xω) = 0. As such, ∇u(x 0 ) is parallel to p.

Let x(p) be a perturbation of x 0 (p), say xx 0 < δ for small δ > 0. If x does not satisfy the budget constraint, we perturb ω to ω so that p • (xω) = 0. In addition, we shall choose ω so that the hyperplane described by p • (xω) = 0 is parallel to the original one. Denote the solution to the UMP defined by u and the new restriction by x 0 (p). We have

d(x 0 , x) ≤ d(x 0 , x 0 ) + d(x 0 , x),
where d(x 0 , x) < δ and d(x 0 , x 0 ) can be made small by continuity of the demand with ω. Since d(ω, ω) ≤ d(x 0 , x) < δ, we have d(x 0 , x 0 ) < ε for a convenient choice of δ. Hence, x is also a perturbation of x 0 and we can henceforth suppose that the demand for the unperturbed problem and x belong to the same budget constraint, dropping the use of the prime.

It remains to show that we can perturb the utility so that x(p) is the solution to the UMP.

Define Φ : Ω → Ω by

Φ(x) = x -ϕ( x -x 0 )(x -x 0 ),
where ϕ : R → R is constant and equal to 1 for 0 ≤ t ≤ δ 1 , ϕ(t) is constant and equal to zero for t ≥ 2δ 1 and ϕ(t) is smooth and decreasing for for some constants A and B. We choose δ 1 so that δ < δ 1 in order to guarantee that ϕ( xx 0 ) = 1. Clearly, Φ is the identity for xx 0 ≥ 2δ 1 and a translation for xx 0 ≤ δ 1 .

δ 1 ≤ t ≤ 2δ 1 . (See Figure 1.) We can choose ϕ such that |ϕ (t)| ≤ A δ 1 and |ϕ (t)| ≤ B δ 2 1 (3) δ 1 δ 1 2 1
Define ū(x) = u(Φ(x)). We have

ū(x) = u(Φ(x)) = u(x -ϕ( x -x 0 )(x -x 0 )) = u(x 0 )
and ∇ū(x) = ∇u(x 0 ) (see Lemma 3.4 below). The level hypersurface of ū at x is thus tangent to p • (xω) = 0 and hence, x is a solution to the UMP defined by ū. Lemmas 3.5 and 3.6 below show that ū is increasing in x and that its indifference curves are convex, thus ensuring that the underlying preferences satisfy (1)-( 4), thereby completing the proof. Lemma 3.3. Let x be the unique solution to the UMP defined by a utility u and endowment ω. Then x is a continuous function of p, ω and u.

Proof. It is clear that x is a continuous function of p and ω. To prove continuity on u, we proceed by contradiction. Let u n be a sequence of utility functions such that as n → +∞, we have u n → u. Let x n = x n (p) be the unique maximum of u n on

H(p, ω) = {x : p • (x -ω) = 0}.
Assume lim n→+∞ x n = x. By compactness of H(p, ω), we may assume that x n → x * = x n , passing to a subsequence if necessary.

Define δ = u(x)u(x * ).

Lemma 3.4. Let u and ū be as defined in the proof of Theorem 3.1. Then we have ∇u(x 0 ) = ∇ū(x).

Proof. We use the definition of ū to calculate partial derivatives and obtain

∂ ū ∂x j (x) = m i=1 ∂u ∂x i (Φ(x)) ∂Φ i ∂x j (x)
and

∂Φ i ∂x j (x) = δ ij -ϕ ( x -x 0 )(x -x 0 ) ∂ x -x 0 ∂x j ,
where

δ ij = 1 if i = j and δ ij = 0 otherwise. At x = x, ϕ ( x -x 0 ) = 0 because ϕ( x -x 0 ) = 1 for 0 ≤ x -x 0 ≤ δ 1 . Hence, ∂ ū ∂x j (x) = ∂ ū ∂x j (Φ(x)) = ∂ ū ∂x j (x 0 ).
Lemma 3.5. Let u and ū be as defined in the proof of Theorem 3.1. Then ū is increasing in each of its arguments.

Proof. From Lemma 3.4, we know that

∂ ū ∂x j (x) = m i=1 ∂u ∂x i (Φ(x)) ∂Φ i ∂x j (x)
and

∂Φ i ∂x j (x) = δ ij -ϕ ( x -x 0 )(x -x 0 ) ∂ x -x 0 ∂x j = δ ij -ϕ ( x -x 0 )(x -x 0 ) x j -x 0j x -x 0 .
By choosing δ

δ 1 A so that A δ 1 δ 1, we have ϕ ( x -x 0 )(x -x 0 ) x j -x 0j x -x 0 ≤ |ϕ ( x -x 0 )(x -x 0 )| ≤ A δ 1 δ 1. Again, if x -x 0 ≤ δ 1 or x -x 0 ≥ 2δ 1 , we have ∂ ū ∂x j (x) = ∂u ∂x j (Φ(x)).
Otherwise, x is in a compact (in the annular region bounded by the discs of radius δ 1 and 2δ 1 ) and ∂u ∂x j is bounded away from zero (as u is increasing in each of its arguments). Hence, by making the absolute value of ∂Φ i ∂x j sufficiently small for i = j and close to one for i = j, we obtain

∂ ū ∂x j (x) > 0.
Lemma 3.6. The indifference curves of ū as defined in the proof of Theorem 3.1 are convex.

Proof. From Thorpe [START_REF] Thorpe | Elementary Topics in Differential Geometry, Ergebnisse Band[END_REF], we know that the normal curvature of a level hypersurface of a function u in the direction of a vector v ( v = 1) perpendicular to ∇u(x) is given by

- 1 ∇u(x) < v, H u (x)v >,
where H u (x) is the Hessian of u at x and < v, H u (x)v > represents the quadratic form defined by H u (x) (see exercise 2.1 in [START_REF] Thorpe | Elementary Topics in Differential Geometry, Ergebnisse Band[END_REF] and section 2 of Gladiali and Grossi [START_REF] Gladiali | Strict convexity of level sets of solutions of some nonlinear elliptic equations[END_REF]).

Because the level curves of u are convex, we know that H u (x) is positive definite for all x. We show that the entries of H ū(x) are close to the entries of H u (Φ(x)). Hence, ū can be chosen so that H ū(x) is also positive definite.

By differentiating the first derivatives obtained in the proof of Lemma 3.4, we obtain

∂ 2 ū(x) ∂x k ∂x j = ∂ ∂x k m i=1 ∂u ∂x i (Φ(x)) ∂Φ i ∂x j (x) = = i,l ∂ 2 u ∂x l ∂x i (Φ(x)) ∂Φ l (x) ∂x k ∂Φ i (x) ∂x j + i ∂u ∂x i (Φ(x)) ∂ 2 Φ i (x) ∂x k ∂x j = = ∂ 2 u ∂x k ∂x j (Φ(x)) + ∂ 2 u ∂x k ∂x j (Φ(x))[ϕ ( x -x 0 )] 2 . .(x -x 0 ) j (x -x 0 ) k ∂ x -x 0 ∂x k ∂ x -x 0 ∂x j + + i =j,l =k ∂ 2 u ∂x l ∂x i (Φ(x)) ∂Φ l (x) ∂x k ∂Φ i (x) ∂x j + i ∂u ∂x i (Φ(x)) ∂ 2 Φ i (x) ∂x k ∂x j . Note that ∂ 2 Φ i (x) ∂x k ∂x j
depends on terms of the form (xx 0 ) j . Note also that for x such that either 0 ≤ xx 0 ≤ δ 1 or xx 0 ≥ 2δ 1 , the derivatives of ϕ are zero (because ϕ is constant) and therefore, the second derivatives of u and ū coincide. Since

{x ∈ Ω : δ 1 ≤ x -x 0 ≤ 2δ 1 }
is compact, the derivatives of u are bounded. We may then use our choice of δ < δ 1 to bound the remaining derivatives in the same way as in the proof of Lemma 3.5, so that the products involving derivatives of ϕ, and hence of Φ become small.

We can now state our main result.

Theorem 3.2. Let z 0 (p) be the AED for an economy with L goods and I traders characterized by preferences i 0 satisfying ( 1)-( 4) and endowments ω i 0 , i = 1, . . . , L. An AED z(p) is a perturbation of z 0 (p) if and only if z(p) is the AED for an economy with L goods and I consumers such that the new preferences 1 of the first consumer are perturbations of 1 0 and the new endowments ω 1 are perturbations of ω 1 0 .

Proof. We need to show that ū is strictly quasi-concave and increasing in each coordinate. We use a convolution kernel or mollifier, θ ε : Ω → R, as in Hirsch [START_REF] Hirsch | Differential Topology[END_REF], chapter 2, or Ghomi [START_REF] Ghomi | The problem of optimal smoothing of convex functions[END_REF]. The mollifier is a non-negative function which takes the value zero outside a ball of radius ε > 0 and such that

Ω θ ε = 1.
An explicit construction for such a θ ε is given in Ghomi [START_REF] Ghomi | The problem of optimal smoothing of convex functions[END_REF]. Define ū(x) = Ω u(xy)θ ε (y)dy. Theorem 2.3 in Hirsch [START_REF] Hirsch | Differential Topology[END_REF] asserts that ū and u are ε-close on compact sets. Since u is strictly quasi-concave, using Lemma 3. Since monotonicity is preserved by integration, ū is monotonous and, by Lemma 3.7, it is strictly quasi-concave. 4

Lemma 3 . 1 .

 31 Consider preferences 0 ∈ Ξ represented by a utility function u 0 ∈ C * . Let u ∈ C * , representing preferences , be a perturbation of u 0 . Then ∈ Ξ is a perturbation of 0 .3

Figure 1 :

 1 Figure 1: Graph of a function satisfying the conditions imposed on ϕ.

  7, we have, for λ ∈ [0, 1], ū(λx 1 + (1λ)x 2 ) = Ω u(λx 1 + (1λ)x 2y)θ ε (y)dy = Ω u(λ(x 1y) + (1λ)(x 2y))θ ε (y)dy > Ω [λu((x 1y) + (1λ)u(x 2y)] θ ε (y)dy = λū(x 1 ) + (1λ)ū(x 2 ).

[START_REF] Lehmann-Waffenschmidt | On the equilibrium price set of a continuous perturbation of exchange economies[END_REF] obtains a result in the same spirit for the family of one-parameterized exchange economies with an equal number of goods and consumers.

This is just the uniform norm on compact sets, i.e., the compact-open topology.

In fact, is as close to 0 as u is close to u 0 .

For concave utility functions, an analogous result was obtained by[START_REF] Kannai | Approximation of convex preferences[END_REF].

Sofia Castro and Peter Gothen acknowledge support from Centro de Matemática da Universidade do Porto (CMUP) and Fundação

Continuity of each utility in conjunction with the hypothesis that x n → x * implies that, for each fixed m ∈ N, we have u m (x n ) → u m (x * ), that is, for all δ > 0

In particular,

Let d(u, u n ) < δ/3, that is, for all y ∈ K ⊂ Ω, a compact set also containing x and x * ,

In particular, replacing y as appropriate, we have

Using (4), we obtain

Since n ≥ M, in particular for n = M, we have

which is a contradiction because x M is the unique maximum of u M . Hence x * = x, proving that demand is a continuous function of utility.

Proof. Choose utilities in C * to represent the preferences. Using Lemma 3.8, we can approximate these by C 2 utilities. Moreover, because of Lemmas 3.1 and 3.2, perturbations in utility correspond to perturbations in preferences and vice-versa. The result then follows from Theorem 3.1 and in a straightforward way from the definition of AED.

Note that the preferences represented by the C 2 utilities obtained through Theorem 3.1 can be described by utilities in C * .

In the proof of Lemma 3.8, we need the following: Lemma 3.7. Let f : R n → R be continuous and such that x > y ⇒ f (x) > f (y), where x > y means, as usual, that

Then f is strictly quasi-concave if and only if f is strictly concave on level sets, that is, if

Proof. The if part of the statement is trivial. To show the converse, suppose that (5) holds. Note that λf

. We want to show that

which holds trivially if f (x) = f (y). Let f (x) = f (y) and assume f (x) < f (y). Denote by L z = {x : f (x) = f (z)} the level set of z. By monotonicity, L x ∩ L y = ∅. Take w ∈ L y such that w > x, that is, f (w) > f(x). Consider the points on the segment connecting x to w,

By monotonicity, L wt lies between L x and L w and f (w t ) > f(x). Since y λ ∈ L wt for some t, we have f (y t ) > f(x), concluding the proof. Lemma 3.8. Let u ∈ C * be a utility representing preferences that satisfy (1)- [START_REF] Ghomi | The problem of optimal smoothing of convex functions[END_REF]. Endow C * with the uniform norm on compact sets. There exists ū which is C 2 and close to u representing preferences that satisfy (1)-( 4).