Juste Raimbault
email: juste.raimbault@polytechnique.edu

A Discrepancy-based Framework to Compare Robustness between Multi-Attribute Evaluations

Keywords: Multi-attribute Evaluation, Model-Independent Robustness, Urban System, Discrepancy

Multi-objective evaluation is a necessary aspect when managing complex systems, as the intrinsic complexity of a system is generally closely linked to the potential number of optimization objectives. However, an evaluation makes no sense without its robustness being given (in the sense of its reliability). Statistical robustness computation methods are highly dependent of underlying statistical models. We propose a formulation of a model-independent framework in the case of integrated aggregated indicators (Multi-attribute evaluation), that allows to define a relative measure of robustness taking into account data structure and indicator values. We implement and apply it to a synthetic case of urban systems based on Paris districts geography, and to real data for evaluation of income segregation for Greater Paris metropolitan area. First numerical results show the potentialities of this new method. Furthermore, its relative independence to system type and system model may position it as an alternative to classical statistical robustness methods.

Introduction

General Context

Multi-objective problems are organically linked to the complexity of underlying systems. Indeed, either in the field of Complex Industrial Systems, in the sense of engineered systems, where construction of Systems of Systems (SoS) by coupling and integration often leads to contradictory objectives [START_REF] Marler | Survey of multi-objective optimization methods for engineering[END_REF], or in the field of Natural Complex Systems, in the sense of non ingeneered physical, biological or social systems that exhibit emergence and self-organization properties, where objectives can e.g. be the result of heterogeneous interacting agents (see [START_REF] Mej Newman | Complex systems: A survey[END_REF] for a large survey of systems concerned by this approach), multi-objective optimization can be explicitly introduced to study or design the system but is often already implicitly ruling the internal mechanisms of the system. The case of socio-technical Complex Systems is particularly interesting as, following [START_REF] Haken | The face of the city is its information[END_REF], they can be seen as hybrid systems embedding social agents into "technical artifacts" (sometimes to an unexpected degree creating what Picon describes as cyborgs [START_REF] Picon | Smart cities: théorie et critique d'un idéal auto-réalisateur[END_REF]), and thus cumulate propensity to be at the origin of multi-objective issues 1 . The new notion of eco-districts [START_REF] Souami | Ecoquartiers: secrets de fabrication[END_REF] is a typical example where sustainability implies contradictory objectives. The example of transportation systems, which conception shifted during the second half of the 20th century from cost-benefit analysis to multi-criteria decision-making, is also typical of such systems [START_REF] Bavoux | Géographie des transports[END_REF]. Geographical system are now well studied from such a point of view in particular thanks to the integration of multi-objective frameworks within Geographical Information Systems [START_REF] Stephen | Integrating multi-criteria evaluation with geographical information systems[END_REF]. As for the micro-case of eco-districts, meso and macro urban planning and design may be made sustainable through indicators evaluation [START_REF] Jégou | L'évaluation par indicateurs: un outil nécessaire d'aménagement urbain durable?. réflexions à partir de la démarche parisienne pour le géographe et l'aménageur[END_REF].

A crucial aspect of an evaluation is a certain notion of its reliability, that we call here robustness.Statistics naturally include this notion since the construction and estimation of statistical models give diverse indicators of the consistence of results [START_REF] Robert | Robustness in statistics[END_REF]. The first example that comes to mind is the application of the law of large numbers to obtain the p-value of a model fit, that can be interpreted as a confidence measure of estimates. Besides, confidence intervals and beta-power are other important indicators of statistical robustness. Bayesian inference provide also measures of robustness when distribution of parameters are sequentially estimated. Concerning multi-objective optimization, in particular through heuristic algorithms (for example genetic algorithms, or operational research solvers), the notion of robustness of a solution concerns more the stability of the solution on the phase space of the corresponding dynamical system. Recent progresses have been done towards unified formulation of robustness for a multi-objective optimization problem, such as [START_REF] Deb | Introducing robustness in multi-objective optimization[END_REF] where robust Pareto-front as defined as solutions that are insensitive to small perturbations. In [START_REF] Barrico | Robustness analysis in multi-objective optimization using a degree of robustness concept[END_REF], the notion of degree of robustness is introduced, formalized as a sort of continuity of other solutions in successive neighborhood of a solution.

However, there still lack generic methods to estimate robustness of an evaluation that would be model-independent, i.e. that would be extracted from data structure and indicators but that would not depend on the method used. Some advantages could be for example an a priori estimation of potential robustness of an evaluation and thus to decide if the evaluation is worth doing. We propose here a framework answering this issue in the particular case of Multi-attribute evaluations, i.e. when the problem is made unidimensional by objectives aggregation. It is data-driven and not model-driven in the sense that robustness estimation does not depend on how indicators are computed, as soon as they respect some assumptions that will be detailed in the following.

Proposed Approach

Objectives as Spatial Integrals We assume that objectives can be expressed as spatial integrals, so it should apply to any territorial system and our application cases are urban systems. It is not that restrictive in terms of possible indicators if one uses suitable variables and integrated kernels : in a way analog to the method of geographically weighted regression [START_REF] Brunsdon | Geographically weighted regression[END_REF], any spatial variable can be integrated against regular kernels of variable size and the result will be a spatial aggregation which sense depends on kernel size. The example we use in the following such as conditional means or sums suit well the assumption. Even an already spatially aggregated indicator can be interpreted as a spatial indicator by using a Dirac distribution on the centroid of the corresponding area.

Linearly Aggregated Objectives A second assumption we make is that the multiobjective evaluation is done through linear aggregation of objectives, i.e. that we are tackling a multi-attribute optimization problem. If (q i (x)) i are values of objectives functions, then weights (w i ) i are defined in order to build the aggregated decision-making function q(x) = i w i q i (x), which value determines then the performance of the solution. It is analog to aggregated utility techniques in economics and is used in many fields. The subtlety lies in the choice of weights, i.e. the shape of the projection function, and various approaches have been developed to find weights depending on the nature of the problem. Recent work [START_REF] Melissa | Robustness and sensitivity of weighting and aggregation in constructing composite indices[END_REF] proposed to compare robustness of different aggregation techniques through sensitivity analysis, performed by Monte-Carlo simulations on synthetic data. Distribution of biases where obtained for various techniques and some showed to perform significantly better than others. Robustness assessment still depended on models used in that work.

The rest of the paper is organized as follows : section 2 describes intuitively and mathematically the proposed framework ; section 3 then details implementation, data collection for case studies and numerical results for an artificial intra-urban case and a metropolitan real case ; section 4 finally discuss limitations and potentialities of the method.

Framework Description

Intuitive Description

We describe now the abstract framework allowing theoretically to compare robustnesses of evaluations of two different urban systems. Our framework is a generalization of an empirical method proposed in [START_REF] Ali | Les eco-quartiers lus par la mobilité : vers une évaluation intégrée[END_REF] besides a more general benchmarking study on indicator sense and relevance in a sustainability context. Intuitively, it relies on empirical base resulting from the following axioms :

-Urban systems can be seen from the information available, i.e. raw data describing the system. As a data-driven approach, this raw data is the basis of our framework and robustness will be determined by its structure. -From data are computed indicators (objective functions). We assume that a choice of indicators is a will to translate particular aspects of the system, that is to capture a realization of an "urban fact" (fait urbain) in the sense of Mangin [15] -a sort of stylized fact in terms of processes and mechanisms, having various realizations on spatially distinct systems, depending on each precise context. -Given many systems and associated indicators, a common space can be built to compare themselves. In that space, data will represent more or less well real systems, depending e.g. on initial scale, precision of data, missing data. We precisely capture that through the notion of point cloud discrepancy, which is a mathematical tool coming from sampling theory which express how a dataset is well distributed in the space it is embedded in [START_REF] Dick | Digital nets and sequences: Discrepancy Theory and Quasi-Monte Carlo Integration[END_REF]. Coupling discrepancies with appropriate weights depending on indicator importance allows to introduce a relative ratio of robustness between two evaluations.

Synthesizing these requirements, we propose a notion of Robustness of an evaluation that captures both, by combining data reliability with relative importance, 1. Missing Data : an evaluation based on more refined datasets will naturally be more robust. 2. Indicator importance : indicators with more relative influence will weight more on the total robustness.

Formal Description

Indicators Let (S i ) 1≤i≤N be a finite number of geographically disjoints territorial systems, that we assume described through raw data and intermediate indicators, yielding

S i = (X i , Y i ) ∈ X i × Y i with X i = k X i,k such that each subspace contain real matrices : X i,k = R n X i,k p X i,k
(the same holding for Y i ). We also define an ontological index function I X (i, k) (resp. I Y (i, k)) taking integer values which coincide if and only if the two variables have the same ontology in the sense of [START_REF] Livet | Ontology, a mediator for agent-based modeling in social science[END_REF], i.e. they are supposed to represent the same real object. We distinguish "raw data" X i from which indicators are computed via explicit deterministic functions, from "intermediate indicators" Y i that are already integrated and can be e.g. outputs of elaborated models simulating some aspects of the urban system.

We define the partial characteristic space of the "urban fact" by

(X , Y) = def Xc × Ỹc =   X i,k ∈D X R p X i,k   ×   Y i,k ∈D Y R p Y i,k   (1)
, with

D X = {X i,k |I(i, k) distincts, n X i,k maximal} (the same holding for Y i ).
It is indeed the abstract space on which indicators are integrated. The indices c introduced as a definition here correspond to different indicators across all systems. This space is the minimal space common to all systems allowing a common definition for indicators on each.

Let X i,c be the data canonically projected in the corresponding subspace, well defined for all i and all c. We make the key assumption that all indicators are computed by integration against a certain kernel, i.e. that for all c, there exists H c space of real-valued functions on ( Xc , Ỹc ), such that for all h ∈ H c :

1. h is "enough" regular (tempered distributions e.g.) 2. q c = ( Xc, Ỹc) h is a function describing the "urban fact" (the indicator in itself)

Typical concrete example of kernels can be :

-A mean of rows of X i,c is computed with h(x) = x • f i,c (x) where f i,c
is the density of the distribution of the assumed underlying variable. -A rate of elements respecting a given condition C, h(x) = f i,c (x)χ C(x) -For already aggregated variables Y, a Dirac distribution allows to express them also as a kernel integral.

Aggregation Weighting objectives in multi-attribute decision-making is indeed the crucial point of the processes, and numerous methods are available (see [START_REF] Wang | Review on multi-criteria decision analysis aid in sustainable energy decision-making[END_REF] for a review for the particular case of sustainable energy management). Let define weights for the linear aggregation. We assume the indicators normalized, i.e. q c ∈ [0, 1], for a more simple construction of relative weights. For i, c and h c ∈ H c given, the weight w i,c is simply constituted by the relative importance of the indicator w L i,c = qi,c c qi,c where qi,c is an estimator of q c for data X i,c (i.e. the effectively calculated value). Note that this step can be extended to any sets of weight attributions, by taking for example wi,c = w i,c • w i,c if w are the weights attributed by the decision-maker. We focus here on the relative influence of attributes and thus choose this simple form for weights.

Robustness Estimation

The scene is now set up to be able to estimate the robustness of the evaluation done through the aggregated function. Therefore, we apply an integral approximation method similar to methods introduced in [START_REF] Varet | Développement de méthodes statistiques pour la prédiction d'un gabarit de signature infrarouge[END_REF], since the integrated form of indicators indeed brings the benefits of such powerful theoretical results.

Let

X i,c = (X i,c,l ) 1≤l≤ni,c and
D i,c = Discp Xc,∞ (X i,c )
the discrepancy of data points cloud [START_REF] Niederreiter | Discrepancy and convex programming[END_REF].

With h ∈ H c , we have the upper bound on the integral approximation error

h c - 1 n i,c l h c (X i,c,l ) ≤ K • |||h c ||| • D i,c
where K is a constant regarding data points and objective function. It directly yields

w i,c h c - 1 n i,c l w i,c h c (X i,c,l ) ≤ K c |w i,c | |||h c ||| • D i,c
Assuming the error reasonably realized ("worst case" scenario for knowledge of the theoretical value of aggregated function), we take this upper bound as an approximation of its magnitude. Furthermore, taking normalized indicators implies |||h c ||| = 1. We propose then to compare error bounds between two evaluations. They depend only on data distribution (equivalent to statistical robustness) and on indicators chosen (sort of ontological robustness, i.e. do the indicators have a real sense in the chosen context and do their values make sense), and are a way to combine these two type of robustnesses into a single value.

We thus define a robustness ratio to compare the robustness of two evaluations by

R i,i = c w i,c • D i,c c w i ,c • D i ,c (2)
The intuitive sense of this definition is that one compares robustness of evaluations by comparing the highest error done in each based on data structure and relative importance.

By taking then an order relation on evaluations by comparing the position of the ratio to one, it is obvious that we obtain a complete order on all possible evaluations. This ratio should theoretically allow to compare any evaluation of an urban system. To keep an ontological sense to it, it should be used to compare disjoints sub-systems with a reasonable proportion of indicators in common, or the same sub-system with varying indicators. Note that it provides a way to test the influence of indicators on an evaluation by analyzing the sensitivity if the ratio to their removal. On the contrary, finding a "minimal" number of indicators each making the ratio strongly vary should be a way to isolate essential parameters ruling the sub-system.

Results

Implementation Preprocessing of geographical data is made through QGIS [START_REF] Dt Qgis | Quantum gis geographic information system[END_REF] for performance reasons. Core implementation of the framework is done in R [START_REF]R language definition[END_REF] for the flexibility of data management and statistical computations. Furthermore, the package DiceDesign [START_REF] Franco | Dicedesign-package[END_REF] written for numerical experiments and sampling purposes, allows an efficient and direct computation of discrepancies. Last but not least, all source code is openly available on the git repository of the project2 for reproducibility purposes [START_REF] Ram | Git can facilitate greater reproducibility and increased transparency in science[END_REF].

Implementation on Synthetic Data

We propose in a first stage to illustrate the implementation with an application to synthetic data and indicators, for intra-urban quality indicators in the city of Paris.

Data Collection

We base our virtual case on real geographical data, in particular for arrondissements of Paris. We use open data available through the OpenStreetMap project [START_REF] Bennett | OpenStreetMap[END_REF] that provides accurate high definition data for many urban features. We use the street network and position of buildings within the city of Paris3 . Limits of arrondissements, used to overlay and extract features when working on single districts, are also extracted from the same source. We use centroids of buildings polygons, and segments of street network. Dataset overall consists of around 200k building features and 100k road segments.

Virtual Cases We work on each district of Paris (from the 1st to the 20th) as an evaluated urban system. We construct random synthetic data associated to spatial features, so each district has to be evaluated many time to obtain mean statistical behavior of toy indicators and robustness ratios. The indicators chosen need to be computed on residential and street network spatial data. We implement two mean kernels and a conditional mean to show different examples, linked to environmental sustainability and quality of life, that are required to be maximized. Note that these indicators have a real meaning but no particular reason to be aggregated, they are chosen here for the convenience of the toy

φ = 1 - 1 |s ∈ A(a)| • s∈A(a) ϕ(s) • l(s) max (l(s))
-Relative length of pedestrian streets p, computed through a randomly uniformly generated dummy variable adjusted to have a fixed global proportion of segments that are pedestrian.

As synthetic data are stochastic, we run the computation for each district N = 50 times, what was a reasonable compromise between statistical convergence and time required for computation. Table 1 shows results (mean and standard deviations) of indicator values and robustness ratio computation. Obtained standard deviation confirm that this number of repetitions give consistent results. Indicators obtained through a fixed ratio show small variability what may a limit of this toy approach. However, we obtain the interesting result that a majority of districts give more robust evaluations than 1st district, what was expected because of the size and content of this district : it is indeed a small one with large administrative buildings, what means less spatial elements and thus a less robust evaluation following our definition of the robustness.

Application to a Real Case : Metropolitan Segregation

The first example was aimed to show potentialities of the method but was purely synthetic, hence yielding no concrete conclusion nor implications for policy. We propose now to apply it to real data for the example of metropolitan segregation.

Data We work on income data available for France at an intra-urban level (basic statistical units IRIS) for the year 2011 under the form of summary statistics (deciles if the area is populated enough to ensure anonymity), provided by IN-SEE 4 . Data are associated with geographical extent of statistical units, allowing computation of spatial analysis indicators.

Indicators We use here three indicators of segregation integrated on a geographical area. Let assume the area divided into covering units S i for 1 ≤ i ≤ N with centroids (x i , y i ). Each unit has characteristics of population P i and median income X i . We define spatial weights used to quantify strength of geographical interactions between units i, j, with d ij euclidian distance between centroids :

w ij = PiPj ( k P k ) 2 • 1 dij if i = i and w ii = 0.
The normalized indicators are the following -Spatial autocorrelation Moran index, defined as weighted normalized covariance of median income by 1. Numerical results of simulation for each district with N = 50 repetitions. Each toy indicator value is given by mean on repetitions and associated standard deviation. Robustness ratio is computed relative to first district (arbitrary choice). A ratio smaller than 1 means that integral bound is smaller for upper district, i.e. that evaluation is more robust for this district. Because of the small size of first district, we expected a majority of district to give ratio smaller than 1, what is confirmed by results, even when adding standard deviations.

ρ = N ij w ij • ij w ij X i -X X j -X i X i -X 2 -Dissimilarity index (close to Moran but integrating local dissimilarities rather than correlations), with Xi = Xi-min(X k ) max(X k )-min(X k ) d = 1 ij w ij ij w ij Xi -Xj Arrdt < dw > ±σ( dw) < φ > ±σ( φ) < p > ±σ(p) Ri,
-Complementary of the entropy of income distribution that is a way to capture global inequalities

ε = 1 + 1 log(N ) i X i k X k • log X i k X k
Numerous measures of segregation with various meanings and at different scales are available, as for example at the level of the unit by comparison of empirical wage distribution with a theoretical null model [START_REF] Louf | Patterns of residential segregation[END_REF]. The choice here is arbitrary in order to illustrate our method with a reasonable number of dimensions.

= N/ jwij • j w i j(X j -X)(X i -X) i (X i -X) 2
. The most segregated areas coincide with the richest and the poorest, suggesting an increase of segregation in extreme situations.

Results

We apply our method with these indicators on the Greater Paris area, constituted of four départements that are intermediate administrative units. The recent creation of a new metropolitan governance system [START_REF] Gilli | Paris, métropole hors les murs: aménager et gouverner un Grand Paris[END_REF] underlines interrogations on its consistence, and in particular on its relation to intermediate spatial inequalities. We show in Fig. 1 maps of spatial distribution of median income and corresponding local index of autocorrelation. We observe the well-known West-East opposition and district disparities inside Paris as they were formulated in various studies, such as [START_REF] Guérois | La dynamique spatio-temporelle des prix immobiliers à différentes échelles: le cas des appartements anciens à paris (1990-2003[END_REF] through the analysis of real estate transactions dynamics. We then apply our framework to answer a concrete question that has implications for urban policy : how are the evaluation of segregation within different territories sensitive to missing data ? To do so, we proceed to Monte Carlo simulations (50 repetitions) during which a fixed proportion of data is randomly removed, and the corresponding robustness index is evaluated. Simulations are done on each department separately, each time relatively to the robustness of the evaluation of full Greater Paris. Results are shown in Fig. 2. One area (92) present a slightly better robustness than the reference, what could be explained by its homogeneity and thus highly fiable segregation values. Implications for policy that can be drawn are for example direct comparisons between areas : a loss of 8% of information on Paris area yields an evaluation as robust as with all data in 93 area. The latter being a deprived area, the inequality is increased by this relative lower quality of statistical information. The study of standard deviations suggest further investigations as different response regimes to data removal seem to exist. For each department, Monte Carlo simulations (N=50 repetitions) are used to determine the impact of missing data on robustness of segregation evaluation. Robustness ratios are all computed relatively to full metropolitan area with all available data. Quasi-linear behavior translates an approximative linear decrease of discrepancy as a function of data size. The convergence of richest (75,92) and poorest departments suggest the correction to linear behavior being driven be segregation patterns. Right. Corresponding standard deviations of robustness ratios. Different regimes (in particular 94 against other) unveil phase transitions at different levels of missing data, meaning that the evaluation in 94 is from this point of view more sensitive to missing data.

Discussion

Applicability to Real situations

Implications for Decision-making The application of our method to concrete decision-making can be thought in different ways. First in the case of a comparative multi-attribute decision process, such as the determination of a transportation corridor, the identification of territories on which the evaluation may be flawed (i.e. has a poor relative robustness) could allow a more refined focus on these and a corresponding revision of datasets or an adapted revision of weights.

In any case the overall decision-making process should be made more reliable.

A second direction lays in the spirit of the real application we have proposed, i.e. the sensitivity of evaluation to various parameters such as missing data. If a decision appears as reliable because data have few missing points, but the evaluation is very sensitive to it, one will be more careful in the interpretation of results and taking the final decision.

Integration Within Existing Frameworks The applicability of the method on real cases will directly depend on its potential integration within existing framework. Beyond technical difficulties that will surely appear when trying to couple or integrate implementations, more theoretical obstacles could occur, such as fuzzy formulations of functions or data types, consistency issues in databases, etc. Such multi-criteria framework are numerous. Further interesting work would be to attempt integration into an open one, such as e.g. the one described in [START_REF] Tivadar | Oasis-un outil d'analyse de la ségrégation et des inégalités spatiales[END_REF] which calculates various indices of urban segregation, as we have already illustrated the application on metropolitan segregation indexes.

Availability of Raw Data

In general, sensitive data such as transportation questionnaires, or very fine granularity census data are not openly available but provided already aggregated at a certain level (for instance French Insee Data are publicly available at basic statistical unit level or larger areas depending on variables and minimal population constraints, more precise data is under restricted access). It means that applying the framework may imply complicated data research procedure, its advantage to be flexible being thus reduced through additional constraints.

Validity of Theoretical Assumptions

A possible limitation of our approach is the validity of the assumption formulating indicators as spatial integrals. Indeed, many socio-economic indicators are not necessarily depending explicitly on space, and trying to associate them with spatial coordinates may become a slippery slope (e.g. associate individual economic variables with individual residential coordinates will have a sense only if the use of the variable has a relation with space, otherwise it is a non-legitimate artifact). Even indicators which have a spatial value may derive from non-spatial variables, as [START_REF] Kwan | Space-time and integral measures of individual accessibility: a comparative analysis using a point-based framework[END_REF] points out concerning accessibility, when opposing integrated accessibility measures with individual-based non necessarily spatial-based (e.g. individual decisions) measures. Constraining a theoretical representation of a system to fit a framework by changing some of its ontological properties (always in the sense of real meaning of objects) can be understood as a violation of a fundamental rule of modeling and simulation in social science given in [START_REF] Banos | Pour des pratiques de modélisation et de simulation libérées en géographie et shs[END_REF], that is that there can be an universal "language" for modeling and some can not express some systems, having for consequence misleading conclusion due to ontology breaking in the case of an overly constrained formulation.

Framework Generality

We argue that the fundamental advantage of the proposed framework is its generality and flexibility, since robustness of the evaluations are obtained only through data structure if ones relaxes constraints on the value of weight. Further work should go towards a more general formulation, suppressing for example the linear aggregation assumption. Non-linear aggregation functions would require however to present particular properties regarding integral inequalities. For example, similar results could search in the direction of integral inequalities for Lipschitzian functions such as the one-dimensional results of [START_REF] Ss Dragomir | The ostrowski's integral inequality for lipschitzian mappings and applications[END_REF].

Conclusion

We have proposed a model-independent framework to compare the robustness of multi-attribute evaluations between different urban systems. Based on data discrepancy, it provide a general definition of relative robustness without any assumption on model for the system, but with limiting assumptions that are the need of linear aggregation and of indicators being expressed through spatial kernel integrals. We propose a toy implementation based on real data for the city of Paris, numerical results confirming general expected behavior, and an implementation on real data for income segregation on Greater Paris metropolitan areas, giving possible insights into concrete policy questions. Further work should be oriented towards sensitivity analysis of the method, application to other real cases and theoretical assumptions relaxation, i.e. the relaxation of linear aggregation and spatial integration.

-

  model and the generation of synthetic data. With a ∈ {1 . . . 20} the number of the district, A(a) corresponding spatial extent, b ∈ B building coordinates and s ∈ S street segments, we take -Complementary of the average daily distance to work with car per individual, approximated by, with n cars (b) number of cars in the building (randomly generated by associated of cars to a number of building proportional to motorization rate α m 0.4 in Paris), d w distance to work of individuals (generated from the building to a uniformly generated random point in spatial extent of the dataset), and d max the diameter of Paris area, dw = 1 -1 |b ∈ A(a)| • b∈A(a) n cars (b) • d w d max Complementary of average car flows within the streets in the district, approximated by, with ϕ(s) relative flow in street segment s, generated through the minimum of 1 and a log-normal distribution adjusted to have 95% of mass smaller than 1 what mimics the hierarchical distribution of street use (corresponding to betweenness centrality), and l(s) segment length,

Fig. 1 .

 1 Fig.1. Maps of Metropolitan Segregation. Maps show yearly median income on basic statistical units (IRIS) for the three departments constituting mainly the Great Paris metropolitan area, and the corresponding local Moran spatial autocorrelation index, defined for unit i as ρi = N/ jwij • j w i j(X j -X)(X i -X)

Fig. 2 .

 2 Fig.2. Sensitivity of robustness to missing data. Left. For each department, Monte Carlo simulations (N=50 repetitions) are used to determine the impact of missing data on robustness of segregation evaluation. Robustness ratios are all computed relatively to full metropolitan area with all available data. Quasi-linear behavior translates an approximative linear decrease of discrepancy as a function of data size. The convergence of richest (75,92) and poorest departments suggest the correction to linear behavior being driven be segregation patterns. Right. Corresponding standard deviations of robustness ratios. Different regimes (in particular 94 against other) unveil phase transitions at different levels of missing data, meaning that the evaluation in 94 is from this point of view more sensitive to missing data.

We design by Multi-Objective Evaluation all practices including the computation of multiple indicators of a system (it can be multi-objective optimization for system design or multi-objective evaluation of an existing system ; our examples being in the second case but the overall method staying rather generic).

at https://github.com/JusteRaimbault/RobustnessDiscrepancy

for which the third-party website https://mapzen.com/metro-extracts/ provides already packaged extracts as for many metropolitan areas of the world

http://www.insee.fr