Myriam Blanchin
email: myriam.blanchin@univ-nantes.fr

Alice Guilleux

Jean-Benoit Hardouin

Véronique Sébille

Comparison of SEM, IRT and RMT-based methods for response shift detection at item level: a simulation study

Keywords: Structural equation modelling, Item response theory, Rasch models, response shift, item level

come

Comparison of SEM, IRT and RMT-based methods for response shift detection at item level: a simulation study

Introduction

The growing incorporation of patients' perspective in clinical trials and cohort studies has largely increased the use of longitudinal Patient-Reported Outcome (PRO) measures in which different items are usually grouped in several dimensions (physical, emotional, social…). The report of patients' experience is essential to understand the impact of disease burden and treatment over the course of illness and can be crucial for shared clinical decision-making and in daily clinical practice [START_REF] Nipp | Harnessing the Power of Patient-Reported Outcomes in Oncology[END_REF][START_REF] Kluetz | Incorporating the patient experience into regulatory decision making in the USA, Europe, and Canada[END_REF]. When assessing change in PRO data, longitudinal measurement invariance is usually assumed suggesting that patients respond consistently to the PRO instrument and that patients' item responses are directly comparable over time, which can be questioned. Indeed, the meaning in patients' self-evaluations of the target construct is likely to change over time and this change may cause longitudinal non-invariance of the measurement model parameters [START_REF] Widaman | Factorial Invariance within Longitudinal Structural Equation Models: Measuring the Same Construct across Time[END_REF][START_REF] Verhagen | Longitudinal measurement in health-related surveys. A Bayesian joint growth model for multivariate ordinal responses[END_REF][START_REF] Pastor | Longitudinal Rasch Modeling in the Context of Psychotherapy Outcomes Assessment[END_REF]. On one hand, this change in meaning, known as response shift (RS) [START_REF] Sprangers | Integrating response shift into health-related quality of life research: a theoretical model[END_REF] in health sciences, is a concern as it can bias the estimation of longitudinal change in PRO data. On the other hand, RS is also viewed as a change [START_REF] Mcclimans | Philosophical perspectives on response shift[END_REF] that should be identified and quantified because of its possible link with patients' adaptation processes [START_REF] Sprangers | Integrating response shift into health-related quality of life research: a theoretical model[END_REF][START_REF] Postulart | Response shift and adaptation in chronically ill patients[END_REF] triggered by the disease itself, treatments or interventions such as educational programs or support for disease selfmanagement. Three types of response shifts have been defined [START_REF] Sprangers | Integrating response shift into health-related quality of life research: a theoretical model[END_REF] Until recently, all statistical methods for RS detection in longitudinal PRO data were developed and applied at dimension-level regarding the relationship between dimensions and the construct of interest. Amongst the statistical methods proposed to detect and account for dimension-level RS, the Oort's procedure [START_REF] Oort | Using structural equation modeling to detect response shifts and true change[END_REF] based on structural equation modelling (SEM) is now applied in a majority of studies (in 20/47 articles in a recent scoping review of response shift methods [START_REF] Sajobi | Scoping review of response shift methods: current reporting practices and recommendations[END_REF]). The widespread application of Oort's procedure is probably related to its advantages to not only detect the different types of RS but also to quantify and account for RS in estimating the longitudinal change in PRO, if appropriate.

Lately, the importance and significance of RS at item-level (relationship between itemlevel responses and the construct of interest within a dimension) was raised [START_REF] Schwartz | Introduction to special section on response shift at the item level[END_REF]. As item-level RS began to be considered as providing interesting and complementary insight into the understanding of RS, Oort's procedure was applied at item-level in different ways according to the technical aspects of applying SEM models to dichotomous or ordinal data [START_REF] Vanier | Overall performance of Oort's procedure for response shift detection at item level: a pilot simulation study[END_REF][START_REF] Nolte | On the validity of measuring change over time in routine clinical assessment: a close examination of item-level response shifts in psychosomatic inpatients[END_REF][START_REF] Gandhi | An itemlevel response shift study on the change of health state with the rating of asthmaspecific quality of life: a report from the PROMIS® Pediatric Asthma Study[END_REF][START_REF] Verdam | Using structural equation modeling to detect response shifts and true change in discrete variables: an application to the items of the SF-36[END_REF][START_REF] Ahmed | Minimal evidence of response shift in the absence of a catalyst[END_REF]. At the same time, statistical methods based on item response theory (IRT) [START_REF] Guilleux | RespOnse Shift ALgorithm in Item response theory (ROSALI) for response shift detection with missing data in longitudinal patient-reported outcome studies[END_REF][START_REF] Anota | Item response theory and factor analysis as a mean to characterize occurrence of response shift in a longitudinal quality of life study in breast cancer patients[END_REF][START_REF] Blanchin | The Guttman errors as a tool for response shift detection at subgroup and item levels[END_REF] were also applied as this approach is naturally suitable for item-level detection of RS. In addition to SEM (Oort's procedure) and IRT approaches, methods coming from Rasch Measurement Theory (RMT) [START_REF] Fischer | Rasch models: foundations, recent developments, and applications[END_REF] could also be valuable at itemlevel. Indeed, RMT models possess the specific objectivity property that allows obtaining consistent estimations of the parameters associated to the latent trait independently from the items used for these estimations [START_REF] Andrich | Rating scales and Rasch measurement[END_REF]. Consequently, as previously shown in simulation studies [START_REF] De Bock | Rasch-family models are more valuable than score-based approaches for analysing longitudinal patient-reported outcomes with missing data[END_REF][START_REF] Hamel | Biases and Power for Groups Comparison on Subjective Health Measurements[END_REF][START_REF] De Bock | Assessment of score-and Rasch-based methods for group comparison of longitudinal patientreported outcomes with intermittent missing data (informative and non-informative)[END_REF], unbiased estimation of the latent trait can be obtained even when some items are missing, in an ignorable way or not. Hence, RMT models could be more appropriate in case of missing data.

Although the performance of the Oort's procedure based on SEM was evaluated once [START_REF] Vanier | Overall performance of Oort's procedure for response shift detection at item level: a pilot simulation study[END_REF] in a simulation study in the context of dichotomous data, the lack of evaluation of SEM and IRT approaches for RS detection at item-level has been highlighted [START_REF] Sajobi | Scoping review of response shift methods: current reporting practices and recommendations[END_REF]. Since now, these approaches were mostly published and applied on real data. Hence, it seems important to assess their performance in order to know if the results of RS detection of SEM and IRT approaches are trustworthy. It is also of interest to evaluate if these approaches are really able to detect RS when it occurs and to distinguish the different types of RS. In this article, we present in detail three approaches for item-level RS detection: Oort's procedure (SEM), the ROSALI algorithm (IRT) and propose a version of ROSALI based on RMT. The aim of this study is to compare and evaluate the performance of SEM, IRT and RMT approaches for item-level RS detection using a simulation study.

Methods

Original Oort's Procedure

All statistical methods evaluated in this study are based on the algorithm of the 4-step Oort's procedure initially proposed at dimension level [START_REF] Oort | Using structural equation modeling to detect response shifts and true change[END_REF] for testing measurement invariance between two times of measurement using SEM. In SEM, the different types of RS are operationalized as change in patterns of factor loadings (reconceptualization), in values of factor loadings (reprioritization, abbreviated as RP), item intercepts (uniform recalibration, abbreviated as UR) and residual variances (non-uniform recalibration, abbreviated as NUR). Reconceptualization of one target construct is only appraisable in comparison with other constructs and its detection should therefore take place in a multidimensional context. This study pertains to a unidimensional setting and reconceptualization will not be assessed. Focusing on reprioritization and uniform or nonuniform recalibration, RS parameters are then factor loadings, intercepts and residual variances at both times of measurement.

In step 1 of the Oort's procedure, an appropriate measurement model (longitudinal SEM, Oort's Procedure at Item Level (OPIL) The Oort's procedure at item-level (OPIL) method follows the same 4 previously described steps but is based on longitudinal SEM models modeling the relationship between item responses of a single dimension and a latent variable. Let be the vector of observed item responses of patient i (i=1,…,N) to the J items of the dimension (j=1,…,J) at time t={1,2}, the measurement model (model 1-OPIL) can be written as follows:

where is the vector of intercepts at time t, is the matrix of factor loadings at time t, is the unobserved latent variable and is the vector of unobserved residual errors of patient i at time t ( ). The latent variable and the residual errors are assumed to be uncorrelated. In steps 2, 3 and 4, RS constraints (equality across time) on a given item can be imposed on: residual variances for NUR, item intercepts for UR ( vector components) and factor loadings ( matrix components) for RP.

Parameters of all SEM models in OPIL are estimated by maximum likelihood, assuming continuous and normally distributed item responses . The mean and variance of the latent variable at time 1 are constrained to 0 and 1 for identifiability in all steps. In addition, the mean and variance of the latent variable at time 2 are constrained to 0 and 1 for identifiability at step 1 only. -Starting from the model 2, constraints on residual variances are relaxed item-byitem (NUR detection) producing different models named models 3. In each model 3, the relevance of relaxing the constraint is evaluated using a LRT (model 3 vs model 2). The retained model 3 is the one with the most significant LRT (NUR has been detected and will be accounted for on the item on which it was evidenced). Starting from the retained model 3, constraints on residual variances are again relaxed item-by-item on the remaining items until no more LRT are significant.

-Starting from the last model 3 accounting for all detected NUR, constraints on intercepts (UR detection) are relaxed item-by-item iteratively, retaining the most significant LRT and updating model 3 at each iteration of UR detection until no more LRT are significant.

-Starting from the last model 3 accounting for all detected NUR and UR, similarly, RP detection is performed by relaxing constraints on factor loadings one-by-one.

As multiple models are compared at each iteration of step 3, a Bonferroni correction ( 26) is applied to adjust the type I error rate for multiple testing.

ROSALI-IRT

The method called the RespOnse Shift ALgorithm for Item Response Theory (ROSALI-IRT) [START_REF] Guilleux | RespOnse Shift ALgorithm in Item response theory (ROSALI) for response shift detection with missing data in longitudinal patient-reported outcome studies[END_REF] relies on an IRT model, the longitudinal generalized partial credit model (GPCM) [START_REF] Muraki | A Generalized Partial Credit Model: Application of an EM Algorithm[END_REF]. The polytomous items are assumed to have m j +1 response categories from 0 to m j. The measurement model (model ) for each item also applying a Bonferroni correction for multiple testing.

ROSALI-RMT

Quite naturally, ROSALI-IRT can be adapted using longitudinal partial credit models (PCM) [START_REF] Fischer | An extension of the partial credit model with an application to the measurement of change[END_REF][START_REF] Masters | A rasch model for partial credit scoring[END_REF] in order to detect RS based on Rasch Measurement Theory (RMT). In the ROSALI-RMT method, the measurement model (model 1, ROSALI-RMT) between two times of measurement (t={1,2}) can be written as follows:

where , the latent trait level of patient i at time t, a realization of the random variable Θ. is assumed to be normally distributed with mean and covariance matrix . is the item difficulty and the change in item difficulties between time 1 and time 2 for answer category p of item j (1≤p≤m j ).

Uniform or non-uniform recalibration is operationalized as change in item difficulties widespread questionnaires evaluating health-related quality of life [START_REF] Ware | The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection[END_REF][START_REF] Aaronson | The European Organization for Research and Treatment of Cancer QLQ-C30: a qualityof-life instrument for use in international clinical trials in oncology[END_REF] or anxiety [START_REF] Zigmond | The Hospital Anxiety and Depression Scale[END_REF] for example (J=4 or 7 items, M=4 or 7 answer categories).

Response shift

Item difficulties were chosen to reflect the situation where the questionnaire is suitable for a population with a normally distributed latent trait (item difficulties regularly spaced on the latent trait continuum, overlaid distributions of item and latent trait). For the first answer category of each item, item difficulties were drawn from the percentiles of the item distribution defined as a standard normal distribution. For other answer categories (p>1), item difficulties were regularly spaced from the first item difficulty with . All item difficulties are then centered on the mean of item difficulties so that item difficulties are centered on the same mean as the latent trait distribution. All discrimination power parameters were equal to 1 at first measurement occasion ( .

When no RS was simulated, all discrimination power parameters and item difficulties As the ROSALI-RMT method is not able to detect RP, the analysis of datasets affected by this type of RS was considered as an assessment of the robustness of this method to model deviation. For ROSALI-RMT, the rates of most flexible and flexible detection of RS were computed but the rate of perfect detection of RS when RP was simulated cannot be assessed. However, the proportion of the different possible combinations of the types of RS detected at step 3 (uniform only, non-uniform only, both uniform and non-uniform or no recalibration) among the datasets where RS was accounted for on the correct items only were assessed.

Data simulation and analyses were performed using Stata.

Results

Results for lower values of reprioritization ( ) and of non-uniform recalibration (shift between -0.5 and 0.5) are not presented here due to bad performances of all approaches for these values.

Response shift detection when no RS was simulated Rates of false detection of RS for each method are presented in Table 1. The rates of false detection of RS based on the LRT range between 5% and 23% for OPIL, between 48% and 95% for ROSALI-IRT and between 4% and 7% for ROSALI-RMT. Therefore, only ROSALI-RMT shows rates of false detection based on the LRT close to the expected value of 5%. OPIL performs well for small values of the simulation parameters whereas ROSALI-IRT always shows dramatically high rates of false detection. For OPIL and ROSALI-IRT, rates of false detection increase with the sample size (N), number of items (J) and number of answer categories (M).

The rates of false detection of RS in model 4 (final model accounting for detected response shifts) range between 4% and 22% for OPIL, between 39% and 81% for ROSALI-IRT and between 1% and 3% for ROSALI-RMT. For all methods, the rates of false detection based on model 4 are lower than the rates based on the LRT. Thus, for some datasets where overall presence of RS was concluded based on the LRT at step 2, no RS was detected on any item during step 3 correcting the wrong decision made by the LRT. To quantify the correction at step 3, the ratio between the difference in rates based on the LRT and on model 4 over the rates based on the LRT were computed. For example, OPIL shows a correction of 20% for N=100, J=4 and M=4 meaning that RS 20

was not considered in model 4 in 20% of the datasets where overall presence of RS was wrongly detected by the LRT. The wrong decision based on the LRT was very often corrected in ROSALI-RMT (between 40% and 91%), less often in ROSALI-IRT (between 17% and 31%) and quite rarely in OPIL (between 0 and 20%). However, presenting the different rates of RS detection for ROSALI-IRT allows giving some insight on steps that are problematic in the procedure (determining the items affected by RS, distinguishing between the types of RS).

Datasets with one item affected by uniform or non-uniform recalibration

Rates of most flexible, flexible and perfect detection of RS for each method are presented in Table 2.

 Most flexible RS detection rates

The rates of most flexible detection range between 48% and 100% for OPIL, between 97% and 100% for ROSALI-IRT and between 74% and 100% for ROSALI-RMT. In most of the cases, all methods were able to identify at least the item affected by RS. The lowest rates of most flexible RS detection were observed for small sample sizes (N=100)

and small number of answer categories (M=4).

In cases where non-uniform recalibration was simulated, ROSALI-IRT has the highest most flexible rates and OPIL the lowest whereas in cases where uniform recalibration was simulated ROSALI-IRT also has the highest rates whereas OPIL and ROSALI-RMT have nearly the same. Rates of most flexible RS detection in case of uniform recalibration are generally quite close to non-uniform recalibration rates for ROSALI-IRT and ROSALI-RMT. However, for OPIL non-uniform recalibration rates are remarkably smaller than uniform recalibration (differences range from 34% to 40%) when the sample size and the number of answer categories are small (N=100, M=4). This effect was also observed for ROSALI-RMT but to a lesser extent when N=100 (differences range from 10% to 19%).

 Flexible RS detection rates

Contrary to the rate of most flexible RS detection where RS could have been accounted

for on several items including the correct one, the rate of flexible RS detection indicates the proportion of datasets where RS was accounted for on the correct item only. The rates of flexible RS detection range between 28% 83% for OPIL, between 21% and 60%

for ROSALI-IRT and between 71% and 97% for ROSALI-RMT. In almost all cases, ROSALI-RMT performs better than OPIL followed by ROSALI-IRT. Therefore, a large difference was observed between the rates of most flexible and flexible RS detection for ROSALI-IRT (between 40% and 76%) indicating that this method very often concluded to the presence of RS on the correct item plus other items when RS was simulated on only one item. This effect was also observed for OPIL but to a lesser extent (difference from 13% to 56%).The differences between most flexible and flexible RS detection rates for ROSALI-IRT increase with J and M.

As for rates of most flexible RS detection, rates of flexible RS detection in case of uniform recalibration are generally close to rates in case of non-uniform recalibration, except for OPIL when N=100 and M=4 and for ROSALI-RMT when N=100.

 Perfect RS detection rates

The rates of perfect RS detection range between 0% and 81% for OPIL, between 14% and 59% for ROSALI-IRT and between 71% and 97% for ROSALI-RMT. In all cases, ROSALI-RMT has the best performance to detect exactly what has been simulated, i.e.

the correct type of response shift on the correct item only. Furthermore, the rates of perfect RS detection of OPIL and ROSALI-IRT depend on the type of recalibration. In fact, OPIL performs better than ROSALI-IRT in case of simulated uniform recalibration with rates ranging between 42% and 81% and between 14% and 47% respectively. On the opposite, ROSALI-IRT performs better than OPIL in case of simulated non-uniform recalibration with rates ranging between 21% and 59% and between 0% and 21% respectively. The difference in performance of OPIL according to type of recalibration can be related to the observed differences between rates of flexible and perfect detection. Large differences between rates of flexible and perfect detection were observed in case of non-uniform recalibration (between 26% and 72%) but smaller differences for uniform recalibration (between 7% and 13%). Thus, when non-uniform recalibration was simulated, OPIL very often concluded to either another type of RS than non-uniform recalibration or to non-uniform recalibration jointly with another type of RS on the same item. But OPIL was rather able to conclude to uniform recalibration when it was simulated. Most flexible RS detection: proportion of datasets where the method was able to identify at least the correct item among others, when RS was simulated Flexible RS detection: proportion of datasets where any type of RS was detected and accounted for on the correct item only, when RS was simulated Perfect RS detection: proportion of datasets where the method has detected exactly the simulated RS (correct type of RS on the correct item only), when RS was simulated * Due to high rates of false RS detection, results for ROSALI-IRT should be interpreted with caution

Datasets with one item affected by reprioritization

Rates of most flexible, flexible and perfect detection of reprioritization for OPIL and ROSALI-IRT are presented in Table 3. Only rates of most flexible and flexible RS detection are presented for ROSALI-RMT as this method is unable to detect RP.

The rates of most flexible RS detection range between 5% and 62% for OPIL, between 13% and 89% for ROSALI-IRT and 0% and 19% for ROSALI-RMT; the rates increase with the sample size (N) and decreased with the number of items (J). The rates of most flexible RS detection are lower in case of simulated RP than those observed for UR or NUR which is unexpected for OPIL and ROSALI-IRT that are supposed to be able to detect RP. OPIL and ROSALI-IRT also had difficulties in identifying which item was affected by RS contrary to ROSALI-RMT. The rates of flexible RS detection range between 3% and 51% for OPIL, between 5% and 34% for ROSALI-IRT and between 0% and 17% for ROSALI-RMT; the rates increase with the sample size and decrease with J in most cases. A larger difference between most flexible and flexible rates were observed for ROSALI-IRT as compared to OPIL indicating that ROSALI-IRT more often concluded that RS affected other items jointly with the correct item. The rates of perfect RS detection are dramatically low and range between 0% and 8% for OPIL and between 0% and 21% for ROSALI-IRT. OPIL and ROSALI-IRT were hardly ever able to conclude that reprioritization occurred when it was simulated. A larger difference between flexible and perfect rates were observed for OPIL than for ROSALI-IRT because most of the time OPIL identified non-uniform recalibration instead of reprioritization or jointly to reprioritization. Most flexible RS detection: proportion of datasets where the method was able to identify at least the correct item among others, when RS was simulated Flexible RS detection: proportion of datasets where any type of RS was detected and accounted for on the correct item only, when RS was simulated Perfect RS detection: proportion of datasets where the method has detected exactly the simulated RS (correct type of RS on the correct item only), when RS was simulated NA: not applicable * Due to high rates of false RS detection, results for ROSALI-IRT should be interpreted with caution

As ROSALI-RMT cannot account for RP, only UR or NUR can be detected at step 4.The proportion of each type of detected recalibration among datasets meeting flexible rate requirements (correct item where RS was simulated) at step 4 for ROSALI-RMT are presented in Table 4. For datasets with J=4 items, ROSALI-RMT accounted more often for uniform recalibration whereas non-uniform recalibration was more often accounted for with 7 items. Flexible RS detection: proportion of datasets where RS was detected and accounted for on the correct item only, when RS was simulated UR (NUR): proportion of Model 4 with only uniform (non-uniform) recalibration accounted for among significant tests Datasets with two or three items affected by RS Globally, all methods perform similarly for simulated datasets where two or three items were affected by RS (results not shown) as compared to datasets where only one item was affected by RS. For simulated uniform and non-uniform recalibrations, ROSALI-IRT still shows the highest most flexible RS detection rates whereas ROSALI-RMT performs better than OPIL and ROSALI-IRT according to flexible and perfect detection rates.

Largest differences between the rates of most flexible and flexible RS detection were observed for ROSALI-IRT indicating that this method very often concluded to the presence of RS on the correct items plus other items. On the opposite, large differences between rates of flexible and perfect detection were observed in case of non-uniform recalibration for OPIL when non-uniform recalibration was simulated indicating that OPIL very often concluded to NUR jointly with RP on the same item. Last, the different rates of reprioritization detection are dramatically low for OPIL and ROSALI-IRT.

The cases where 2 items over 4 were affected by uniform recalibration have to be interpreted separately. In these cases, a pair of items was affected by RS and a pair of items was not affected by RS. The different rates of detection for datasets with 2 items affected by UR were surprisingly lower than for datasets with 2 items affected by NUR and lower than for datasets with 1 item affected by UR. It appeared that, in datasets with 2 items affected by UR, methods most identified the pair on which no RS was simulated. For example, in datasets with items 1 and 2 affected by UR, the different rates of detection computed regarding the pair of items on which no RS was simulated (items 3 and 4) were higher than the different rates of detection computed regarding the pair of items on which RS was simulated (items 1 and 2). Detection of RS on the simulated pair or on the other pair of items gives equivalent model and we had to consider the rates of perfect detection on the other pair of items as well. Rates of RS perfect detection of the simulated pair of items (items 1 and 2) and of the other pair of items (items 3 and 4) for each method are presented in Table 5. Summing perfect detection rates computed on the pair of items on which RS was simulated and perfect detection rates computed on the pair of items on which no RS was simulated ("sum" column when assessing change) gave an idea of the total proportion of equivalent models.

This sum gave similar rates and trends than for datasets with 1 item affected by UR.

Therefore, it seems that all methods perform similarly for this particular case with a higher performance of ROSALI-RMT. Perfect criterion: proportion of datasets where the method has detected exactly the simulated RS (correct type of RS on the correct pair of item only), when RS was simulated Other criterion: proportion of datasets where the method has detected the correct type of RS on the other pair of item only, when RS was simulated Sum: sum of perfect and other criteria Gray cells: Rates for other criterion is higher than rates for perfect criterion * Due to high rates of false RS detection, results for ROSALI-IRT should be interpreted with caution

Discussion

This study compared the performance of SEM (OPIL method), IRT (ROSALI-IRT method) and RMT (ROSALI-RMT method) approaches for item-level RS detection. ROSALI-RMT performs better than OPIL and ROSALI-IRT in the light of rates of false detection, flexible detection and perfect RS detection when either UR or NUR was simulated. Consequently, if a partial-credit model fits the data, the RMT-based approach should be preferred for investigating only recalibration RS at item-level.

For all methods, a Bonferroni correction was applied at step 3 to adjust for multiple testing which is not a frequent correction in Oort's procedure and its extensions.

However, adjusting for multiple testing seems adequate as the rates of false detection of RS based on the LRT at step 2 in datasets with no RS simulated were corrected during step 3 leading to lower rates of false detection of RS based on model 4.

Results for lower values of reprioritization (

) and of non-uniform recalibration (shift between -0.5 and 0.5) were not shown. For all methods, rates of RS detection were much lower for these values of simulated RS parameters indicating that these values are certainly too small to be detectable by OPIL, ROSALI-IRT and ROSALI-RMT with the sample sizes that were simulated.

ROSALI-IRT method showed high most flexible RS detection rates when either UR or NUR was simulated. These high rates might be misinterpreted as an indicator of good performance. Indeed, ROSALI-IRT also presented high rates of false detection and large differences between most flexible and flexible rates of RS detection rates that tend to indicate that very often ROSALI-IRT overdetected RS by concluding to RS when no RS was simulated or identified items that were not affected by RS when RS was simulated.

The preliminary step used to help reaching model convergence and reduce time for parameter estimation with the NLMIXED procedure could have been detrimental to ROSALI-IRT. In fact, the item difficulties at the first time of measurement were fixed to their estimated values at the preliminary step for all the next steps of the algorithm. This might have been unfair as the main interest in RS analysis is based on the change in item difficulties over time. Hence, ignoring the uncertainty related to the estimation of item difficulties in the preliminary step in further steps might have led to an underestimation of the variance of RS parameters and to reject the null hypothesis too often in the Wald tests of step 3. Thus, this preliminary step might be the cause of over detection of RS when no RS was simulated and to over detection of RS on items unaffected by RS when RS was simulated.

The SEM-based method OPIL showed worse performances than ROSALI-RMT. Data simulated from a different measurement theory (GPCM coming from IRT) could have penalized OPIL. The size of simulated effects are not known in SEM and might be lower than in IRT/RMT leading to RS effects more difficult to detect.

Of note, the parameters of OPIL were estimated with maximum likelihood in line with what has been frequently observed regarding the first applications of Oort's procedure at item level [START_REF] Bl | Using structural equation modeling to detect response shift in performance and healthrelated quality of life scores of multiple sclerosis patients[END_REF][START_REF] Bl | Testing the Assumption of Measurement Invariance in the SAMHSA Mental Health and Alcohol Abuse Stigma Assessment in Older Adults[END_REF]. In case of ordinal data, obtaining estimations using maximum likelihood theory is problematic as assumption of multivariate normality of the item responses is violated. In particular, chi-squares are known to be biased and so likelihood ratio tests that use the chi-square values in OPIL may lead to erroneous conclusions.

Hence, the performance of OPIL could be improved by using an alternative estimation method. Some studies [START_REF] Dolan | Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data[END_REF][START_REF] Muthen | A comparison of some methodologies for the factor analysis of non-normal Likert variables: A note on the size of the model[END_REF] suggest that for ordinal data with at least five answer categories and approximately normal distribution, the ML estimation performs well.

Therefore, for simulated datasets with 7 response categories, we can be quite confident that the results were not much impacted by ML estimation. However, for datasets with less than 5 response categories, OPIL performances might be improved by using techniques to estimate SEM parameters for ordinal data recently implemented in Oort's procedure [START_REF] Verdam | Using structural equation modeling to detect response shifts and true change in discrete variables: an application to the items of the SF-36[END_REF][START_REF] Gadermann | Minimal impact of response shift for SF-12 mental and physical health status in homeless and vulnerably housed individuals: an item-level multi-group analysis[END_REF] such as diagonally weighted least squares parameter estimation.

Despite the fact that it should be encouraged to use these methods which are more suitable for ordinal data, in our study, it seemed that OPIL using maximum likelihood was pretty robust to the least favorable case (i.e. 4 response categories) because we could observe that the performance of OPIL were as good or even better with 4 as compared to 7 response categories (e.g. for false and perfect detection rates).

Apart from these technical points, large differences between flexible and perfect detection rates were observed when NUR was simulated but not when UR was simulated.

It therefore seemed that either OPIL jointly detected another type of RS along with NUR on the correct item or detected another type of RS than NUR on the correct item. By looking more thoroughly into the RS detected in datasets with one item affected by NUR, it appeared that OPIL often concluded that only one item was affected by RS (the simulated item) but that this item was affected by NUR and RP simultaneously.

While reprioritization is easily conceptualized at dimension level (e.g. the social dimension becoming a more important indicator of quality of life after a salient health event than before), the potential different meaning and interpretation of RP at item-level from a methodological or conceptual point of view has already been raised [START_REF] Schwartz | Introduction to special section on response shift at the item level[END_REF]. To even go further, the existence of the concept of RP at item-level can be questioned. If it is easy to conceptualize that the importance of some dimensions in the definition of a multidimensional concept can be subject to change over the disease course, it is not straightforward to expect that some items can become more important than others over time in the unidimensional context of a single dimension. The dramatically low rates of perfect detection of RP for OPIL and ROSALI-IRT are indicative of the inability of these approaches to detect RP. Although these low rates might be explained by the simulated size of RP, we can also note that due to the formulation of IRT models, a variation in discrimination power parameters (RP) might cause a variation in item difficulties (recalibration) and inversely. Hence, trying to simulate RP in the datasets could have led to simulate also recalibration. The same problem could have occurred in SEM as a variation of factor loadings parameters (RP) might cause a variation in residual variances (NUR) and inversely. Thus, OPIL and ROSALI-IRT could have adjusted for NUR instead of RP due to the link between the RS parameters. Observed differences between the rates of flexible RS detection and of perfect detection revealed that ROSALI-IRT was able to identify the items affected by RS but not the type of simulated RS on these items.

Indeed, ROSALI-IRT often concluded that items were affected by UR or NUR (change in item difficulties) when RP (change in discrimination parameters) was simulated.

Similarly, OPIL often concluded that items were affected by NUR (change in residual variances) when RP (change in factor loadings) was simulated.

As expected, ROSALI-RMT showed low most flexible rates of detection for RP in particular for datasets with 7 items as RP is not operationalized in RMT. But, when RS was accounted for in model 4, the item affected by RS was quite often correctly identified.

Reconceptualization was not assessed in the simulation study. The SEM-based method OPIL is able to provide clues of reconceptualization as poor fit of measurement model in step 1 can indicate that the measurement model does not hold for both times of measurement. On the opposite, fit indices or tests are not available for longitudinal IRT or RMT models so that ROSALI-IRT and ROSALI-RMT cannot help investigating reconceptualization. Multidimensional IRT or RMT models would be required to do so.

From a conceptual point of view, item-level reconceptualization means that some items can load on one factor (the dimension of interest) at one time of measurement and load on another factor (an already existing or new dimension) at another time of measurement.

Thus, item-level reconceptualization will probably lead to dimension-level RS. As itemlevel RS detection is operationalized in a unidimensional context, it seems important to also look at dimension-level RS to have a complementary insight and a comprehensive overview of RS and longitudinal change in PRO data.

ROSALI-IRT and ROSALI-RMT were described as different methods for RS detection even though it could be argued that Rasch models are embedded within IRT and that they can be shown to be mathematically equivalent (e.g. Rasch as a "special case" of IRT with all discrimination power parameters fixed to one). However, it does not mean that the models are philosophically or conceptually equivalent, RMT was originally developed to find data that fits the model while IRT models were developed to be altered in order to fit the data. The rationale for distinguishing item-level RS analysis with ROSALI-RMT as compared to ROSALI-IRT was that models coming from Rasch Measurement Theory possess the specific objectivity property that can be valuable when some items are missing, in an ignorable way or not. In this study, ROSALI-RMT has shown better performances in the context of complete data. Further simulation studies taking into account the shortcomings of the present study (e.g. using more suitable SEM-based methods for ordinal data, not fixing the item difficulties at the first time of measurement to their estimated values in ROSALI-IRT, including a generating model based on the SEM framework) are now needed to confirm that ROSALI-RMT performs better than OPIL and ROSALI-IRT in particular in case of non-ignorable missing items.

RS detection relies on two strong hypotheses: all individuals of the sample are expected to experience RS the same way and RS is evaluated before and after a salient health event. We can easily understand that adaptation may not occur in the same manner for all individuals according to the various history and personality of patients even in a homogeneous sample at baseline and that RS should be evaluated at a more individual degree. Adaptation is also not likely to occur at the same time for all individuals especially as a health event of interest can have consequences at different times in the patient's life or the health event can be a chronic disease which can be viewed as a complex sum of events at different times over the course of illness. Investigating RS at a more individual degree and over the disease course instead than between two times of measurement [START_REF] Barclay-Goddard | Response shift: a brief overview and proposed research priorities[END_REF] are important future paths of research to help understanding adaptation and improving care with specific therapies for maladaptative patients and provide adequate support for a better adaptation of patients to their disease over time.

  : recalibration (change in the patient's internal standards of measurements), reprioritization (change in the patient's values), and reconceptualization (change in the patient's definition of the measured concept).

model 1 )

 1 is established in which no constraints on RS parameters across time are imposed. Failing to establish a measurement model with satisfactory fit is indicative of reconceptualization. In step 2, all RS parameters are constrained to be equal across time constituting a model assuming no RS i.e. longitudinal measurement invariance (model 2). The fit of model 2 and model 1 are compared using a Likelihood ratio test (LRT). If the LRT is not significant, no RS is assumed and the procedure goes directly to step 4. If the LRT is significant, a global occurrence of RS is assumed and the procedure goes on to step 3 to identify the types of RS on the affected dimensions. Step 3 consists of a step-bystep improvement of model 2 by relaxing one by one RS parameters constraints leading to model 3 accounting for all detected RS. In step 4, the final model or model 4 (last updated model 3 if the LRT was significant, model 2 otherwise) assesses differences in latent trait means across time, adjusted for identified RS if appropriate, to evaluate longitudinal change. Unlike the Oort's procedure investigating RS at dimension level, methods that focus on item-level RS detection are applied on a single dimension.

(

  ) as for ROSALI-IRT. Discrimination power parameters are assumed to be equal to 1 and not to change over time. So, ROSALI-RMT is not able to detect reprioritization. Parameters of all PCM in ROSALI-RMT are estimated by marginal maximum likelihood.The ROSALI-RMT method is also based on the 4 previously described steps testing the global occurrence of RS using a LRT and possibly imposing RS constraints on a given item on change in item difficulties for NUR and UR in steps 2,3 and 4. Parameter estimation was faster and model convergence was easier to reach in ROSALI-RMT than in ROSALI-IRT. Thus, item difficulties were freely estimated in ROSALI-RMT and no preliminary step was used.The main consequence of using PCM is to skip RP detection in step 3. Hierarchy of measurement invariance is therefore meaningless in ROSALI-RMT. In step 3, recalibration detection follows the same process than in ROSALI-IRT: a Bonferroni correction is applied to compare the results of the Wald tests of the different models (recalibration detection) and the Wald test to determine if recalibration is uniform or not is performed at 5% significance level. Data simulation Item responses Data were simulated with a longitudinal generalized partial credit model including RS or not. The latent variable is assumed to be normally distributed with mean and covariance matrix . The negative average change of the latent variable over time (simulated at represents a deterioration of the concept of interest over time (of quality of life for example, if a high level on the latent trait represents a high level of quality of life). Datasets are composed of simulated item responses to a dimension of a questionnaire composed of J polytomous items with m j +1 response categories of N individuals at two measurement occasions. The effect of the sample size, the number of items and the number of answer categories were studied by simulating different values for these parameters. The choice of values for these parameters was guided by what can be encountered in practice in clinical research: small to moderate sample sizes (N=100, 200 or 300 individuals), sizes of dimensions of

  were the same over time (). When RS was simulated, all the individuals of the dataset were affected by only one type of RS (UR or NUR or RP). The number of affected items and the size of the RS also varied to analyze the effect of these parameters. The number of items affected by RS for each dataset could vary from 1 to 3 items. The affected items have been randomly selected among the set of items. For the datasets with 4 (7) items, the maximum number of affected items was set at 2 (3 respectively). Reprioritization was simulated by changing values of discrimination power parameters at time 2 ( =1.5 or 2). Recalibration was simulated by affecting values to change in item difficulties ( ). For uniform recalibration, all item difficulties of the affected item were shifted by -1 at time 2 ( ). For non-uniform recalibration, item difficulties were shifted with different values regularly spaced between [-0.5 and 0.5] or between [-1.5 and 1.5]. Results are presented for the unique value of uniform recalibration and higher values of reprioritization ( =2) and of nonuniform recalibration (shift between [-1.5 and 1.5]). Comparison criteria The combination of the different values of the simulated parameters (sample size, number of items and answer categories in the dimension, dataset affected by RS or not and, in case of simulated RS, type of RS, size of RS and number of affected items) leads to consider 162 different cases. 500 simulated datasets were replicated for each case and then analyzed using OPIL, ROSALI-IRT, and ROSALI-RMT. Performance of the methods were evaluated at step 4 (model 4) with the rate of false detection of RS and the rates of most flexible, flexible and perfect RS detection (presented in figure 1). The rate of false detection (proportion of datasets for which some RS was detected and accounted for, when RS was not simulated), indicates in what proportion the method has concluded to the presence of RS mistakenly. This rate can be based either on the significance of the test of Model 1 (M1) versus Model 2 (M2) at step 2 (LRT) or on the proportion of datasets where RS was detected and accounted for in model 4 (model of step 4 assessing longitudinal change adjusted for identified RS if appropriate). As the LRT was performed at 5% significance level, the rate of false detection based on the LRT is expected to be close to 5%. The rate of false detection based on model 4 can be lower or equal than the rate of false detection based on the LRT. A lower rate of false detection based on model 4indicates that for some datasets where overall presence of RS was evidenced based on the LRT at step 2, no RS was detected on any item during step 3 correcting the wrong decision based on the LRT. To quantify the correction at step 3, the ratio between the difference in rates based on the LRT and on model 4 over the rates based on the LRT were computed.

Figure 1 :

 1 Figure 1: Rates of RS detection from the least strict (most flexible) to the strictest (perfect). Rates are computed as the proportion of datasets meeting requirements at step 4 among datasets with simulated RS.

  The fit of models 1 and 4 is assessed by inspecting fit indices. In OPIL, a model is considered to have an acceptable fit if the root-mean-square error of approximation is <0.08 or the comparative fit index is >0.90. Models with poor fit are improved by relaxing constraints on error covariance of the items measured at the same time. If the fit indices cannot be improved for Model 1 or Model 4, the dataset is not retained for the analysis of performance of OPIL.

	In step 3, OPIL method takes into account the hierarchy of measurement invariance for
	RS (13,25). In longitudinal SEM, factorial invariance is tested to ensure meaningful
	comparisons of sample estimates as means and variances (3). The invariance testing
	strategy follows different steps to evaluate different levels of factorial invariance, from
	the less restricted model to the most restricted one. The hierarchy of the different
	response shifts detection in OPIL is derived from the levels of factorial invariance in
	longitudinal SEM. The constraints on RS parameters are relaxed sequentially in the order
	proposed in Nolte et al. (25):

Table 1 :

 1 Rates of false Response Shift (RS) detection based on the test M1 versus M2 at step 2 (LRT) or on 4 according to simulation values of sample size (N), number of items (J) and number of answer categories (M) for datasets with no simulated RS

			OPIL	ROSALI-IRT		ROSALI-RMT
	N	J M Nb LRT	M4	Corr. Nb LRT	M4	Corr.	Nb LRT M4 Corr.
	100 4 4 499 5.0% 4.0% 20.0% 500 51.0% 41.8% 22.0% 500 6.0% 2.6% 56.7%
	200 4 4 500 5.2% 4.2% 19.2% 500 47.8% 39.2% 21.9% 500 4.0% 2.4% 40.0%
	300 4 4 500 6.0% 5.6% 6.7% 500 47.8% 40.2% 18.9% 500 5.6% 2.8% 50.0%
	100 7 4 500 6.0% 5.6% 6.7% 500 68.6% 53.4% 28.5% 500 6.4% 1.4% 78.1%
	200 7 4 500 5.8% 5.8% 0.0% 500 66.6% 51.0% 30.6% 500 6.0% 1.4% 76.7%
	300 7 4 500 7.8% 7.4% 5.1% 500 68.4% 53.4% 28.1% 500 5.0% 2.6% 48.0%
	100 4 7 498 7.6% 7.4% 2.6% 499 74.4% 62.3% 19.3% 500 6.6% 0.6% 90.9%
	200 4 7 500 9.2% 9.0% 2.2% 500 71.6% 58.6% 22.2% 500 5.8% 2.2% 62.1%
	300 4 7 500 11.8% 11.2% 5.1% 500 73.0% 60.8% 20.1% 500 5.2% 1.6% 69.2%
	100 7 7 500 8.6% 7.6% 11.6% 495 94.6% 80.6% 17.3% 500 4.8% 0.8% 83.3%
	200 7 7 500 14.8% 14.6% 1.4% 500 95.4% 79.6% 19.9% 500 5.4% 1.6% 70.4%
	300 7 7 500 23.0% 22.2% 3.5% 500 94.6% 80.0% 18.3% 500 5.6% 1.4% 75.0%

Nb: number of analyzed datasets. Models excluded from the results: models with poor fit for OPIL or convergence not achieved for ROSALI-IRT LRT: false detection rate based on test M1 vs M2 (step 2) M4: false detection rate based on model 4 Corr.: correction at step 3 computed as the difference of false detection rates divided by the false detection rate based on the LRTcorr.=(LRT-M4)/LRT Response shift detection when RS was simulated Due to high rates of false RS detection, all rates of most flexible, flexible and perfect detection of RS for ROSALI-IRT presented below should be interpreted with caution.

Table 2 :

 2 Rates of RS detection according to simulation values of sample size (N), number of items (J) and number of answer categories (M) for datasets with RS (uniform recalibration and non-uniform recalibration) simulated on 1 item

		Uniform recalibration	Non-uniform recalibration
	N J M	OPIL	ROSALI-IRT*	ROSALI-RMT	OPIL	ROSALI-IRT*	ROSALI-RMT
	100 4 4 Most flexible	93.3%	98.4%	94.0%	59.5%	97.4%	83.8%
	Flexible	78.6%	56.8%	89.6%	46.6%	55.7%	80.6%
	Perfect	75.8%	45.1%	83.2%	20.8%	54.6%	80.6%
	200 4 4 Most flexible	100.0%	100.0%	100.0%	96.0%	100.0%	99.8%
	Flexible	83.0%	58.9%	96.0%	82.0%	59.8%	96.2%
	Perfect	81.4%	46.6%	91.0%	21.0%	59.2%	96.2%
	300 4 4 Most flexible	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%
	Flexible	78.0%	56.9%	95.2%	79.6%	58.0%	95.4%
	Perfect	75.8%	46.7%	92.0%	7.6%	57.8%	95.4%
	100 7 4 Most flexible	87.4%	99.6%	92.4%	47.6%	98.8%	73.8%
	Flexible	62.0%	47.6%	88.6%	28.0%	42.4%	71.4%
	Perfect	59.8%	37.2%	84.4%	10.0%	42.2%	71.2%
	200 7 4 Most flexible	100.0%	100.0%	100.0%	89.8%	100.0%	98.6%
	Flexible	70.8%	43.6%	96.8%	64.4%	44.6%	95.2%
	Perfect	70.2%	35.4%	92.0%	11.4%	44.4%	95.2%
	300 7 4 Most flexible	100.0%	100.0%	100.0%	99.2%	100.0%	100.0%
	Flexible	62.8%	43.4%	95.0%	70.8%	49.0%	95.4%
	Perfect	62.4%	33.9%	91.0%	4.4%	48.8%	95.4%
	100 4 7 Most flexible	100.0%	100.0%	99.8%	95.6%	99.6%	92.4%
	Flexible	76.0%	33.8%	96.6%	75.0%	34.3%	90.6%
	Perfect	72.8%	23.5%	93.6%	5.7%	33.3%	89.6%
	200 4 7 Most flexible	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%
	Flexible	65.9%	36.6%	96.8%	75.2%	38.2%	94.8%
	Perfect	61.9%	23.8%	92.6%	0.0%	37.8%	94.8%
	300 4 7 Most flexible	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%
	Flexible	57.6%	37.3%	95.4%	66.2%	39.0%	97.4%
	Perfect	52.8%	24.0%	90.8%	0.0%	38.0%	97.4%
	100 7 7 Most flexible	100.0%	100.0%	99.6%	86.8%	100.0%	89.8%
	Flexible	65.8%	21.0%	95.4%	56.2%	21.6%	84.2%
	Perfect	64.6%	14.2%	91.2%	3.2%	21.4%	84.0%
	200 7 7 Most flexible	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%
	Flexible	48.8%	25.0%	97.0%	62.4%	30.0%	96.8%
	Perfect	48.0%	16.1%	92.6%	0.0%	30.0%	96.8%
	300 7 7 Most flexible	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%
	Flexible	43.6%	24.1%	96.4%	55.6%	24.0%	96.8%
	Perfect	42.4%	15.2%	93.6%	0.0%	23.4%	96.8%

Table 3 :

 3 Rates of RS detection according to simulation values of sample size (N), number of items (J) and number of answer categories (M) for datasets with RS (high values of reprioritization) simulated on 1 item.

					OPIL		ROSALI-IRT*		ROSALI-RMT
	N	J M Most flexible Flexible Perfect Most flexible Flexible Perfect Most flexible Flexible Perfect
		100 4	4	13.9%	9.9%	0.0%	21.0% 10.2%	0.0%	4.0%	3.8%	NA
		200 4	4	42.0% 32.0%	0.4%	57.2% 29.4%	6.2%	12.4% 11.2%	NA
		300 4	4	62.0% 51.4%	0.0%	79.6% 33.4%	9.0%	19.4% 17.4%	NA
		100 7	4	5.2%	2.6%	0.8%	12.8%	5.4%	0.0%	0.2%	0.2%	NA
		200 7	4	13.8%	7.0%	1.8%	32.0% 12.6%	5.6%	0.8%	0.6%	NA
		300 7	4	29.6% 16.2%	4.6%	70.0% 34.4% 21.4%	1.6%	1.2%	NA
		100 4	7	15.2% 10.4%	0.2%	28.9%	9.4%	0.8%	3.0%	2.8%	NA
		200 4	7	31.8% 23.8%	0.2%	70.4% 21.2%	6.4%	8.8%	8.2%	NA
		300 4	7	51.4% 36.6%	0.0%	88.6% 30.6%	7.4%	16.6% 16.0%	NA
		100 7	7	5.6%	3.0%	1.4%	22.3%	5.6%	0.2%	0.4%	0.4%	NA
		200 7	7	23.6%	8.6%	4.2%	52.4% 11.8%	4.6%	0.6%	0.6%	NA
		300 7	7	39.2% 15.0%	7.6%	85.2% 21.6% 13.2%	2.0%	1.8%	NA

Table 4 :

 4 Type of detected recalibration among datasets meeting flexible rate requirements according to simulation values of sample size (N), number of items (J) and number of answer categories (M) for datasets with RS (high values of reprioritization) simulated on 1 item.

	ROSALI-RMT

Table 5 :

 5 Rates of RS perfect detection according to simulation values of sample size (N), number of items (J) and number of answer categories (M) for datasets with RS (uniform and non-uniform recalibration) simulated on 2 items.

		Non-uniform recalibration		Uniform recalibration
		ROSALI-	ROSALI-
	OPIL	IRT*	RMT	OPIL	ROSALI-IRT*	ROSALI-RMT
	N J M Perfect Other Perfect Other Perfect Other Perfect Other Sum Perfect Other Sum Perfect Other Sum
	100 4 4 10.5% 0.0% 54.2% 0.2% 72.0% 0.0% 24.3% 48.2% 72.5% 13.6% 22.2% 35.8% 31.4% 38.0% 69.4%
	200 4 4 3.2% 0.0% 58.5% 0.0% 94.0% 0.0% 20.0% 50.1% 70.1% 15.8% 29.0% 44.8% 36.4% 51.4% 87.8%
	300 4 4 0.0% 0.0% 66.3% 0.0% 96.2% 0.0% 17.1% 36.6% 53.7% 15.6% 22.8% 38.4% 38.0% 47.6% 85.6%
	100 7 4 3.2% 0.0% 43.2% 0.0% 72.4% 0.0% 53.6% 0.0% 53.6% 25.0% 0.0% 25.0% 81.0% 0.0% 81.0%
	200 7 4 0.2% 0.0% 46.4% 0.0% 96.4% 0.0% 32.2% 0.0% 32.2% 24.0% 0.0% 24.0% 84.2% 0.0% 84.2%
	300 7 4 0.0% 0.0% 45.6% 0.0% 95.4% 0.0% 17.6% 0.0% 17.6% 23.2% 0.0% 23.2% 85.6% 0.0% 85.6%
	100 4 7 1.6% 0.0% 39.4% 0.0% 75.2% 0.0% 20.1% 38.8% 58.8% 4.3% 7.6% 11.9% 32.0% 55.6% 87.6%
	200 4 7 0.0% 0.0% 44.4% 0.0% 95.8% 0.0% 7.6% 23.7% 31.3% 3.0% 9.6% 12.6% 31.6% 51.2% 82.8%
	300 4 7 0.0% 0.0% 41.8% 0.0% 95.0% 0.0% 4.2% 14.8% 19.0% 0.8% 11.4% 12.2% 25.0% 53.4% 78.4%
	100 7 7 0.0% 0.0% 21.0% 0.0% 76.8% 0.0% 25.6% 0.0% 25.6% 4.8% 0.0% 4.8% 84.8% 0.0% 84.8%
	200 7 7 0.0% 0.0% 24.4% 0.0% 95.6% 0.0% 8.2% 0.0% 8.2% 5.2% 0.0% 5.2% 84.4% 0.0% 84.4%
	300 7 7 0.0% 0.0% 29.8% 0.0% 95.8% 0.0% 0.8% 0.0% 0.8% 7.4% 0.0% 7.4% 86.6% 0.0% 86.6%

Funding Acknowledgments

This study was supported by the Institut National du Cancer, under reference "INCA_6931". Computations of ROSALI-RMT were performed thanks to the Centre de Calcul Intensif des Pays de la Loire (CCIPL) resources.

Declaration of Conflicting Interests

The Author(s) declare(s) that there is no conflict of interest


Ngurah Indra
email: indra@unud.ac.id

Deep Kamal

Singh

Jean-Marie Bonnin
email: jean-marie.bonnin@irisa.fr

CityView: A Vehicle Based Image and Data Collection for Smart Cities

The development of innovative applications for smart cities has been made possible by the rise of Internet of Things. The situational viewing and surveillance in cities is one such category of applications which can benefit from various networking solutions available to transport images or data from installed sensor cameras. In this paper, we propose and evaluate a citywide image and data collection service based on Vehicular Delay-Tolerant Networks (VDTN) and a simple hierarchical routing mechanism named Data Collection for Low Energy Devices (DC4LED). We study the networking performance in terms of increasing image sizes that can be transported with respect to varying vehicular density in city. We focus mainly on two technologies for sensors to vehicles communications: ZigBee and ITS-G5.

I. INTRODUCTION

In the current era, with the rise of Internet of Things (IoT), innovative applications for smart cities have been gaining momentum [START_REF] Zanella | Internet of Things for Smart Cities[END_REF], [START_REF] Rawat | Wireless sensor networks: a survey on recent developments and potential synergies[END_REF]. One such area of novel applications is for a city's situational viewing and surveillance. Stationary image sensors or low cost cameras can be installed throughout the city to capture images for direct display or further analysis. Such a system can provide services such as detection of a crowded area, information for tourists, traffic jam detection, snow buildup detection, weather check and emergencies, among others. Note that the image quality required for each application may vary. For some types of services, even low-resolution images might be adequate, for example: differentiating between crowded and not so crowded area, distinguishing between traffic jam and regular traffic, weather check or to recognize the snow build-up on the road. On the other hand, high resolution images may be required when counting the number of people or vehicles, estimating the amount of snow build-up, or even: conducting face recognition.

From the networking point of view, we are interested in questions such as what are the networking solutions available for such applications? What is their capacity and performance in terms of QoS parameters? Up to what size of images or data can be transported using such solutions?

In this paper: • We study our VDTN solution's network performance in terms of delivery ratio, latency and data or image sizes that can be transported.

• We focus on studying two technologies that can be implemented to opportunistically connect sensor cameras with vehicles: ZigBee and ITS-G5 Vehicleto-infrastructure (V2I), and discuss ways to further increase its performance. Some trade-offs and criteria should be taken into consideration before choosing the network solution to implement. The first criteria can be the data latency that can be tolerated by the application. Some applications need to receive data in real-time or near real-time, and some others can tolerate higher latency. The second is the bandwidth of the network available to transfer the corresponding resolution of images from the sensor camera. Here the solutions like the Long Range Wide Area Networking (LoRaWAN) will have a disadvantage that they only support low bandwidth not suitable for transport images. The third is the cost incurred by the network utilization, where networks such as cellular networks may cost more. Nevertheless, given the cost of integrating cellular communication capability in communicating objects and the cost of the communication, a few works explore the usage of LoRaWAN network to transport images.

We organize the rest of this paper as follows. Section II describes related works and how our work differs from them. Section III presents the vehicle-based image and data collections service in more details. Section IV describes the simulation setup. Section V presents and discusses simulation results. Lastly, some conclusions and the possibility of future works are provided in Section VI.

II. RELATED WORKS

Previous works in [START_REF] Pham | Low-cost, Low-power and Long-range Image Sensor for Visual Surveillance[END_REF] and [START_REF]Robust CSMA for long-range LoRa transmissions with image sensing devices[END_REF] have developed an image collection solution that utilized off-the-shelves components to gain flexibility and modularity. They implemented LoRa technology for low-power and longrange communications. Their solution, which requires a dedicated access network, can only transmit an image of size up to 1200 bytes every one hour, to be within the duty-cycle limitation set by ETSI. The limitations of LoRaWAN is further discussed in [START_REF] Adelantado | Understanding the Limits of LoRaWAN[END_REF]. Therefore, this solution cannot be adopted for applications which need to transport larger images with higher frequency for viewing or image analysis. In our work, we propose a solution which can deliver images or data with larger sizes within minutes of their generation by the sensor camera.

Another IoT technology to consider is the IEEE 802.15.4 with ZigBee as its most widely deployed upper-layer enhancement. The implementation of IEEE 802.15.4 for smart metering is presented in [START_REF] Veloso | An IoT Smart Metering Solution Based on IEEE 802[END_REF], where they concluded that communications between devices ranging more than 25 meters can significantly harm the reliability. In [START_REF] Oliveira | ZigBee Wireless Dynamic Sensor Networks: Feasibility Analysis and Implementation Guide[END_REF], the authors presented their proof-ofconcept for the feasibility of ZigBee's data collection solution using a vehicle as a mobile sink. They also pointed out certain limitations for the ZigBee implementation in specific applications, which necessitates some improvements. Our work will push further the capability of the technology to deliver larger size of data or images.

The most recent technology designated for vehicular communications is the IEEE 802.11p, with ITS-G5 as its standardization for Intelligent Transportation Systems (ITS) in Europe. It allows Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communications, which can seamlessly be extended for communications with objects in the environment, i.e. the Vehicle-to-Everything (V2X) communications. An experimental analysis of the technology presented in [START_REF] Teixeira | Vehicular networks using the IEEE 802.11p standard: An experimental analysis[END_REF] concluded that V2V communications without noticeable decrease in bitrate can be achieved if the distance is not exceeding 300 m in urban area. Furthermore, the technology have also been utilized in [START_REF] Pereira | Vehicle-to-Vehicle Real-Time Video Transmission through IEEE 802.11p for Assisted-Driving[END_REF] for a successful V2V video transmission. It demonstrated that on an average of 4 Mbps bandwidth is achievable for exchanging video during car overtaking scenario. In a scenario of data collection from stationary sensors, the contact duration with moving vehicles is generally very brief and the connection is intermittent. Thus, we employ VDTN solution which is known for its store, carry and forward approach [START_REF] Benamar | Routing protocols in vehicular delay tolerant networks: A comprehensive survey[END_REF]. Our work in [START_REF] Er | DC4LED: A Hierarchical VDTN Routing for Data Collection in Smart Cities[END_REF] concluded that for small-sized sensor data, such as air quality measurements, a simple hierarchical routing mechanism for VDTN, which we named Data Collection for Low Energy Devices (DC4LED), could be utilized. We showed that it was a high performing data collection service for delay-tolerant applications in smart cities. However in [START_REF]Contact Strategy for VDTN Data Collection in Smart Cities[END_REF], we also observed that disparity of vehicle's mobility affects data collection from sensors in some areas of the city. In some parts of the city, a large amount of data from sensors stays waiting very long before it gets picked by passing vehicles. Some data may even have to be dropped due to exceeding its time-to-live (TTL) or due to buffer overflow in sensors.

Image collection scenario is different. First the data size will be relatively higher. Such cameras may have bigger buffer capacity and also they will be able to choose from a broader range of networking technologies. Thus, we evaluate the performance of the VDTN routing strategy for the collection of bigger sized data coming from sensor cameras.

III. VEHICLE-BASED IMAGE AND DATA COLLECTION SERVICE

The focus of our current work is to provide image or data collection service for delay-tolerant applications. This service can be provided by an opportunistic network such as VDTN via its V2X communication capabilities. The open standardization and the use of unlicensed spectrum have been the key driver for the implementation of the technology in both public and private sectors. We focus on two link layer technologies: ZigBee and I compares their specifications with parameter values corresponding to their respective standardization. It also shows specifications of LoRaWAN for comparison. We already discussed the bandwidth and duty-cycle limitation of LoRaWAN for large-size image and data collection in the previous section, which make it unsuitable for our purposes.

Table I also shows four types of image sizes which we later use in our evaluation. The smallest image size of 1.2 KB is the size of images that can be delivered using LoRa, as discussed in the previous section. The largest image size is calculated from a 640x480 32 bpp raw RGBA color image, which divided by 8 bits makes it 1200 KB or 1.2 MB in size (of course, with compression, higher resolution images can be assumed, but we still consider sizes up to 1.2 MB in this study). By assuming 10:1 JPEG compression ratio for the raw image, we arrive at the size of 120 KB, and compressing it further with the same ratio makes the image size to be 12 KB. We then devise two sets of simulation scenarios based on the available bandwidth for the two technologies in focus. The goal is to study the size of images the technology can deliver in the context of VDTN.

In the context of VDTN, DC4LED is a forwarding algorithm which implements a simple hierarchical routing scheme. Algorithm 1 details DC4LED's forwarding algorithm. The idea is to statistically assign a level to the nodes in the city, instead of having complex routing decisions and metrics. This level is based on their reliability and capability to deliver the data to the server.

DC4LED considers mobility features of vehicles such as buses, taxis, and cars. Buses follow fixed routes and almost always on the move during their service hours, which makes them advantageous for VDTN routing in the case where the internet Point-of-Presence (PoP) is along their path. Cars and taxis, on the other hand, can roam streets which are not passed by buses and can gather data from sensors located close to any streets in the city. Furthermore, taxis roam the cities more as compared to cars, and generally, cars make only a few trips and have longer stationary times.

Thus, based on differences in the mobility pattern, we consider cars as the least reliable forwarder among vehicles, while buses are the most reliable in the algorithm. Note that a node does not forward the data to another node if that node's hierarchical level is inferior or equal to the current node. Thus, a bus or a taxi will not forward the data to a car, etc. In scenario 1, all sensor cameras equipped with ZigBee link transmit images with the size of 1.2 KB and 12 KB to in-range vehicles. For V2V communications, we assume that ITS-G5 is used. By means of V2V communication the data is then forwarded using the DC4LED's hierarchical forwarding algorithm. The data is forwarded until it reaches one of the Internet Pointof-Presence (PoP) and then onto the central server. We do not include image sizes of 120 KB and 1.2 MB for the simulation in scenario 1, because ZigBee's available bandwidth will be much smaller than that is required to forward this size of data.

In scenario 2, all sensor cameras are assumed to be equipped with the ITS-G5 V2I link. They transmit images with the size of 1.2 KB, 12 KB, 120 KB and 1.2 MB to vehicles that are in-range. The forwarding algorithm from vehicles to the central server is again assumed to be DC4LED as in scenario 1. In this case, we assume that sensor cameras in the city are connected to a power source and thus have sufficient energy. Also they have larger buffer size for storing high-resolution images.

IV. SIMULATION SETUP

We evaluate the performance by using the Opportunistic Networking Environment (ONE) simulator [START_REF] Keranen | The ONE Simulator for DTN Protocol Evaluation[END_REF] [START_REF] Dede | Simulating Opportunistic Networks: Survey and Future Directions[END_REF]. An overview of our simulations is shown in figure 2. It illustrates centralized situational monitoring in the city of Helsinki, where 37 wireless image capturing cameras are placed almost evenly in an area of about 9 km 2 . Each camera is positioned approximately at 500 meters of other cameras which gave one image data from 37 locations in the city every 5 minutes.

Cars, taxis, and buses equipped with V2X capabilities then opportunistically collect data from in-range sensors to be delivered to the first PoP they encounter, which in turn relays data to a central server. We implement the DC4LED routing algorithm for forwarding the data hierarchically. We assume two bus routes in the city, bus routes 17 and 24, where a PoP is strategically placed at each end of bus routes. The last PoP is positioned in the city center, where traffic usually converges. This in total makes 5 PoPs available to pick up data from cars, taxis, and buses. Now lets see some parameters which are common to all scenarios. Table II provides the parameters and values used in the simulation. Each data packet has a Time to Live (TTL) of 5 hours. We assume each sensor camera has enough buffer size to store all data that it generates during the simulation, to make sure that we solely evaluate the network performances. Each car and taxi with pseudo-random way-points and shortest-path map-based movement has two wireless interfaces. Buses with predefined way-points and routes also have two wireless interfaces identical to cars and taxis in each scenario, but they have different speed range: 10 km/h to 50 km/h for cars and taxis, and 10 km/h to 30 km/h for buses. The stationary time after arriving at each waypoint (or bus stop in the case of buses) is 1 to 120 minutes for cars, 1 to 5 minutes for taxis, and 5 to 30 seconds for buses. We also provide enough buffer size for each type of vehicle to store all data generated by all sensor cameras during the simulation. Each PoP has two interfaces; the first is an ITS-G5 V2I radio interface to connect to cars, taxis, and buses, and the second is a long-range IEEE 802.11 on 5 GHz band, point to point interface (5 km in range and data rate of 300 Mbps) for the link to the server. Such specifications are found in some commercial products. [START_REF] Zanella | Internet of Things for Smart Cities[END_REF] The server also has the same long-distance IEEE 802.11 radio connection (5 km in range and data rate of 300 Mbps) to receive data from each PoP. We then assess the performance with an increasing number of cars and taxis with similar proportion, from 1 car and 1 taxi, which corresponds to vehicles density of 0.22 per km 2

, to 45 cars and 45 taxis, which corresponds to vehicles density of 10 per km 2

. We run each scenario ten times with different initial positioning of cars and taxis in the city, and accumulate all results to come up with convincing trends. Figure 3 shows the delivery probability and the average latency for the collection of 1.2 KB and 12 KB size images with increasing vehicular density. The figure also emphasizes the performance difference between ZigBee and ITS-G5 V2I implementation in the sensor camera. Generally, the part showing the probability of delivery illustrates an increasing trend with more vehicles involved in the data collection.

In scenario 1, where sensor cameras use ZigBee to connect to vehicles, comparison between 1.2 KB and 12 KB image collection shows lower probability of delivery for the larger data size. This is due to the longer contact duration needed between sensors and vehicles to successfully forward larger data, i.e., the probability of successful data transfer in a short duration of contact will be higher for smaller data size. The contact duration is directly related to the vehicle's speed, where higher speed means shorter contact duration. The figure also shows that with higher vehicular density, where the number of slower moving vehicles is also statistically higher, a larger data size can be delivered more successfully.

In term of the average latency, Figure 3 shows the trends of decreasing latency with the increasing vehicles density. Comparison between 1.2 KB and 12 KB data sizes also emphasizes that forwarding larger data introduces higher latency.

In scenario 2, where sensor cameras use ITS-G5 V2I to connect to vehicles, comparison between 1.2 KB and 12 KB size image collection shows minor difference in the probability of delivery and average latency. The performance is almost the same due to the much larger bandwidth available for forwarding the image compared to the size of the image itself.

Lastly in figure 3, we can also observe the difference in performance caused by ZigBee and ITS-G5 V2I utilization in the sensor cameras. The figure shows that ITS-G5 connection (scenario 2), with its longer communications range and higher bandwidth, provides higher probability of delivery and lower average latency than ZigBee. The difference in performance is more apparent in a low vehicles density, for example: in the collection of 1.2 KB images and vehicles density of 0.22 per km 2 , the ITS-G5 probability of delivery is already as high as 0.89, while for ZigBee it is only 0.54. The probability is even worse for ZigBee for the collection of 12 KB images, where the value is as low as 0.31. The average latency for ITS-G5 implementation is shown to be below 10 minutes during high vehicular density, which might be suitable for some delay-tolerant applications requiring lower latency.

Figure 4 emphasizes the percentages of data drop and the latency distribution for scenario 1, where the sensor cameras are connected via ZigBee, for the smallest image size of 1.2 KB. The line graphs which represent percentages of data drop, show two statistics. The line graph on top illustrates the percentages of data being dropped out of all 3071 data instances generated during the simulation. It shows a high drop percentages of 46.45% during the lowest vehicular density. The drop percentage decreases as the vehicle density increases. The drop percentages are as low as 0.13% at the highest vehicle density simulated. Furthermore, the line graph in the middle shows the percentages of data being dropped by sensors, out of all dropped data. The value is very high, 98.51% at the lowest vehicular density, and continues to be higher than 90% up to a vehicular density of 6 per km 2

, even though the overall data drop decreases significantly. This is mainly due to the short communication range offered by the technology, which limits the number of contacts with vehicles and also shortens the contact duration.

The bar graph in Figure 4 shows the distribution of average latency for the successfully delivered data from sensors to the central server in scenario 1. It shows significant data latency in sensors as compared to latency while being carried in vehicles. At the lowest vehicular density, data need to wait 71.29 minutes on an average in the sensor camera's buffer, before any vehicle is in the range to pick-up data. The latency decreases as more vehicles become available to collect data, but the latency at sensor camera's buffer continues to be higher than the latency in vehicles.

Figure 5 shows the delivery probability and the average latency of scenario 2. In scenario 2, sensor cameras use ITS-G5 V2I to connect to vehicles, for the collection of 120 KB and 1.2 MB images with increasing vehicular density. It shows a higher delivery probability and lower latency as compared to ZigBee's performance shown before. This is thanks to the ITS-G5's longer communication range between sensors and vehicles. At the lowest vehicular density of 0.22 per km 2

, the probability of delivery is already at 0.89 with an average latency slightly below 40 minutes. A comparison between 120 KB and 1.2 MB images transmission shows only slight differences in performance, i.e., of around 3 minutes lower average latency at the lowest vehicular density. As more vehicles are involved in the image collection, the probability of delivery becomes higher and the average latency gets lower, reaching slightly below 10 minutes at a vehicular density of 10 per km 2 . Figure 6 emphasizes the percentages of data drop and the latency distribution for the second scenario with the image size of 120 KB. It reveals more on the dynamics of the image collections process. On the percentages of data drop, we can observe a low value of 10.66% as compared to 46.45% for the ZigBee's implementation shown in Figure 4 at the same vehicular density of 0.22 per km 2 . The percentage of data drop by sensors is still dominant at a lower vehicular density, which continues to decrease as the density increases. The same trend can be seen for average latency, where predominantly the latency occurs in sensors before the vehicular density reaches 4 per km 2

. At a vehicular density of 4 per km 2 , when perfect delivery is reached, the average latency trend is reversed. At that point, the latency that occurs in sensors become lower than the latency in vehicles, particularly in taxis and buses. This shows that, given sufficient opportunities, data from sensors can be forwarded to vehicles faster than the data being forwarded from taxis and buses to PoPs. This points out that further latency reduction is possible by adding more PoPs in the city, naturally along the bus routes. The fact that only 2 bus routes with 4 buses are included in the simulation also opens the possibility to further reduce the latency by involving more buses and routes to the data collection process.

By reviewing all results, we can observe the dynamics due to different data sizes from sensors, connection bandwidth between sensors and vehicles, and their communication ranges. A combination of larger data size, lower connection bandwidth, and shorter communication range leads to lower performances, while smaller data size, higher connection bandwidth, and a higher range of communication produces higher performances. This is important for planning the networking part of the data collection, to achieve the desired performance of the planned application. Furthermore, as communication technologies embedded in objects are usually deployed specifically for a given set of usages/applications, their networking capabilities are more or less already fixed.

Finally note that, in future it is highly likely that the vehicles will embark multiple technologies in order to cope with heterogeneous objects and needs. Thus, some technologies can be used in an adaptive way according to their performance in given situations. Another example is that some technologies like LoRaWAN can be used as a fallback solution. When in remote places the images are not being forwarded due to low vehicular density, then either just the meta-data or very low resolution may be forwarded using LoRaWAN. Such minimal data might still be useful depending on the nature of the application.

VI. CONCLUSIONS AND FUTURE WORKS

In this paper, we presented and analyzed the performance of our VDTN based image or data collection with two possible technologies to link sensor cameras with vehicles. The VDTN routing using technologies such as ZigBee and ITS-G5, can forward low-resolution images or data with high delivery probability, provided enough vehicles are involved in the collection process. Highresolution images, on the other hand, can only be forwarded by ITS-G5 with its higher bandwidth and longer communications range. These characteristics proved to be significant in maximizing the delivery probability and minimizing the average latency, thus broadening the possibility of services that can be supported by VDTN. Furthermore, the trend in network performances are consistent with findings in our previous work: a simple routing scheme provides a low network overhead. This in turn is advantageous for its implementation scalability.

Indeed, some solutions need to be devised to raise performance in the case where some areas of the city might experience low vehicular density and mobility. Our future works will study possible strategies to address such issues, such as providing a mechanism for the sensor camera to forward critically-delayed images with its lower resolution through alternative means of communications with the server.

Finally, note that there can be security and privacy concerns with image capture depending on the type of application. They are beyond the scope of this paper. Nevertheless, in the case of public applications, one way to deal could be to blur the images of people and other sensitive information as is done in some street-view, map applications. Moreover, in our networkbased approach, end-to-end security is enforced through encryption directly between the object and the server.

Fig. 1 .

 1 Fig. 1. The Vehicle-Based Image and Data Collection Architecture Overview

Fig. 2 .

 2 Fig. 2. Simulation Overview

  Initialize Nodes: Server.Level = 6; PoP.Level = 5; Tram.Level = 4; Bus.Level = 3; Taxi.Level = 2; Car.Level = 1; Sensor.Level = 0; Input: Connected NeighbourNode while (CurrentNode.Level != 0) do if CurrentNode.Level < NeighbourNode.Level then forward message; end end Algorithm 1: DC4LED Hierarchical Routing Forwarding Decision. Now let us consider two scenarios for VDTN routing.

.

  1 https://greentech-electronics.com/product/ens500/.

Fig. 3 .

 3 Fig. 3. Comparison of Delivery Probability and Average Latency for 1.2 KB and 12 KB Data Size

Fig. 5 .Fig. 6 .

 56 Fig. 5. Delivery Probability and Average Latency for 120 KB and 1.2 MB Data Size

ACKNOWLEDGMENT

This work is supported by a Doctoral Scholarship from the Indonesian Endowment Fund for Education (LPDP) of the Indonesian Ministry of Finance, under Contract Number PRJ-43/LPDP.3/2016.