Alexandre Abraham
email: alexandre.abraham@inria.fr

Fabian Pedregosa

Michael Eickenberg

Philippe Gervais

Andreas Muller

Jean Kossaifi

Alexandre Gramfort

Bertrand Thirion

Gaël Varoquaux

Ga Ël Varoquaux

Machine Learning for Neuroimaging with Scikit-Learn

Keywords: machine learning, statistical learning, neuroimaging, scikit-learn, Python

INTRODUCTION

Interest in applying statistical machine learning to neuroimaging data analysis is growing. Neuroscientists use it as a powerful, albeit complex, tool for statistical inference. The tools are developed by computer scientists who may lack a deep understanding of the neuroscience questions. This paper aims to fill the gap between machine learning and neuroimaging by demonstrating how a general-purpose machine-learning toolbox, scikit-learn, can provide state-of-the-art methods for neuroimaging analysis while keeping the code simple and understandable by both worlds. Here, we focus on software; for a more conceptual introduction to machine learning methods in fMRI analysis, see [START_REF] Pereira | Machine learning classifiers and fMRI: a tutorial overview[END_REF] or [START_REF] Mur | Revealing representational content with patterninformation fMRI-an introductory guide[END_REF], while [START_REF] Hastie | The elements of statistical learning[END_REF] provides a good reference on machine learning. We discuss the use of the scikit-learn toolkit as it is a reference machine learning tool and has and a variety of algorithms that is matched by few packages, but also because it is implemented in Python, and thus dovetails nicely in the rich neuroimaging Python ecosystem.

This paper explores a few applications of statistical learning to resolve common neuroimaging needs, detailing the corresponding code, the choice of the methods, and the underlying assumptions. We discuss not only prediction scores, but also the interpretability of the results, which leads us to explore the internal model of various methods. Importantly, the GitHub repository of the paper1 provides complete scripts to generate figures. The scope of this paper is not to present a neuroimaging-specific library, but rather code patterns related to scikit-learn. However, the nilearn library -http://nilearn.github.iois a software package under development that seeks to simplify the use of scikit-learn for neuroimaging. Rather than relying on an immature and black-box library, we prefer here to unravel simple and didactic examples of code that enable readers to build their own analysis strategies.

The paper is organized as follows. After introducing the scikit-learn toolbox, we show how to prepare the data to apply scikit-learn routines. Then we describe the application of supervised learning techniques to learn the links between brain images and stimuli. Finally we demonstrate how unsupervised learning techniques can extract useful structure from the images.

OUR TOOLS: SCIKIT-LEARN AND THE PYTHON ECOSYSTEM

BASIC SCIENTIFIC PYTHON TOOLS FOR THE NEUROIMAGER

With its mature scientific stack, Python is a growing contender in the landscape of neuroimaging data analysis with tools such as Nipy [START_REF] Millman | Analysis of functional magnetic resonance imaging in python[END_REF] or Nipype [START_REF] Gorgolewski | Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python[END_REF]. The scientific Python libraries used in this paper are:

• NumPy. Provides the ndarray data type to python, an efcient n-dimensional data representation for array-based numerical computation, similar to that used in Matlab [START_REF] Van Der Walt | The numpy array: a structure for efficient numerical computation[END_REF]. It handles efficient array persistance (input and output) and provides basic operations such as dot product. Most scientific Python libraries, including scikit-learn, use NumPy arrays as input and output data type.

• SciPy: higher level mathematical functions that operate on ndarrays for a variety of domains including linear algebra, optimization and signal processing. SciPy is linked to compiled libraries to ensure high performances (BLAS, Arpack and MKL for linear algebra and mathematical operations). Together, NumPy and SciPy provide a robust scientific environment for numerical computing and they are the elementary bricks that we use in all our algorithms. • Matplotlib, a plotting library tightly integrated into the scientific Python stack [START_REF] Hunter | Matplotlib: A 2d graphics environment[END_REF]. It offers publication-quality figures in different formats and is used to generate the figures in this paper.

• Nibabel, to access data in neuroimaging file formats. We use it at the beginning of all our scripts.

SCIKIT-LEARN AND THE MACHINE LEARNING ECOSYSTEM

Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] is a general purpose machine learning library written in Python. It provides efficient implementations of state-of-the-art algorithms, accessible to non-machine learning experts, and reusable across scientific disciplines and application fields. It also takes advantage of Python interactivity and modularity to supply fast and easy prototyping. There is a variety of other learning packages. For instance, in Python, PyBrain (Schaul et al., 2010) is best at neural networks and reinforcement learning approaches, but its models are fairly black box, and do not match our need to interpret the results. Beyond Python, Weka [START_REF] Hall | The weka data mining software: an update[END_REF] is a rich machine learning framework written in Java, however it is more oriented toward data mining. Some higher level frameworks provides full pipeline to apply machine learning techniques to neuroimaging. PyMVPA [START_REF] Hanke | PyMVPA: A python toolbox for multivariate pattern analysis of fMRI data[END_REF] is a Python packaging that does data preparation, loading and analysis, as well as result visualization. It performs multi-variate pattern analysis and can make use of external tools such as R, scikit-learn or Shogun [START_REF] Sonnenburg | The SHOGUN Machine Learning Toolbox[END_REF]. PRoNTo [START_REF] Schrouff | Pronto: Pattern recognition for neuroimaging toolbox[END_REF] is written in Matlab and can easily interface with SPM but does not propose many machine learning algorithms. Here, rather than full-blown neuroimaging analysis pipelines, we discuss lower-level patterns that break down how neuroimaging data is input to scikit-learn and processed with it. Indeed, the breadth of machine learning techniques in scikit-learn and the variety of possible applications are too wide to be fully exposed in a high-level interface. Note that a package like PyMVPA that can rely on scikit-learn for neuroimaging data analysis implements similar patterns behind its high-level interface.

SCIKIT-LEARN CONCEPTS

In scikit-learn, all objects and algorithms accept input data in the form of 2-dimensional arrays of size samples × features. This convention makes it generic and domain-independent. Scikit-learn objects share a uniform set of methods that depends on their purpose: estimators can fit models from data, predictors can make predictions on new data and transformers convert data from one representation to another.

• Estimator. The estimator interface, the core of the library, exposes a fit method for learning model parameters from training data. All supervised and unsupervised learning algorithms (e.g., for classification, regression or clustering) are available as objects implementing this interface. Machine learning tasks such as feature selection or dimensionality reduction are also provided as estimators. • Predictor. A predictor is an estimator with a predict method that takes an input array X test and makes predictions for each sample in it. We denote this input parameter "X test" in order to emphasize that predict generalizes to new data. In the case of supervised learning estimators, this method typically returns the predicted labels or values computed from the estimated model. • Transformer. As it is common to modify or filter data before feeding it to a learning algorithm, some estimators, named transformers, implement a transform method. Preprocessing, feature selection and dimensionality reduction algorithms are all provided as transformers within the library. If the transformation can be inverted, a method called inverse transform also exists.

When testing an estimator or setting hyperparameters, one needs a reliable metric to evaluate its performance. Using the same data for training and testing is not acceptable because it leads to overly confident model performance, a phenomenon also known as overfitting. Cross-validation is a technique that allows one to reliably evaluate an estimator on a given dataset. It consists in iteratively fitting the estimator on a fraction of the data, called training set, and testing it on the left-out unseen data, called test set. Several strategies exists to partition the data. For example, k-fold cross-validation consists in dividing (randomly or not) the samples in k subsets: each subset is then used once as testing set while the others k -1 subsets are used to train the estimator. This is one of the simplest and most widely used crossvalidation strategies. The parameter k is commonly set to 5 or 10. Another strategy, sometimes called Monte-Carlo cross-validation, uses many random partitions in the data.

For a given model and some fixed value of hyperparameters, the scores on the various test sets can be averaged to give a quantitative score to assess how good the model is. Maximizing this cross-validation score offers a principled way to set hyperparameters and allows to choose between different models. This procedure is known as model selection. In scikit-learn, hyperparameters tuning can be conviently done with the GridSearchCV estimator. It takes as input an estimator and a set of candidate hyperparameters. Cross-validation scores are then computed for all hyperparameters combinations, possibly in parallel, in order to find the best one. In this paper, we set the regularization coefficient with grid search in section 5.

DATA PREPARATION: FROM MR VOLUMES TO A DATA MATRIX

Before applying statistical learning to neuroimaging data, standard preprocessing must be applied. For fMRI, this includes motion correction, slice timing correction, coregistration with an anatomical image and normalization to a common template like the MNI (Montreal Neurologic Institute) one if necessary. Reference softwares for these tasks are SPM [START_REF] Friston | Statistical parametric mapping: the analysis of functional brain images[END_REF] and FSL [START_REF] Smith | Advances in functional and structural MR image analysis and implementation as FSL[END_REF]. A Python interface to these tools is available in nipype Python library [START_REF] Gorgolewski | Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python[END_REF]. Below we discuss shaping preprocessed data into a format that can be fed to scikit-learn. For the machine learning settings, we need a data matrix, that we will denote X, and optionally a target variable to predict, y.

SPATIAL RESAMPLING

Neuroimaging data often come as Nifti files, 4-dimensional data (3D scans with time series at each location or voxel) along with a transformation matrix (called affine) used to compute voxel locations from array indices to world coordinates. When working with several subjects, each individual data is registered on a common template (MNI, Talairach...), hence on a common affine, during preprocessing.

Affine matrix can express data anisotropy, when the distance between two voxels is not the same depending on the direction. This information is used by algorithms relying on the spatial structure of the data, for instance the Searchlight.

SciPy routine scipy.ndimage.affine transform can be used to perform image resampling: changing the spatial resolution of the data2 . This is an interpolation and alters the data, that is why it should be used carefully. Downsampling is commonly used to reduce the size of data to process. Typical sizes are 2mm or 3mm resolution, but scan spatial resolution is increasing with progress in MR physics. The affine matrix can encode the scaling factors for each direction.

SIGNAL CLEANING

Due to its complex and indirect acquisition process, neuroimaging data often have a low signal-to-noise ratio. They contain trends and artifacts that must be removed to ensure maximum machine learning algorithms efficiency. Signal cleaning includes:

• Detrending removes a linear trend over the time series of each voxel. This is a useful step when studying fMRI data, as the voxel intensity itself has no meaning and we want to study its variation and correlation with other voxels. Detrending can be done thanks to SciPy (scipy.signal.detrend). • Normalization consists in setting the timeseries variance to 1. This harmonization is necessary as some machine learning algorithms are sensible to different value ranges.

FROM 4-DIMENSIONAL IMAGES TO 2-DIMENSIONAL ARRAY: MASKING

Neuroimaging data are represented in 4 dimensions: 3 spatial dimensions, and one dimension to index time or trials. Scikit-learn algorithms, on the other hand, only accept 2-dimensional samples × features matrices (see Section 2.3). Depending on the setting, voxels and time series can be considered as features or samples. For example, in spatial independent component analysis (ICA), voxels are samples. The reduction process from 4D-images to feature vectors comes with the loss of spatial structure. It however allows to discard uninformative voxels, such as the ones outside of the brain. Such voxels that only carry noise and scanner artifacts would reduce SNR and affect the quality of the estimation. The selected voxels form a brain mask. Such a mask is often given along with the datasets or can be computed with software tools such as FSL or SPM.

Applying the mask is made easy by NumPy advanced indexing using boolean arrays. Two-dimensional masked data will be referred to as X to follow scikit-learn conventions: mask = nibabel.load( ' mask . n i i ').get_data() func_data = nibabel.load( ' e p i . n i i ').get_data() # Ensure that the mask is boolean mask = mask.astype(bool) # Apply the mask, X = timeseries * voxels n e a r e s t ', cmap= ' h o t ') # Cosmetics: disable axis plt.axis( ' o f f ') plt.show() Note that a background is needed to display partial maps. Overlaying two images can be done thanks to the numpy.ma.masked array data structure. Several options exist to enhance the overall aspect of the plot. Some of them can be found in the full scripts provided with this paper. It generally boils down to a good knowledge of Matplotlib. Note that the Nipy package provides a plot map function that is tuned to display activation maps (a background is even provided if needed).

X = func_data[mask].T # Unmask data

DECODING THE MENTAL REPRESENTATION OF OBJECTS IN THE BRAIN

In the context of neuroimaging, decoding refers to learning a model that predicts behavioral or phenotypic variables from brain imaging data. The alternative that consists in predicting the imaging data given external variables, such as stimuli descriptors, is called encoding [START_REF] Naselaris | Encoding and decoding in fMRI[END_REF]. It is further discussed in the next section.

First, we illustrate decoding with a simplified version of the experiment presented in [START_REF] Haxby | Distributed and overlapping representations of faces and objects in ventral temporal cortex[END_REF]. In the original work, visual stimuli from 8 different categories are presented to 6 subjects during 12 sessions. The goal is to predict the category of the stimulus presented to the subject given the recorded fMRI volumes. This example has already been widely analyzed [START_REF] Hanson | Combinatorial codes in ventral temporal lobe for object recognition: Haxby (2001) revisited: is there a face area?[END_REF][START_REF] Detre | The multi-voxel pattern analysis (mvpa) toolbox[END_REF][START_REF] O'toole | Theoretical, statistical, and practical perspectives on pattern-based classification approaches to the analysis of functional neuroimaging data[END_REF][START_REF] Hanson | Brain reading using full brain support vector machines for object recognition: there is no face identification area[END_REF][START_REF] Hanke | PyMVPA: A python toolbox for multivariate pattern analysis of fMRI data[END_REF] and has become a reference example in matter of decoding. For the sake of simplicity, we restrict the example to one subject and to two categories, faces and houses.

As there is a target variable y to predict, this is a supervised learning problem. Here y represents the two object categories, a.k.a. classes in machine-learning terms. In such settings, where y takes discrete values the learning problem is known as classification, as opposed to regression when the variable y can take continuous values, such as age.

CLASSIFICATION WITH FEATURE SELECTION AND LINEAR SVM

Many classification methods are available in scikit-learn. In this example we chose to combine the use of univariate feature selection and Support Vector Machines (SVM). Such a classification strategy is simple yet efficient when used on neuroimaging data.

After applying a brain mask, the data consist of 40 000 voxels, here the features, for only 1 400 volumes, here the samples. Machine learning with many more features than samples is challenging, due to the socalled curse of dimensionality. Several strategies exist to reduce the number of features. A first one is based on prior neuroscientific knowledge. Here one could restrict the mask to occipital areas, where the visual cortex is located. Feature selection is a second, data-driven, approach that relies on a univariate statistical test for each individual feature. Variables with high individual discriminative power are kept.

Scikit-learn offers a panel of strategies to select features. In supervised learning, the most popular feature selection method is the F-test. The null hypothesis of this test is that the feature takes the same value independently of the value of y to predict. In scikit-learn, sklearn.feature_selection proposes a panel of feature selection strategies. One can choose to take a percentile of the features (SelectPercentile), or a fixed number of features (SelectKBest). All these objects are implemented as transformers (see section 2.3). The code below uses the f_classif function (ANOVA F-Test) along with the selection of a fixed number of features.

On the reduced feature set, we use a linear SVM classifier, sklearn.svm.SVC, to find the hyperplane that maximally separates the samples belonging to the different classes. Classifying a new sample boils down to determining on which side of the hyperplane it lies. With a linear kernel, the separating hyperplane is defined in the input data space and its coefficients can be related to the voxels. Such coefficients can therefore be visualized as an image (after unmasking step described in 3.3) where voxels with high values have more influence on the prediction than the others (see figure 2). feature_selection = SelectKBest(f_classif, k=500) clf = SVC(kernel= ' l i n e a r ') X_reduced = feature_selection.fit_transform(X) clf.fit(X_reduced, y) ### Look at the discriminating weights coef = clf.coef_ ### Reverse feature selection coef = feature_selection.inverse_transform(coef)

SEARCHLIGHT

Searchlight [START_REF] Kriegeskorte | Information-based functional brain mapping[END_REF]) is a popular algorithm in the neuroimaging community. It runs a predictive model on a spatial neighborhood of each voxel and tests the out-of-sample prediction performance as proxy measure of the link between the local brain activity and the target behavioral variable. In practice, it entails performing cross-validation of the model, most often an SVM, on voxels contained in balls centered on each voxel of interest. The procedure implies solving a large number of SVMs and is computationally expensive. Detailing an efficient implementation of this algorithm is beyond the scope of this paper. However, code for searchlight and to generate figure 2 is available in the GitHub repository accompanying the paper.

RESULTS

Figure 2: Maps derived by different methods for face versus house recognition in the Haxby experiment left: standard analysis; center: SVM weights after screening voxels with an ANOVA; right: Searchlight map. The masks derived from standard analysis in the original paper [START_REF] Haxby | Distributed and overlapping representations of faces and objects in ventral temporal cortex[END_REF] are displayed in blue and green.

Results are shown in figure 2: first F-score, that is standard analysis in brain mapping but also the statistic used to select features; second the SVC weights after feature selection and last the Searchlight map. Note that the voxels with larger weights roughly match for all methods and are located in the house-responsive areas as defined by the original paper. The Searchlight is more expanded and blurry than the other methods as it iterates over a ball around the voxels.

These results match neuroscientific knowledge as they highlight the high level regions of the ventral visual cortex which is known to contain category-specific visual areas. While Searchlight only gives a score to each voxel, the SVC can be used afterward to classify unseen brain scans. Most of the final example script (haxby decoding.py on GitHub) is for data loading and result visualization. Only 5 lines are needed to run a scikit-learn classifier. In addition, thanks to the scikitlearn modularity, the SVC can be easily replaced by any other classifier in this example. As all linear models share the same interface, replacing the SVC by another linear model, such as ElasticNet or LogisticRegression, requires changing only one line. Gaussian Naive Bayes is a non-linear classifier that should perform well in this case, and modifiying display can be done by replacing coef by theta .

ENCODING BRAIN ACTIVITY AND DECODING IMAGES

In the previous experiment, the category of a visual stimulus was inferred from brain activity measured in the visual cortex. One can go further by inferring a direct link between the image seen by the subject and the associated fMRI data.

In the experiment of [START_REF] Miyawaki | Visual image reconstruction from human brain activity using a combination of multiscale local image decoders[END_REF] several series of 10×10 binary images are presented to two subjects while activity on the visual cortex is recorded. In the original paper, the training set is composed of random images (where black and white pixels are balanced) while the testing set is composed of structured images containing geometric shapes (square, cross...) and letters. Here, for the sake of simplicity, we consider only the training set and use cross-validation to obtain scores on unseen data. In the following example, we study the relation between stimuli pixels and brain voxels in both directions: the reconstruction of the visual stimuli from fMRI, which is a decoding task, and the prediction of fMRI data from descriptors of the visual stimuli, which is an encoding task.

DECODING

In this setting, we want to infer the binary visual stimulus presented to the subject from the recorded fMRI data. As the stimuli are binary, we will treat this problem as a classification problem. This implies that the method presented here cannot be extended as-is to natural stimuli described with gray values.

In the original work, [START_REF] Miyawaki | Visual image reconstruction from human brain activity using a combination of multiscale local image decoders[END_REF] uses a Bayesian logistic regression promoting sparsity along with a sophisticated multi-scale strategy. As one can indeed expect the number of predictive voxels to be limited, we compare the 2 SVM used above with a logistic regression and a SVM penalized with the 1 norm known to promote sparsity. The 1 penalized SVM classifier compared here uses a square-hinge loss while the logistic regression uses a logit function.

C value 0.0005 0.001 0.005 0.01 0.05 0.1

1 Logistic Regression 0.50 ± .02 0.50 ± .02 0.57 ± .13 0.63 ± .11 0.70 ± .12 0.70 ± .12 2 Logistic Regression 0.60 ± .11 0.61 ± .12 0.63 ± .13 0.63 ± .13 0.64 ± .13 0.64 ± .13 1 SVM classifier (SVC) 0.50 ± .06 0.55 ± .12 0.69 ± .11 0.71 ± .12 0.69 ± .12 0.68 ± .12 2 SVM classifier (SVC) 0.67 ± .12 0.67 ± .12 0.67 ± .12 0.66 ± .12 0.65 ± .12 0.65 ± .12 Table 1. 5-fold cross validation accuracy scores obtained for different values of parameter C (± standard deviation).

Table 1 reports the performance of the different classifiers for various values of C using a 5-fold crossvalidation. We first observe that setting the parameter C is crucial as performance drops for inappropriate values of C. It is particularly true for 1 regularized models. Both 1 logistic regression and SVM yield similar performances, which is not surprising as they implement similar models. from sklearn.linear_model import LogisticRegression as LR from sklearn.cross_validation import cross_val_score pipeline_LR = Pipeline([( ' s e l e c t i o n ', SelectKBest(f_classif, 500)), ( ' c l f ', LR(penalty= ' l 1 ', C=0.05)]) scores_lr = [] # y_train = n_samples x n_voxels # To iterate on voxels, we transpose it. for pixel in y_train.T: score = cross_val_score(pipeline_LR, X_train, pixel, cv=5) scores_lr.append(score)

ENCODING

Given an appropriate model of the stimulus, e.g. one which can provide an approximately linear representation of BOLD activation, an encoding approach allows one to quantify for each voxel to what extent its variability is captured by the model. A popular evaluation method is the predictive r 2 score, which uses a prediction on left out data to quantify the decrease in residual norm brought about by fitting a regression function as opposed to fitting a constant. The remaining variance consists of potentially unmodelled, but reproducible signal and spurious noise.

On the Miyawaki dataset, we can observe that mere black and white pixel values can explain a large part of the BOLD variance in many visual voxels. Sticking to the notation that X represesents BOLD signal and y the stimulus, we can write an encoding model using the ridge regression estimator: Note here that the Ridge can be replaced by a Lasso estimator, which can give better prediction performance at the cost of computation time.

Receptive fields

Given the retinotopic structure of early visual areas, it is expected that the voxels well predicted by the presence of a black or white pixel are strongly localized in so-called population receptive fields (prf ). This suggests that only very few stimulus pixels should suffice to explain the activity in each brain voxel of the posterior visual cortex. This information can be exploited by using a sparse linear regression -the Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF]-to find the receptive fields. Here we use the LassoLarsCV estimator that relies on the LARS algorithm [START_REF] Efron | Least angle regression[END_REF] and cross-validation to set the Lasso parameter. from sklearn.linear_model import LassoLarsCV # choose number of voxels to treat, set to None for all voxels n_voxels = 50 # choose best voxels indices = mean_scores.argsort()[::-1][:n_voxels] lasso = LassoLarsCV(max_iter=10) receptive_fields = [] for index in indices: lasso.fit(y_train, X_train[:, index]) receptive_fields.append(lasso.coef_.reshape(10, 10))

RESULTS

Figure 3 gives encoding and decoding results: the relationship between a given image pixel and four voxels of interest in the brain. In decoding settings, Figures 3a and3c show the classifier's weights as brain maps for both methods. They both give roughly the same results and we can see that the weights are centered in the V1 and nearby retinotopic areas. Figures 3b and3d show reconstruction accuracy score using Logistic Regression (LR) and SVM (variable mean scores in the code above). Both methods give almost identical results. As in the original work [START_REF] Miyawaki | Visual image reconstruction from human brain activity using a combination of multiscale local image decoders[END_REF], reconstruction is more accurate in the fovea. This is explained by the higher density of neurons dedicated to foveal representation in the primary visual area.

In encoding settings, figure 3e shows classifiers weights for encoding, that we interpret as receptive fields. We can see that receptive fields of neighboring voxels are neighboring pixels, which is expected from retinotopy: primary visual cortex maps the visual field in a topologically organized manner.

Both encoding and decoding analysis show a link between the selected pixel and brain voxels. In the absence of ground truth, seeing that different methods come to the same conclusion comes as face validity.

Figure 3: Miyawaki results in both decoding and encoding. Relations between one pixel and four brain voxels is highlighted for both methods. Top: Decoding. Classifier weights for the pixel highlighted (a. Logistic regression, c. SVM). Reconstruction accuracy per pixel (b. Logistic regression, c. SVM). Bottom: Encoding. e: receptive fields corresponding to voxels with highest scores and its neighbors. f : reconstruction accuracy depending on pixel position in the stimulus. -Note that the pixels and voxels highlighted are the same in both decoding and encoding figures and that encoding and decoding roughly match as both approach highlight a relationship between the same pixel and voxels.

RESTING-STATE AND FUNCTIONAL CONNECTIVITY ANALYSIS

Even in the absence of external behavioral or clinical variable, studying the structure of brain signals can reveal interesting information. Indeed, [START_REF] Biswal | Functional connectivity in the motor cortex of resting human brain using echo-planar MRI[END_REF] have shown that brain activation exhibits coherent spatial patterns during rest. These correlated voxel activations form functional networks that are consistent with known task-related networks [START_REF] Smith | Correspondence of the brain's functional architecture during activation and rest[END_REF].

Biomarkers found via predictive modeling on resting-state fMRI would be particularly useful, as they could be applied to diminished subjects that cannot execute a specific task. Here we use a dataset containing control and ADHD (Attention Disorder Hyperactivity Disorder) patients resting state data (subjects are scanned without giving them any specific task to capture the cerebral background activity).

Resting state fMRI is unlabeled data in the sense that the brain activity at a given instant in time cannot be related to an output variable. In machine learning, this class of problems is known as unsupervised learning. To extract functional networks or regions, we use methods that group together similar voxels by comparing their time series. In neuroimaging, the most popular method is ICA that is the subject of our first example. We then show how to obtained functionally-homogeneous regions with clustering methods.

INDEPENDENT COMPONENT ANALYSIS (ICA) TO EXTRACT NETWORKS

ICA is a blind source separation method. Its principle is to separate a multivariate signal into several components by maximizing their non-Gaussianity. A typical example is the cocktail party problem where ICA is able to separate voices from several people using signal from microphones located across the room.

6.1.1 ICA in neuroimaging ICA is the reference method to extract networks from resting state fMRI [START_REF] Kiviniemi | Independent component analysis of nondeterministic fmri signal sources[END_REF]. Several strategies have been used to syndicate ICA results across several subjects. [START_REF] Calhoun | A method for making group inferences from fMRI data using independent component analysis[END_REF] propose a dimension reduction (using PCA) followed by a concatenation of timeseries (used in this example). [START_REF] Varoquaux | A group model for stable multi-subject ICA on fMRI datasets[END_REF] use dimension reduction and canonical correlation analysis to aggregate subject data. Melodic [START_REF] Beckmann | Probabilistic independent component analysis for functional magnetic resonance imaging[END_REF], the ICA tool in the FSL suite, uses a concatenation approach not detailed here.

Application

As data preparation steps, we not only center, but also detrend the time series to avoid capturing linear trends with the ICA. Applying to the resulting time series the FastICA algorithm [START_REF] Hyvärinen | Independent component analysis: algorithms and applications[END_REF] with scikit-learn is straightforward thanks to the transformer concept. The data matrix must be transposed, as we are using spatial ICA, in other words the direction considered as random is that of the voxels and not the time points. The maps obtained capture different components of the signal, including noise components as well as resting-state functional networks. To produce the figures, we extract only 10 components, as we are interested here in exploring only the main signal structures. # Here we start with Xs: a list of subject-level data matrices # First we concatenate them in the time-direction, thus implementing # a concat-ICA X = np.vstack(Xs) from sklearn.decomposition import FastICA ica = FastICA(n_components=10) components_masked = ica.fit_transform(data_masked.T).T 6.1.3 Results On fig. 4 we compare a simple concat ICA as implemented by the code above to more sophisticated multi-subject methods, namely Melodic's concat ICA and CanICA-also implemented using scikit-learn although we do not discuss the code here. We display here only the default mode network as it is a well-known resting-state network. It is hard to draw conclusions from a single map but, at first sight, it seems that both CanICA and Melodic approaches are less subject to noise and give similar results.

Scikit-learn proposes several other matrix decomposition strategies listed in the module 'sklearn.decomposition'. A good alternative to ICA is the dictionary learning that applies a 1 regularization on the extracted components [START_REF] Varoquaux | Multi-subject dictionary learning to segment an atlas of brain spontaneous activity[END_REF]. This leads to more sparse and compact components than ICA ones, which are full-brain and require thresholding.

LEARNING FUNCTIONALLY HOMOGENEOUS REGIONS WITH CLUSTERING

From a machine learning perspective, a clustering method aggregates samples into groups (called clusters) maximizing a measure of similarity between samples within each cluster. If we consider voxels of a functional brain image as samples, this measure can be based on functional similarity, leading to clusters of voxels that form functionally homogeneous regions [START_REF] Thirion | Dealing with the shortcomings of spatial normalization: Multi-subject parcellation of fMRI datasets[END_REF]. 6.2.1 Approaches Several clustering approaches exists, each one having its own pros and cons. Most require setting the number of clusters extracted. This choice depends on the application: a large number of clusters will give a more fine-grained description of the data, with a higher fidelity to the original signal, but also a higher model complexity. Some clustering approaches can make use of spatial information and yield spatially contiguous clusters, i.e. parcels. Here we will describe two clustering approaches that are simple and fast. Ward clustering uses a bottom-up hierarchical approach: voxels are progressively agglomerated together into clusters. In scikit-learn, structural information can be specified via a connectivity graph given to the Ward clustering estimator. This graph is used to allow only merges between neighboring voxels, thus readily producing contiguous parcels. We will rely on the sklearn.feature extraction.image.grid to graph function to construct such a graph using the neighbor structure of an image grid, with optionally a brain mask. K-Means is a more top-down approach, seeking cluster centers to evenly explain the variance of the data. Each voxels are then assigned to the nearest center, thus forming clusters. As imposing a spatial model in K-means is not easy, it is often advisable to spatially smooth the data.

To apply the clustering algorithms, we run the common data preparation steps and produce a data matrix. As both Ward clustering and K-means rely on second-order statistics, we can speed up the algorithms by reducing the dimensionality while preserving these second-order statistics with a PCA. Note that clustering algorithms group samples and that here we want to group voxels. So if the data matrix is, as previously a (time points × voxels) matrix, we need to transpose it before running the scikitlearn clustering estimators. Scikit-learn provides a WardAgglomeration object to do this feature agglomeration with Ward clustering [START_REF] Michel | A supervised clustering approach for fMRI-based inference of brain states[END_REF], but this is not the case when using K-Means. connectivity = grid_to_graph(n_x=mask.shape[0], n_y=mask.shape[1], n_z=mask.shape[2], mask=mask) ward = WardAgglomeration(n_clusters=1000, connectivity=connectivity) ward.fit(X) # The maps of cluster assignment can be retrieved and unmasked cluster_labels = numpy.zeros(mask.shape, dtype=int) cluster_labels[mask] = ward.labels_ 6.2.2 Results Clustering results are shown in figure 5. While clustering extracts some known large scale structure, such as the calcarine sulcus on fig 5.a, it is not guaranteed to delineate functionally specific brain regions. Rather, it can be considered as a compression, that is a useful method of summarizing information, as it groups together similar voxels. Note that, as K-means does not extract spatiallycontiguous clusters, it gives a number of regions that can be much larger than the number of clusters specified, although some of these regions can be very small. On the opposite, spatially-constrained Ward directly creates regions. As it is a bottom-up process, it tends to perform best with a large number of clusters. There exist many more clustering techniques exposed in scikit-learn. Determining which is the best one to process fMRI time-series requires a more precise definition of the target application.

Ward's clustering and K-Means are among the simplest approaches proposed in the scikit-learn. [START_REF] Craddock | A whole brain fmri atlas generated via spatially constrained spectral clustering[END_REF] applied spectral clustering on neuroimaging data, a similar application is available in nilearn as an example.

CONCLUSION

In this paper we have illustrated with simple examples how machine learning techniques can be applied to fMRI data using the scikit-learn Python toolkit in order to tackle neuroscientific problems. Encoding and decoding can rely on supervised learning to link brain images with stimuli. Unsupervised learning can extract structure such as functional networks or brain regions from resting-state data. The accompanying Python code for the machine learning tasks is straightforward. Difficulties lie in applying proper preprocessing to the data, choosing the right model for the problem, and interpreting the results. Tackling these difficulties while providing the scientists with simple and readable code requires building a domainspecific library, dedicated to applying scikit-learn to neuroimaging data. This effort is underway in a nascent project, nilearn, that aims to facilitate the use of scikit-learn on neuroimaging data.

The examples covered in this paper only scratch the surface of applications of statistical learning to neuroimaging. The tool stack presented here shines uniquely in this regard as it opens the door to any combination of the wide range of machine learning methods present in scikit-learn with neuroimagingrelated code. For instance, sparse inverse covariance can extract the functional interaction structure from fMRI time-series [START_REF] Varoquaux | Learning and comparing functional connectomes across subjects[END_REF] using the graph-lasso estimator. Modern neuroimaging data analysis entails fitting rich models on limited data quantities. These are high-dimensional statistics problems which call for statistical-learning techniques. We hope that bridging a general-purpose machine learning tool, scikit-learn, to domain-specific data preparation code will foster new scientific advances.

Figure 1 :

 1 Figure 1: Conversion of brain scans into 2-dimensional data

Figure 4 :

 4 Figure 4: Default mode network extracted using different approaches: left: the simple Concat-ICA approach detailed in this article; middle: CanICA, as implemented in nilearn; right: Melodic's concat-ICA. Data have been normalized (set to unit variance) for display purposes.

Figure 5 :

 5 Figure 5: Brain parcellations extracted by clustering. Colors are random.

http://www.github.com/AlexandreAbraham/frontiers2013Frontiers in Neurosciences

Frontiers in Neurosciences

An easy-to-use implementation is proposed in nilearnFrontiers in Neurosciences

DISCLOSURE/CONFLICT-OF-INTEREST STATEMENT

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Funding: We acknowledge funding from the NiConnect project and NIDA R21 DA034954, SUBSample project from the DIGITEO Institute, France.


Alexandra König
email: a.konig@maastrichtuniversity.nl.

Carlos Fernando

Crispim Junior

Alexandre Derreumaux

Gregory Bensadoun

Pierre-David Petit

Franc ¸ois Bremond

Renaud David

Frans Verhey

Pauline Aalten

Philippe Robert

Validation of an Automatic Video Monitoring System for the Detection of Instrumental Activities of Daily Living in Dementia Patients

Keywords: Alzheimer's disease, assessment, autonomy, dementia, mild cognitive impairment, information and communication technologies, instrumental activities of daily living, video analyses

Over the last few years, the use of new technologies for the support of elderly people and in particular dementia patients received increasing interest. We investigated the use of a video monitoring system for automatic event recognition for the assessment of instrumental activities of daily living (IADL) in dementia patients. Participants (19 healthy subjects (HC) and 19 mild cognitive impairment (MCI) patients) had to carry out a standardized scenario consisting of several IADLs such as making a phone call while they were recorded by 2D video cameras. After the recording session, data was processed by a platform of video signal analysis in order to extract kinematic parameters detecting activities undertaken by the participant. We compared our automated activity quality prediction as well as cognitive health prediction with direct observation annotation and neuropsychological assessment scores. With a sensitivity of 85.31% and a precision of 75.90%, the overall activities were correctly automatically detected. Activity frequency differed significantly between MCI and HC participants (p < 0.05). In all activities, differences in the execution time could be identified in the manually and automatically extracted data. We obtained statistically significant correlations between manually as automatically extracted parameters and neuropsychological test scores (p < 0.05). However, no significant differences were found between the groups according to the IADL scale. The results suggest that it is possible to assess IADL functioning with the help of an automatic video monitoring system and that even based on the extracted data, significant group differences can be obtained.

INTRODUCTION 29

The increase of persons with dementia is accompa- (equipped with various sensors) as long as the activ-86 ities are properly chosen and the learning algorithms are appropriately trained [START_REF] Dawadi | Automated assessment of cognitive health using smart home technologies[END_REF]. Sablier and colleagues developed a technological solution designed for people with difficulties managing ADL, providing a schedule manager as well as the possibility to report occurrences of experiences of symptoms such as depression and agitation [START_REF] Sablier | Ecological assessments of activities of daily living and personal experiences with Mobus, an assistive technology for cognition: A pilot study in schizophrenia[END_REF]. However, indicators of cognitive functioning and autonomy were measured using a test battery and scales [START_REF] Sablier | Ecological assessments of activities of daily living and personal experiences with Mobus, an assistive technology for cognition: A pilot study in schizophrenia[END_REF]. Okahashi et al. created a Virtual Shopping Test-using virtual reality technology to assess cognitive functions in brain-injured patients-correlating variables on the virtual test with scores of conventional assessments of attention and memory [START_REF] Okahashi | A virtual shopping test for realistic assessment of cognitive function[END_REF]. Similar work has been done by Werner et al. using a virtual action planning Supermarket game for the diagnosis of MCI patients [START_REF] Werner | Use of the virtual action planning supermarket for the diagnosis of mild cognitive impairment: A preliminary study[END_REF].

Along this line, a project was launched under the name Sweet-HOME (2012), defining a standardized scenario where patients are asked to carry out a list of autonomy relevant (I)ADLs, such as preparing tea, making a phone call, or writing a check, in an experimental room equipped with video sensors. Within this project, Sacco et al. performed a functional assessment with the help of visual analyses by computing a DAS (Daily Activity Scenario) score able to differentiate MCI from healthy control (HC) subjects [START_REF] Sacco | Detection of activities of daily living impairment in Alzheimer's disease and mild cognitive impairment using information and communication technology[END_REF]. However, analysis was based purely on annotations made by a direct observer, and therefore still risked lack of objectivity and reliability. Automatic, computer-based video analysis, which allows for the recognition of certain events and patients' behavioral patterns, may offer a new solution to the aforementioned assessment problems.

To date, automatic video event recognition has been employed in clinical practice simply for feasibility studies with small samples [START_REF] Romdhane | Automatic video monitoring system for assessment of Alzheimer's disease symptoms[END_REF][START_REF] Stone | Capturing habitual, in-home gait parameter trends using an inexpensive depth camera[END_REF][START_REF] Wang | Testing an in-home gait assessment tool for older adults[END_REF]. Banerjee et al. presented video-monitoring for fall detection in hospital rooms by extracting features from depth information provided by a camera [START_REF] Banerjee | Resident identification using kinect depth image data and fuzzy clustering techniques[END_REF]. Wang et al. used automatic vision analyses for gait assessment using two cameras to differentiate between the gait patterns of residents participating in realistic scenarios [START_REF] Wang | Testing an in-home gait assessment tool for older adults[END_REF].

In order to further evaluate the potential contribution of such technologies for clinical practice, this study aims to validate the use of automatic video analyses for the detection of IADL performance within a larger group of MCI patients and HC subjects carrying out a predefined set of activities. More specifically, the objectives of the study are [START_REF] Reppermund | Impairment 566 in instrumental activities of daily living with high cognitive 567 demand is an early marker of mild cognitive impairment: The 568 Sydney memory and ageing study[END_REF] to compare IADL performances of elderly HC subjects and patients with MCI in a predefined scenario; (2) to compare automatically extracted video data with so-called 'ground-truth' (GT) annotations made manually by a human observer; and (3) to assess the importance of automatic video analyses data for the differentiation between the two populations. As a secondary objective, we investigate the relationship between the participants' performance in the scenario and the results of classical neuropsychological testing, in order to verify whether or not the performance in the created scenario is associated with the status of cognitive functioning.

We expect automatically extracted video detection to achieve results as GT annotations when differentiating between the MCI group and the HC group. We also hypothesize that individuals with MCI will perform poorer in the predefined IADL scenario than HC subjects and that difficulties in executive functioning will be related to the amount of completed activities.

Further, we expect a significant relationship between the video captured performance in the scenario and the classical neuropsychological test results such as the Frontal Assessment Battery (FAB) [START_REF] Dubois | The FAB: A Frontal Assessment Battery at bedside[END_REF] or the Mini-Mental State Examination (MMSE) [START_REF] Folstein | Mini-mental state. A practical method for grading the cognitive state of patients for the clinician[END_REF] and IADL scales [START_REF] Lawton | Assessment of older people: Self-maintaining and instrumental activities of daily living[END_REF].

METHODS

Participants

The study was approved by the local Nice ethics committee and only participants with the capacity to consent to the study were included. Each participant gave informed consent before the first assessment. Participants aged 65 or older were recruited at the memory center in Nice located at the Geriatric Department of the University Hospital. For the MCI group, patients with a MMSE score higher than 24 were included using the Petersen clinical criteria [START_REF] Petersen | Mild cognitive impairment: Clinical char-584 acterization and outcome[END_REF]. Participants were excluded if they had any history of head trauma, loss of consciousness, psychotic aberrant motor behavior, or a score higher than 0 on the Unified Parkinson's Disease Rating scale (UPDRS) [START_REF] Fahn | UPDRS program members. Unified Parkinson's Disease Rating Scale[END_REF] in order to control for any possible motor disorders influencing the ability to carry out IADLs.

Assessments

Participants were administered a cognitive and behavioral examination prior to completing the video monitoring session. General cognitive status was assessed using neuropsychological tests including: MMSE [START_REF] Folstein | Mini-mental state. A practical method for grading the cognitive state of patients for the clinician[END_REF], Frontal Assessment Battery (FAB) [START_REF] Dubois | The FAB: A Frontal Assessment Battery at bedside[END_REF],

Instrumental Activities of Daily Living scale (IADL-E) [START_REF] Mathuranath | Instrumental activities of daily living scale for dementia screening in elderly people[END_REF], Montgometry-Asberg Depression Rating Scale (MADRS) [START_REF] Montgomery | A new depression scale designed to be sensitive to change[END_REF], and Geriatric Depression Scale (GDS) 186 to assess depression levels [START_REF] Yesavage | Development and validation of a geriatric depression screening scale: A preliminary report[END_REF]. Additionally, neu-187 ropsychiatric symptoms were assessed using the 188 Neuropsychiatric Inventory Scale (NPI) [START_REF] Arcy | Speech as a means of monitoring cognitive function of elderly subjects[END_REF]. The event recognition module is composed of a framework for event modeling and a temporal scenario recognition algorithm which assess whether the constraints defined in the event models are satisfied [START_REF] Vu | Automatic video interpretation: A novel algorithm for temporal scenario recognition[END_REF].

Event

Automatic video monitoring results versus ground-truth annotation

The participants performed differently on the IADL scenario according to their diagnostic group; in all three activities (preparing the pillbox, preparing tea, and making/receiving a phone call), the obtained parameters (manually as automatic) showed variations.

All results are presented in detail in -E) range from 0 to 36, with lower score indicating a better functional independency; Scores on the Montgomery Asberg Depression Rating Scale (MADRS) range from 0 to 60 (10 items range from 0 to 6), with higher scores indicating depressive state; Scores on the Geriatric Depression Scale (GDS) range from 0 to 30, with higher scores indicating depressive state. HC, healthy control; MCI, mild cognitive impairment.

378

The results are presented in Fig. 2.

Automatic video monitoring results: Experimental results

Table 5 presents the results of the evaluation of the AVMS with respect to its accuracy at detecting the number of activities of daily living annotated by domain experts while watching the experiment video.

From all 10 proposed activities, 'Reading' was detected automatically with the highest precision of 91.30%, followed by 'Preparing pillbox' with 90.24%, and 'Making phone call' with 89.47%.

DISCUSSION

The presented study demonstrates the additional value of employing new technologies such as automatic video monitoring system in clinical practice for Furthermore, the correlation analyses demonstrated that extracted parameters, particularly execution times of activities, correlated significantly with neuropsy-

431

The study's results were consistent with those pre-432 viously presented in [START_REF] Cf | Evaluation of a monitoring system for event recognition of older people[END_REF], where a recall of 88.30 433 and a precision of 71.23 were demonstrated. Although 434 our evaluation results were obtained from different 435 patients and from a larger cohort, small differences 436 were observed in precision index which is higher by ∼5%, and in the recall index which is lower by 3%.

These differences are a result of a trade-off between AVMS precision and recall performance due to a refinement of the event-modeling step. By opting for more strict constraints in such models, we make the system less prone to errors such as misleading evidence. For instance, instead of patients walking toward the plant to water it, they just stretch from the tea table to do so, as this table is just beside the plant.

Activities where the AVMS presented lower precision refer to at least one of two factors: participants performing the activity far from the camera and/or noise from low-level vision components of the AVMS.

For example, a few patients stopped close by or inside the activity zones for long periods to read the instructions sheet, which caused false-positive detections of the zone-related activities. In addition, noisy data from low-level vision components sometimes shifted the estimation of the position of participants from their actual location to an activity zone close by, mostly when the participants were far from the camera. For the described problems, possible solutions include the adoption of a probabilistic framework to handle noise and event modeling uncertainty, and a multisensor approach for cases where the activities are mis-detected by a lack of view of the participants.

If we try to interpret the results, it is not surprising that MCI participants carried out fewer activities in general and took more time, especially for preparing the pillbox and the phone call, which was detected by the observer as well as by the automatic video analysis.

Recent studies demonstrated that even in MCI patients, difficulties in the execution of complex IADL tasks, could be observed and linked to possible early impairment of executive function [START_REF] Marshall | Alzheimer's Disease Neuroimaging I (2011) 597 Executive function and instrumental activities of daily liv-598 ing in mild cognitive impairment and Alzheimer's disease[END_REF]. This is further in line with our finding of significant group differences in the studied population (see Table 2) on the FAB, a test that specifically measures levels of executive functioning.

Interestingly, the preparing tea activity took longer for HC participants and can be explained by the fact that, for the most part, they correctly completed this activity (which takes at least a minimum of 60 s), whereas MCI patients initiated this activity but did not always finish it completely. Therefore, their execution time was shorter but may serve as an indicator of poor task performance.

One major drawback of this study was that healthy control subjects were recruited through the Memory Clinic and therefore suffered in most cases from subjective memory complaints. However, according to classical assessment tools and diagnostic manual they were cognitively healthy. Thus it is debatable whether or not to classify them as healthy controls, as the MMSE and FAB mean scores for that group were relatively low. Furthermore, the study was only based on a small population size. This does not mean that the chosen parameters were not helpful indicators, and they should be validated with a larger population in the future, potentially combined with other ICT data such as actigraphy [37] or automatic speech analyses [38],

given the fact that certain significant group differences could be observed.

It can be further argued that the experiment was conducted in an artificial laboratory environment and not in a complete natural setting such as a patient's home. This could have had increased the stress level of the participants and consequently an impact on their IADL performance. It is therefore desirable in the future to conduct this type of assessment in more naturalistic set-tings, but that may also represent a less controlled envi-

30 nied

 30 by the need to identify methods that allow for an easy and affordable detection of decline in function-32 ality in the disorder's early stages. Consequently, the 33 development of computerized assessment systems for ically quantify the task quality of daily activities and 83 to perform limited assessment of the cognitive func-84 tioning of individuals in a 'smart' home environment 85

For a more detailed

  analysis, the main focus was placed particularly on three IADLs, namely preparing a pillbox, making a phone call, and preparing tea, because they fall within the commonly used IADL-Lawton scale, and are the most challenging activities for appropriately representing a patient's general autonomy level. However, all other activities were included in the overall IADL assessment procedure and analyses.Automatic video monitoring system and event recognitionIn the first step, after each assessment, a clinician manually gathered data of the amount of activities carried out by the participants. This included parameters such as activity occurrence, activity initiation, and the number of activities carried out completely and correctly. In the next step, a computer vision algorithm was used to automatically extract different parameters representing movement patterns of the participants during the ecological assessment period.The Automatic Video Monitoring System (AVMS) herein used has been fully described[START_REF] Cf | Evaluation of a monitoring system for event recognition of older people[END_REF]. It is composed of two main modules: the vision and the event recognition. The vision module is responsible for detecting and tracking people on the scene. The event recognition module uses the generic constraint-based ontology language proposed by Zouba et al.[START_REF] Zouba | An activity monitoring system for real elderly at home: Validation study[END_REF] for event modeling and the reasoning algorithm proposed by Vu and colleagues[START_REF] Vu | Automatic video interpretation: A novel algorithm for temporal scenario recognition[END_REF] to describe and detect the activities of daily living of interest in this study. The vision module detects people in the scene using an extension of the Gaussian Mixture Model algorithm for background subtraction proposed by Nghiem et al. [35]. People tracking over time is performed by a multi-feature algorithm proposed by Chau et al. using features such as 2D size, 3D displacement, color histogram, and dominant color. The detected people and their tracking information (their current and previous positions in the scene) are then passed to the event recognition module [36].

287

  The semantic information of the observation room 288 where patients conducted the activities of daily living 289 was defined. Contextual or Semantic Elements were 290 defined at the locations where the activities of daily 291 living would be carried out (e.g., telephone zone at 292 top-left corner, tea and plant zones at top-right corner, 293 and pharmacy zone at bottom-left corner). 294 The activity modeling was performed with the sup-295 port of domain experts. The models were mostly made 296 taking into account one or more of the following con-297 straints: the presence of the person in a specific zone, 298 their posture, and their proximity to the object of daily 299 living (when static, e.g., the telephone). These con-300 straints were defined as primitive state models. The 301 combination of these models, along with their tempo-302 ral order, was defined as a composite event. Duration 303 constraints were also used to establish a minimum time 304 of execution for the whole or sub-components of the 305 composite event. 306 Statistical analysis 307 Spearman's correlations were performed to deter-308 mine the association between the extracted video 309 parameters and the established assessment tools in 310 particular for executive functioning, e.g., the FAB.

Fig. 1 .

 1 Fig. 1. The experimental room for the IADL assessment. For the automatic activity detection, the room was divided in different zones according to the designated IADL.participants whereas making tea took longer for the HC

379

  Moreover, the manually and automatically extracted 380 video data parameter 'activity frequency' corre-381 lated significantly with neuropsychological test results 382 namely the MMSE (p < 0.01) and FAB score (p < 0.05). 383 The obtained correlation analyses results are presented in Table 4. Particularly, from the manually annotated parameters, the time spent to prepare the pillbox correlated significantly negatively with the MMSE scores (p < 0.01), whereas the time spent to make a phone call correlated significantly negatively with the FAB scores (p < 0.05). The mean frequency of the activity 'making tea' correlated significantly positively with the FAB scores (p < 0.05). From the automatically extracted parameters, the detected time spent to prepare the pillbox (p < 0.01) and to make the phone call (p < 0.05) correlated significantly negatively with the MMSE scores. None of the extracted parameters correlated with the IADL-E scores.

Fig. 2 .

 2 Fig. 2. The average execution times for each activity in blue annotated manually and in red detected automatically. MCI, mild cognitive impairment; WP, worst performer; BP, best performer; HC, healthy control.

507

  ronment and therefore a bigger challenge from a tech-508 nical point of view. Finally, the current study placed 509 less emphasis on multi-tasking in IADL performances, 510 but rather focused more on the simple execution of 511 tasks sequentially. However, in real life, multi-tasking 512 is of great importance and represents complex cogni-513 tive processing required for functional ability.

514

  It is important to mention that in the field of auto-515 matic video analysis, it is almost impossible to achieve 516 100% accuracy in the activity recognition, often caused 517 as well by inaccurate manual annotations. The chal-518 lenge is to define, for example, the beginning and the 519 end of an activity, which represents a common problem 520 in video analyses. Nevertheless, the activity detection 521 by video analyses might be actually a much closer rep-522 resentation of the reality and the real events happening 523 than annotations of a human observer because the lat-524 est can be influenced by various confounding factors 525 such as fatigue, distraction, lack of concentration, etc. 526 The advantages of using such techniques are that the 527 application in daily practice is easy and reproducible, 528 and add an objective measure to the assessment 529 of autonomy. Furthermore, this evaluation provides 530 quicker results than manual annotations and could be 531 even used as an outcome measure in clinical trials 532 in order to evaluate the effect of certain treatments 533 (pharmacological and non-pharmacological) on the 534 functioning of IADLs of patients. 535 Overall, the study showed in particular that manu-536 ally annotated data gives a more accurate picture of 537 a patient's status to date, and is better validated by 538 traditional diagnostic and neuropsychological assess-539 ment tools. This means that qualitative assessments 540 still seem to better correlate with conventional scor-541 ing than quantitative video extracted parameters. Until 542 now, the obtained data still needs interpretation of an 543 experienced clinician regarding the quality of the car-544 ried out activities. It should be emphasized that this 545 cannot be replaced by technology and is not the objec-546 tive of this research. 547 However, in future studies, we aim for improvement 548 in the activity detection with a larger group sample, in 549 particular to improve the detection of the quality of 550 activity execution, i.e., if an activity was carried out 551 successfully and completely.

  Comparison between the two groups (i.e., MCI patients

	312		with a mean of 25.8 (±2.2) for the MCI group and 28.8
	313	and HC subjects) was performed with a Mann-Whitney	(±1.0) for the HC group (p, 0.001), as well as for the
	314	test for each outcome variable of the automatic video	FAB score with a mean of 14.16 (±1.92) for the MCI
	315	analyses. Differences were reported as significant if	group and 16.2 (±1.44) for the HC group. The mean
	316	p < 0.05.	IADL-E scores did not differ between groups, with a
			mean IADL-E score of 9.9 (±1.7) for the MCI group
	317	Automatic activity recognition evaluation	and 9.6 (±1.1) for the HC group.
	318	The evaluation compared the performance of the
	319	AVMS at automatically detecting IADL with respect
	320	to the annotations manually made by human experts.
	321	The AVMS performance was measured based on the
	322	indices of recall and precision, described in Equations
		1 and 2, respectively. Recall index measures the per-
	338

323

centage of how many of the targeted activities have 324 been detected compared to how many existed. Preci-325 sion index evaluates the performance of the system at 326 discriminating a targeted activity type from others.

327 1. Recall = TP/(TP+FN) 2. Precision = TP/(TP+FP) 328 TP: True Positive rate, FP: False Positive rate, FN: 329 False Negative rate. 330 RESULTS 331 Population 332 19 MCI patients (age = 75.2 ± 4.25) and 19 HC 333 (age = 71.7 ± 5.4) were included.

Table 2 shows the 334 clinical and demographic data of the participants. Sig-335 nificant intergroup differences in demographic factors 336 (gender and age) were not seen. However, significant 337 differences were found between for the MMSE score,

Table 3

 3

. The total frequency of activities as well as the number of correctly completed activities according to manual annotations differed significantly between MCI and HC groups (p < 0.05). Two activities, namely preparing the pillbox and making/receiving the phone call, generally took the MCI participants a longer time to carry out. In turn, for the activity of preparing tea, HC participants took a longer time. The same trends, even if not significant, were detected as well by the automatic video analyses; a significant difference was found between MCI and HC groups (p < 0.05) in the phone call time. Furthermore, MCI and HC participants differed in the total amount of detected activities carried out; the same activities, preparing the pillbox and making/receiving a phone call took longer for MCI

Table 2

 2

		Characteristics of the participants
	Characteristics	HC group n = 19	MCI group n = 19	p
	Female, n (%)	15 (78.9%)	9 (47.4%)	0.091
	Age, years mean ST	71.7 ± 5.37	75.2 ± 4.25	0.07
	Level of Education, n (%)
	Unknown	2 (10.5%)	2 (10.5%)	1
	No formal education	0 (0%)	0 (0%)	-
	Elementary school	1 (5.3%)	5 (26.3%)	0.405
	Middle school	4 (21.0%)	7 (36.8%)	0.269
	High school	4 (21.0%)	3 (15.8%)	1
	Post-secondary education	8 (42.1%)	2 (10.5%)	0.062
	MMSE, mean ± SD	28.8 ± 1.03	25.8 ± 2.22	0.001 * *
	FAB, mean ± SD	16.2 ± 1.44	14.16 ± 1.92	0.002 *
	IADL-E, mean ± SD	9.6 ± 1.12	9.9 ± 1.73	0.488
	NPI total, mean ± SD	0.42 ± 1.43	6.16 ± 6.73	0.00 *
	Data shown as mean ± SD. Bold characters represent significant p-values <0.05. Scores on the Mini Mental
	State Examination (MMSE) range from 0 to 30, with higher scores indicating better cognitive function;
	Scores on the Instrumental Activities of Daily Living for Elderly (IADL

Table 3

 3 Comparison of parameters from video analyses between groups

	Video analyses data	HC n = 19	MCI n = 19	p
	Manually annotated:
	Activities carried out completely and correctly †	9.68 ± 0.48	8.21 ± 1.48	0.00 *
	Activity frequency total ‡ (activities initiated) ‡	11.74 ± 2.62	9.58 ± 1.89	0.007 *
	Preparing Pillbox (f)	1.05 ± 0.23	0.89 ± 0.32	0.086
	Preparing Pillbox time	41.17 ± 17.04	46.17 ± 31.18	0.609
	Making tea (f)	2.68 ± 0.82	2 ± 1	0.068
	Making tea time	41.21 ± 30.60	32.16 ± 35.3	0.175
	Phone call (f)	2 ± 0.47	2.21 ± 0.53	0.198
	Phone call time	66.61 ± 21.75	83.30 ± 30.96	0.118
	Automatically extracted: Activity frequency total	13.26 ± 3.89	10.95 ± 3.15	0.056
	Preparing Pillbox (f)	1.05 ± 0.23	1.17 ± 0.38	0.271
	Preparing Pillbox time	47.64 ± 22.28	70.26 ± 38.01	0.204
	Making tea (f)	2.74 ± 1.33	2.12 ± 1.22	0.136
	Making tea time	102.3 ± 77.3	79.57 ± 40.92	0.531
	Phone call (f)	1.95 ± 0.52	2.17 ± 0.79	0.38
	Phone call time	60.32 ± 21.52	112.61 ± 46.31	0.000

*

Mann-Whitney test: * p < 0.05, * * p < 0.01 HC, healthy control; MCI, mild cognitive impairment; (f), mean frequency of detected event; † Represents the total amount of completely carried out activities without a mistake, ‡ Represents the total of simply initiated activities which are not always necessarily accomplished completely and without mistakes.

Table 4

 4 Correlation between automatic video parameters, manually annotated parameters and conventional cognitive assessments (Spearman's correlation coefficient)

		Video analyses data	MMSE	FAB	IADL-E
		Spearman correlation coefficient (r) / p-values
		Manually annotated	(r)	(r)	(r)
		Activities frequency	0.491 * *	0.394 *	-0.035
				p = 0.002	p = 0.014	p = 0.834
		Activities completed correctly	0.819 * *	0.660 * *	-0.107
				p = 0.000	p = 0.000	p = 0.522
		Automatically extracted
		Activity frequency	0.415 * *	0.273 *	-0.071
				p = 0.005	p = 0.048	p = 0.337
		Manually annotated
		Preparing Pillbox (f)	0.055	0.299	-0.149
				p = 239	p = 0.063	p = 0.127
		Preparing Pillbox time	-0.468 * *	-0.114	-0.179
				p = 0.001	p = 0.409	p = 0.211
		Making tea (f)	0.27	0.363 *	-0.5
				p = 0.083	p = 0.042	p = 0.391
		Making tea time	-0.143	0.053	-0.002
				p = 0.222	p = 0.343	p = 0.396
		Phone call (f)		-0.123	-0.235	0.002
				p = 0.128	p = 0.084	p = 0.465
		Phone call time	-0.280 *	-0.332 *	-0.145
				p = 0.044	p = 0.041	p = 0.291
		Automatically extracted
		Preparing Pillbox (f)	-0.287 *	-0.073	0.125
				p = 0.043	p = 0.295	p = 0.222
		Preparing Pillbox time	-0.618 * *	-0.241	-0.05
				p = 0.001	p = 0.340	p = 0.128
		Making tea (f)	0.223	0.221	-0.264
				p = 0.60	p = 0.083	p = 0.051
		Making tea time	0.016	-0.101	-0.114
				p = 0.392	p = 0.261	p = 0.197
		Phone call (f)		-0.248	0.077	0.158
				p = 0.095	p = 0.330	p = 0.223
		Phone call time	-0.373 *	-0.277 *	-0.054
				P = 0.002	p = 0.049	p = 0.451
		* p < 0.05, * * p < 0.01.
		Table 5
		Activity/Event detection performance
		Activity	Recall	Precision
		Phone call	85	89.47
		Watching TV	83.33	73.77
		Making tea	80.9	80
		Preparing Pillbox	100	90.24
		Watering Plant	75	61.22
		Reading	75	91.3
		Average Recognition	85.31	75.9
		n: 38, MCI: 19 / HC: 19.
	430	chological tests results, namely the MMSE and FAB
		scores.

Microsoft).

ACKNOWLEDGMENTS

This study was supported by grants from the ANR-