Alexandre Naaim
email: alexandre.naaim@univ-lyon1.fr

Alice Bonnefoy-Mazure
email: alice.bonnefoymazure@hcuge.ch

Stéphane Armand

Raphaël Dumas
email: raphael.dumas@ifsttar.fr

Lbmc Umr_T9406

Willy Taillard

Alice Bonnefoy

Stephane Armand
email: stephane.armand@hcuge.ch

Correcting lower limb segment axis misalignment in gait analysis: a simple geometrical method

Highlights

 Hip rotation profile is prone to error due to segment axis misalignment  Thigh medial-lateral axis is redefined a posteriori based on lower limb joint centres  This geometrical method provides less knee crosstalk correction than reference methods  This method provides equal hip rotation repeatability than reference methods  This method is simpler than reference methods and is more widely applicable

Research question: This study proposes an a posteriori geometrical method based solely on segment anatomy that aims to correct the thigh medial-lateral axis definition and crosstalk-related error.

Methods: The proposed method considers the thigh medial-lateral axis as the normal to the mean sagittal plane of the lower limb defined by hip, knee and ankle joint centres during one gait cycle. Its performance was compared to that of an optimisation method which repositions the axis to reduce knee abduction- Three-dimensional hip and knee angles were computed for two gait analysis sessions. Crosstalk was quantified using both the coefficient of determination (r²) between knee flexion-extension and adduction-abduction and the amplitude of knee adduction-abduction. The reproducibility of hip internal-external rotation was also quantified using the inter-trial, inter-session and inter-subject standard deviations and the intraclass coefficient (ICC).

Results: Crosstalk was significantly reduced from r²=0.67 to r²=0.51 by the geometrical method but remained significantly higher than with the optimisation method with a r²<0.01.

Significances: Both methods allowed to improve the hip internal-external reproducibility from poor to moderate (original data: ICC=0.34, geometrical method: ICC=0.65, optimisation method ICC=0.73). One advantage of the geometrical method is that, unlike the optimisation method, it does not require much movement, making it suitable for a wider range of patients.

Keywords:

Introduction

Clinical gait analysis helps clinicians identify gait impairments, thus guiding therapeutic choice (e.g. surgical planning, rehabilitation) in various pathologies (e.g. cerebral palsy, stroke) [START_REF] Wren | Influence of gait analysis on decision-making[END_REF]. However, the joint kinematics involved are often computed with the Conventional Gait Model (CGM) [START_REF] Davis | A gait analysis data collection and reduction technique[END_REF] developed in the 80's, which has several shortcomings. The most common is marker mislocations, which can lead to segment axis misalignment; this can impact the accuracy of the hip rotation profile [START_REF] Della Croce | Human movement analysis using stereophotogrammetry. Part 4: assessment of anatomical landmark misplacement and its effects on joint kinematics[END_REF]. Hip rotation is a critical outcome, but one prone to error [START_REF] Wren | Impact of gait analysis on correction of excessive hip internal rotation in ambulatory children with cerebral palsy: a randomized controlled trial[END_REF].

The medial-lateral axis direction is dependent on the positioning of a wand (or a medial marker) on the thigh. The wand should be positioned in the thigh's frontal plane. This plane can, however, be difficult to define in patients with skeletal deformities: for example, patients with cerebral palsy can exhibit major femoral growth abnormalities such as femoral torsion [START_REF] Gose | Morphometric analysis of the femur in cerebral palsy: 3-dimensional CT study[END_REF]. The resulting misalignment of this medial-lateral axis, which is used to define the flexion-extension axis of the distal joint, leads to a phenomenon called crosstalk. Crosstalk at the knee implies unrealistic adduction-abduction amplitude [START_REF] Della Croce | Human movement analysis using stereophotogrammetry. Part 4: assessment of anatomical landmark misplacement and its effects on joint kinematics[END_REF]. Furthermore, a misalignment of the medial-lateral axis can also lead to errors on the internal-external rotation of the joint [START_REF] Baker | A new approach to determine the hip rotations profile from clinical gait analysis data[END_REF].

Several experimental and computational solutions have been proposed to reduce these errors. One solution is to change the protocol (number and placement of markers) [START_REF] Wu | ISB recommendation on definitions of joint coordinate system of various joints for the reporting of human joint motion--part I: ankle, hip, and spine[END_REF]. Devices such as the Knee Alignment Device can also be used to more accurately define the knee flexion-extension axis [START_REF] Baker | Measuring Walking: A Handbook of Clinical Gait Analysis[END_REF]. Alternatively, the knee flexion-extension axis can be defined using functional methods [START_REF] Ehrig | A survey of formal methods for determining functional joint axes[END_REF]. These methods are based on the joint kinematics rather than the segment anatomy and are applied to a specific movement maximising knee flexion. Functional methods can, however, be problematic for patients with possible adduction-abduction

A C C E P T E D M A N U S C R I P T
mobility or a limited flexion-extension range of motion. In addition, these methods are subject to soft tissue artefacts [START_REF] Sangeux | Evaluation of knee functional calibration with and without the effect of soft tissue artefact[END_REF].

Another error-reducing solution might be to correct data retrospectively. The most classical a posteriori methods optimise the thigh coordinate system orientation to minimise the quadratic variance of the adduction-abduction angle [START_REF] Schache | Defining the knee joint flexionextension axis for purposes of quantitative gait analysis: an evaluation of methods[END_REF] and possibly of the internal-external rotation angle [START_REF] Rivest | A correction for axis misalignment in the joint angle curves representing knee movement in gait analysis[END_REF]. Functional methods, too, can be applied retrospectively to gait records if the flexion during the swing phase is large enough [START_REF] Sangeux | Evaluation of knee functional calibration with and without the effect of soft tissue artefact[END_REF][START_REF] Schache | Defining the knee joint flexionextension axis for purposes of quantitative gait analysis: an evaluation of methods[END_REF]. Different methods have been used such as the axis transformation technics/SARA method [START_REF] Ehrig | A survey of formal methods for determining functional joint axes[END_REF] or STT methods [START_REF] Schwartz | A new method for estimating joint parameters from motion data[END_REF]. Recent findings [START_REF] Sauret | On the use of knee functional calibration to determine the medio-lateral axis of the femur in gait analysis: Comparison with EOS biplanar radiographs as reference[END_REF][START_REF] Passmore | Defining the medial-lateral axis of the femur: Medical imaging, conventional and functional calibration methods lead to differences in hip rotation kinematics for children with torsional deformities[END_REF] tends to suggest that optimisation methods like DynaKad [START_REF] Baker | A new approach to determine the hip rotations profile from clinical gait analysis data[END_REF] gave the best results. Multibody kinematics optimisation methods [START_REF] Begon | Multibody Kinematics Optimization for the Estimation of Upper and Lower Limb Human Joint Kinematics: A Systematized Methodological Review[END_REF] can also define both the segment poses and the joint axes that best track marker trajectories.

However, kinematics obtained from multibody kinematics optimisation methods tends to be strongly impacted by the kinematic model chosen. As a result, if the knee flexion-extension axis, which would correspond to the thigh medial-lateral axis direction, is poorly modelled the final results might be more reproducible but still inaccurate. In addition, all multibody kinematics optimisation methods are liable to suffer from overfitting.

Seeking to reduce crosstalk and enhance hip internal-external rotation reproducibility, the purpose of this study was to propose and evaluate an a posteriori geometrical method of correcting thigh medial-lateral orientation. Based on the position of the hip, knee and ankle joint centres, the method corrects segment axis misalignment so as to enhance the quality of both hip and knee kinematics measurements, in particular hip internal-external rotation and knee adduction-abduction.

A C C E P T E D

M A N U S C R I P T

Methods

Geometrical method

Our geometrical method was inspired by an approach proposed for upper limb kinematics [START_REF] Cooper | Glenohumeral joint kinematics and kinetics for three coordinate system representations during wheelchair propulsion[END_REF], where the upper arm medial-lateral axis is considered normal to the plane defined by three markers placed on the acromion process, humeral lateral epicondyle and ulnar styloid. To adapt this approach to the lower limb, we took the plane defined by the hip, knee and ankle joint centres.

Using the CGM, the thigh anterior-posterior axis was initially defined as that normal to the plane defined by the hip and knee joint centres and the wand marker position [START_REF] Davis | A gait analysis data collection and reduction technique[END_REF]. The medial-lateral axis of the thigh, which defines the knee flexion axis, lies on this normal plane and is orthogonal to the thigh superiorinferior axis defined from the knee joint centre to the hip joint centre. To correct these two axes, the sagittal plane of the lower limb was defined by the hip, knee and ankle joint centres. To avoid collinearity of these points and ensure that the normal was properly oriented to this sagittal plane, the average plane was calculated over all the instants of time when knee flexion was above 20° during the movement. Then the thigh anterior-posterior axis was replaced by the cross product between the normal to the lower limb sagittal plane and the superiorinferior axis of the thigh. Finally, the medial-lateral thigh axis was replaced by another cross product between the newly defined thigh antero-posterior axis and the superior-inferior axis (Figure 1).

Gait Analysis

The method was applied retrospectively to a dataset from a cohort study of 118 patients undergoing total knee joint replacement surgery [START_REF] Bonnefoy-Mazure | Knee Kinematic and Clinical Outcomes Evolution Before, 3 Months, and 1 Year After Total Knee Arthroplasty[END_REF]. The initial project was approved by a local ethics committee (n CRE 09-307), and written consent was obtained from all participants. Each participant was subjected to gait

A C C E P T E D M A N U S C R I P T
analysis before, three months (M3) after and one year (M12) after the surgery. All acquisitions were performed at the Kinesiology Laboratory of Geneva University Hospitals using a Vicon 12-camera motion measurement system (VICON Mx3+; Vicon Peak, Oxford, UK). The cutaneous reflective markers were positioned according to the CGM [START_REF] Davis | A gait analysis data collection and reduction technique[END_REF] on the pelvis and the lower limb.

Data Analysis

To evaluate the performance of the proposed method on knee crosstalk and on hip kinematics reproducibility, only patients undergoing an M3 and M12 follow-up were considered. Although the true bone kinematics was unknown, it was assumed that after a total knee replacement knee adduction-abduction amplitude would be minimal. The hip internal-external rotation reproducibility was also one of our concern. After surgery, all patients went through a standard rehabilitation program for 5 to 6 weeks. As no further treatment was used after this 6-weeks program, it was supposed that the hip internal-external rotation evolution would be minimal between M3 and M12 and could be then assessed.

Our geometrical method was compared to an optimisation method (DynaKad) proposed in the work of Baker et al. [START_REF] Baker | A new approach to determine the hip rotations profile from clinical gait analysis data[END_REF] which minimises knee adductionabduction during gait. Data were analysed using MATLAB 2017a (The MathWorks, Natick, USA) and the Biomechanical Tool Kit [START_REF] Barre | Biomechanical ToolKit: Open-source framework to visualize and process biomechanical data[END_REF]. The hip, knee and ankle joint centres computed using Nexus 1.8.5 (Vicon Peak, Oxford, UK), based on the CGM [START_REF] Davis | A gait analysis data collection and reduction technique[END_REF], were extracted from the C3D files. Finally, joint kinematics was computed using the 3D Kinematics and Inverse Dynamics toolbox proposed by Dumas and freely available on the MathWorks File Exchange [START_REF] Dumas | 3D Kinematics and Inverse Dynamics[END_REF]. This toolbox is based on the International Society of Biomechanics' recommendations for kinematics calculation [START_REF] Wu | ISB recommendation on definitions of joint coordinate system of various joints for the reporting of human joint motion--part I: ankle, hip, and spine[END_REF]. With these recommendations, knee flexion is negative; however, it

A C C E P T E D M A N U S C R I P T
is common to define it as positive in clinical practice. For this reason, and for ease of reading, it was decided to inverse the knee flexion-extension in Figure 2.

To obtain the original data as well as the corrected data yielded by the geometrical and optimisation methods, six quality assessments were performed on hip and knee kinematics. Knee crosstalk was assessed by analysing the amplitude of knee adduction-abduction and the coefficient of determination (r²) between knee adduction-abduction and knee flexion-extension. Then, to assess the reproducibility of hip internal-external rotation, the inter-trial, inter-session and inter-subject standard deviations were investigated, in addition to the intersession-trial intraclass coefficient, following the statistical method proposed by Chia and Sangeux [START_REF] Chia | Quantifying sources of variability in gait analysis[END_REF]. For the interpretation of the intraclass coefficient, the guidelines proposed by Koo and Lin [START_REF] Koo | A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research[END_REF] were used. These values need to be calculated on session with similar number of trials, a trial corresponding to a full gait cycle. In our data set, the trials' number was different for each session and for each subject. In order to take this difference into account, a bootstrap procedure was performed [START_REF] Campbell | Bootstrapping: estimating confidence intervals for cost-effectiveness ratios[END_REF]. As the minimum of trial per session was seven, seven random trials were chosen for each session. The data analysis was then performed. This process has been done multiple times (1000) and the mean value for all the repetition used as the final value. The different parameters were calculated independently on each of 100 frames of the normalised gait cycle and the mean value with the 95% confidence interval given.

Finally, the distribution of the differences between M3 and M12 was plotted.

As all data were assumed to follow a normal distribution, a one-way repeated measure analysis of variance (ANOVA) was used to determine whether the resulting dataset (composed both of original data and of corrected data obtained with the geometrical and optimisation methods) had a significant effect on the

Results

Of the initial 118 patients in the dataset, 75 underwent M3 and M12 follow-ups (28 males and 47 females, Age: 68.4±7.1 years old, BMI: 30.1±5.5 kg.m 2 ) and were therefore included. These patients had at least 7 recorded gait cycles and a maximum knee flexion above 20° in both sessions.

Mean and standard deviation (SD) of knee 3D kinematics are given for all conditions in Figure 2. The amplitude of knee adduction-abduction and the coefficient of determination between knee adduction-abduction and knee flexionextension are given in Table 1. Both correction methods had a significant effect on the coefficient of determination between knee adduction-abduction and knee flexion-extension, and on the amplitude of knee adduction-abduction. While the coefficient of determination between knee adduction-abduction and knee flexionextension was lower in the corrected data than in the original data, it nevertheless remains above 0.5 for the geometrical method. This result is confirmed by Figure 2, which shows increased knee adduction between 60% and 100% of the gait cycle. This part of the gait cycle corresponds to the swing phase during which maximum knee flexion occurs. Similarly, the amplitude of knee adduction-abduction was lower in the corrected data than in the original data.

The lowest amplitude was obtained with the optimisation method (6.5 ± 0.2°).

The methods of correction had a significant effect on both the amplitude of knee adduction-abduction and the coefficient of determination between knee adduction-abduction and knee flexion-extension.

Regarding performance on hip internal-external rotation reproducibility, the intersession standard deviation was significantly lower when correction methods were used (Table 2). Both methods enhanced the inter-session reproducibility of hip internal-external rotation, with a lower standard deviation than in the original data 2). The ICC obtained with the optimisation method was significantly greater than the geometrical method.

Figure 3 confirms this for the distribution of the differences between M3 and M12, with an error distribution centred around 0° (geometrical: -0.5°, optimisation:

1.1°).
Both the geometrical and the optimisation methods had a positive significant effect on all the parameters evaluated. Knee adduction-abduction showed less crosstalk and hip internal-external rotation lower variability and inter-session difference.

A C C E P T E D

M A N U S C R I P T

Discussion

The purpose of this study was to propose an a posteriori geometrical method based on the position of the hip, knee and ankle joint centres to correct the thigh medial-lateral axis. Our aim was to reduce crosstalk and improve hip internalexternal rotation reproducibility. Crosstalk was reduced; however, unlike the optimisation method proposed by Baker et al. [START_REF] Baker | A new approach to determine the hip rotations profile from clinical gait analysis data[END_REF] (Dynakad), our geometrical method did not completely eliminate it. The geometrical method improved the reproducibility of hip internal-external rotation between two sessions, like the optimisation method.

In this study, crosstalk has been chosen as an indicator of knee misorientation.

Cross-talk is widely accepted as an indicator of misorientation of the thigh medial-lateral axis (considered as a surrogate of the knee flexion axis) [START_REF] Baker | A new approach to determine the hip rotations profile from clinical gait analysis data[END_REF][START_REF] Schache | Defining the knee joint flexionextension axis for purposes of quantitative gait analysis: an evaluation of methods[END_REF]. For our population with knee total replacement, it was supposed to be a reasonable hypothesis as knee prostheses are generally designed to promote an asymptomatic knee kinematics pattern. Total knee prostheses do not provide a mechanism to allow varus/valgus movement at the knee. Therefore, in the population investigated in this study, the true knee varus-valgus range of motion was expected to be close to zero degree, and cross-talk was expected to be absent. However, Cross-talk is remains an indirect measurement of the supposed misorientation of the thigh medial-lateral axis and better gold standard exists for defining medio-lateral axis such as medical imaging. However, as gait analysis is always contaminated by the soft tissue artefact [START_REF] Camomilla | Human movement analysis: The soft tissue artefact issue[END_REF] (an artefact which is typically correlated to joint flexion-extension), even with a thigh medial-lateral axis matching at best the prosthesis flexion-extension axis, an apparent cross talk could be found. In this perspective, the optimisation method logically results in

A C C E P T E D M A N U S C R I P T
an apparent amplitude of adduction-abduction. However, this adductionabduction movement becomes totally uncorrelated to the flexion-extension.

We chose to compare our method to this optimisation method because it has been described as the best functional optimisation compared to EOS imaging [START_REF] Sauret | On the use of knee functional calibration to determine the medio-lateral axis of the femur in gait analysis: Comparison with EOS biplanar radiographs as reference[END_REF]. This optimisation method was chosen also because it has been shown to perform better than marker set modification [START_REF] Schache | Defining the knee joint flexionextension axis for purposes of quantitative gait analysis: an evaluation of methods[END_REF]. In addition, in a recent reproducibility study on the hip internal-external rotation for different functional methods [START_REF] Sangeux | Computation of hip rotation kinematics retrospectively using functional knee calibration during gait[END_REF], a similar optimisation method was found to be one with the highest reproducibility for hip internal-external rotation.

Both geometrical and optimisation methods were successful in reducing the knee crosstalk observed in our dataset. However, for both coefficient of determination and amplitude of adduction-abduction, the optimisation method provided better results, eliminating almost all the crosstalk with a correlation coefficient close to zero between knee flexion and adduction. This was expected, as the objective function to be minimised is the knee adduction-abduction amplitude.

Consequently, the correlation between knee flexion-extension and adductionabduction was reduced by the optimisation process. In other words, the optimisation method, like functional methods [START_REF] Ehrig | A survey of formal methods for determining functional joint axes[END_REF][START_REF] Sangeux | Evaluation of knee functional calibration with and without the effect of soft tissue artefact[END_REF][START_REF] Schache | Defining the knee joint flexionextension axis for purposes of quantitative gait analysis: an evaluation of methods[END_REF], captures the flexionextension axis of the knee provided the flexion range of motion is sufficient. The geometrical method, while reducing crosstalk, did not fully eliminate it.

An important clinical parameter, hip internal-external rotation, unlike knee adduction-abduction, is commonly used for patient diagnosis or follow-up [START_REF] Wren | Impact of gait analysis on correction of excessive hip internal rotation in ambulatory children with cerebral palsy: a randomized controlled trial[END_REF], so ensuring good reproducibility is essential [START_REF] Brand | Comment on criteria for patient evaluation tools[END_REF]. Both the geometrical and optimisation methods tested here significantly improved the reproducibility of hip internal-external rotation relative to the original data. Considering the hip internalexternal rotation, the ICC found in our study were lower than the one obtained in a similar study [START_REF] Sangeux | Computation of hip rotation kinematics retrospectively using functional knee calibration during gait[END_REF]. They found an ICC of 0.88 for a similar optimisation method

A C C E P T E D M A N U S C R I P T
when an ICC of 0.73 was found in our study. This difference might be due to our two sessions being 9 months apart instead of being the same day as in this study [START_REF] Sangeux | Computation of hip rotation kinematics retrospectively using functional knee calibration during gait[END_REF]. It seems there is a larger variation than expected in a reproducibility study perform on the same day.

These results highlight one of our main limitation. It was supposed that the hip internal-external rotation evolution would be minimal during the 9 months between M3 and M12. However, as results suggest, it might be not true. Indeed, although no rehabilitation program was performed after M3, potential modification of gait pattern could probably occur. This can be explained by the fact that patients with knee arthrosis can have high pain level before surgery resulting in low level of activity or walking. As a result, if the surgery is successful, patient's activity level and walking distance can increase. This could allow to strengthen abductor muscles which might take more than the 3-month rehabilitation program to fully recover. As the gluteus and the tensor of fascia lata, which are the main hip abductor muscles, are also hip rotator [START_REF] Preece | The influence of gluteus maximus on transverse plane tibial rotation[END_REF] modification of the hip internalexternal rotation profile could then have been expected between M3 and M12. As a result, the lower ICC obtained in our study might be mainly due to intrinsic subject variability rather than a poorer performance of our method. We might expect better result with our geometrical method if we performed a reproducibility study with sessions during the same day or during the same week.

As the main correction of hip internal-external rotation consists in adding an offset in both methods, it is interesting to look at hip data obtained from the true bone kinematics. Using bi-plane fluoroscopy on subjects without total knee replacement, Fiorentino et al. [START_REF] Fiorentino | Soft tissue artifact causes significant errors in the calculation of joint angles and range of motion at the hip[END_REF] found that the hip internal-external rotation was comprised between -15° and 0° during gait and that the soft tissue artefact tended to shift it towards internal rotation. These values are similar both to those from our original dataset and to those obtained after correction by the From a clinical point of view, the geometrical method has some advantages over the optimisation method. As the method is based on segment geometry rather than joint kinematics and does not include any minimisation process, it should be less sensitive to soft tissue artefacts and free from overfitting issues. In addition, it does not require any movement other than a slight knee flexion. This makes it particularly appropriate in certain contexts, for instance for patients with crouch gait. However, a knee flexion of at least 20° must be reach by the patient to be able to use the geometrical method. As a result, it might not be adapted for patients with stiff-knee gait. This limitation is also valid for other knee cross-talk correction methods, like functional or optimisation methods, as they are mainly based on knee movements.

One of the main advantages of this method is also its main weakness. As the geometrical method is mainly based on segment anatomy, skeletal deformities may affect the results. Pathologies commonly analysed in a gait laboratory, such as cerebral palsy, can involve torsional deformities that could compromise the results of the geometrical method. However, where the deformity remains the same across the different sessions, hip internal-external rotation reproducibility should remain constant even if the crosstalk correction is affected. Moreover, as the geometrical method is exclusively based on joint centre location, it might be sensitive to errors on joint centres, as has been reported for the hip [START_REF] Wu | ISB recommendation on definitions of joint coordinate system of various joints for the reporting of human joint motion--part I: ankle, hip, and spine[END_REF][START_REF] Kainz | Effects of hip joint centre mislocation on gait kinematics of children with cerebral palsy calculated using patient-specific direct and inverse kinematic models[END_REF].

However, considering the large distance between the hip, the knee and the ankle, it can be assumed that any effect on the definition of the sagittal plane of the lower limb would be minimal. To conclude on this, however, would have required testing the method on patients with skeletal deformities.

Conclusion

The geometrical method proposed here to correct misalignment of the thigh medial-lateral axis performs similarly to an optimisation method in improving hip internal-external rotation reproducibility, but not in fully eliminating knee crosstalk.

The main advantage of the geometrical method includes the fact that it is based on simple segment geometry, while the optimisation method is based on a more complex minimisation process. Moreover, if the main objective of a posteriori correction of gait data is to obtain repeatable hip internal-external rotation, the geometrical method may be an alternative choice since it is more widely applicable, including in situations where the patient's range of motion is limited.

However, if the main objective is to correct crosstalk, optimisation methods should be chosen provided there is a sufficient range of motion.

  adduction variance. An existing dataset was used: 75 patients with a knee prosthesis undergoing gait analysis three months and one-year post-surgery.

  Knee crosstalk, Clinical gait analysis, Hip internal-

  above quality criteria. If a significant difference was found, a paired t-test was performed with significance level set at p<0.01.

  significant lower values than the geometrical method. Considering the ICC, both the geometrical and the optimisation improved the reproducibility of the original data from poor (ICC = 0.34) to a moderate reproducibility with a respective mean ICC of 0.64 and 0.72 (Table

  geometrical method. Conversely, the optimisation method seems to result in hip internal-external rotation being more shifted toward external rotation.

  for lower extremity orthopaedic surgery: Baseline data from a randomized controlled trial

  Figures:

Figure 3 :

 3 Figure 3:

Figure 1 :

 1 Figure 1:

Figure 2 :

 2 Figure 2:

Figure 3 :

 3 Figure 3:

Acknowledgements:

We thank Katia TURCOT and Yoshimasa SAGAWA JUNIOR for their contributions to data acquisition.

Alexandre Naaim (Investigation) (Formal analysis) (Methodology) (

Conflict of Interest Statement

The authors certify that there is no conflict of interest or any financial and personal relationships with other people or organisations that could inappropriately influence this work

Table 1: Effect of the correction methods on knee crosstalk

Original data

Mean (95 CI%)

Geometrical method

Mean (95 CI%)

Optimisation method

Mean (95 CI%)

Original data

Mean (95 CI%)

Geometrical method

Mean (95 CI%)

Optimisation method

Mean (95 CI%)

Variance

Inter-subject 6.7°(6.4°-6.9°) 7.8° (


Anthony K Cheetham

Thomas D Bennett

François-Xavier Coudert

Andrew L Goodwin

Defects and disorder in metal organic frameworks

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

It is widely recognized in physics and chemistry that the presence of both point and extended scale defects in materials can have a dramatic impact on their properties. In the case of inorganic structures, for example, simple lattice and interstitial vacancies control diffusion and ionic conductivity, while substitutional defects (i.e. impurities and dopants) in semiconductors have a strong influence on electronic conductivity. Dopants can also determine the transport properties of organic materials such as conducting polymers. Point defects in porous materials such as zeolites and carbons affect the chemical nature of the internal pore surface, thus altering in large ways the host-guest interactions (i.e. hydrophobic or hydrophilic nature) and promoting catalysis. On larger length scales, defects such as stacking faults and dislocations have an important bearing on a material's mechanical properties and catalytic activity, and can be tuned to optimize performance. Given the relative youth of the field of metal-organic frameworks (MOFs), it is not surprising that relatively little is known about the identity or propensity of their defects, though a very recent review by Fang et al. 1 gives a good coverage of the literature from a crystal engineering perspective, with a particular emphasis on porous MOFs and their applications. The purpose of this short review is to explore the interplay between defects and disorder in both porous and dense MOFs.

We shall focus primarily on defects that are introduced artificially, since these have been quite widely studied. We know, for example, that defects can be used to introduce new functionality into MOFs, e.g. photoluminescence, active sites for catalysis, and enhanced selective adsorption, as well as for fine-tuning properties such as magnetic and ferroelectric phase transitions. It is however also known that the presence of defects may undermine the performance of MOFs, as in the case of porous solids where selective adsorption capacity can be reduced by poor crystallinity.

It is important to recognize that defects are normally associated with disorder, and so entropic factors are a major driving force in determining the levels of defects in solids. All materials under ambient conditions -however pure they may be -contain intrinsic defects according to the partition function:

𝑛 𝑑 ~ 𝑒 -𝑒 𝑑 𝑅𝑇 ⁄ (1) where n d is the number of defects of a particular type and e d is the energy required to form them. The concepts set out for simple ionic solids by Schottky and Wagner in 1930 2 pointed to the importance of Schottky (equal numbers of complementary cation and anion vacancies) and Frenkel (equal numbers of cation or anion vacancies and interstitials) defects in stoichiometric inorganic solids. These are known as intrinsic defects. Each particular solid normally has a preferred intrinsic, point defect incorporation mechanism; in the case of sodium chloride, for example, the dominant defect is of the Schottky type, and we can estimate that a mole of NaCl contains approximately 10 4 cation and anion vacancies at room temperature (and as many as 10 17 at 1000 K). A solid will also contain extrinsic defects due to impurities, and the relative proportions of intrinsic and extrinsic defects will depend on the purity of the sample as well as the temperature. By contrast, we do not know the nature of the intrinsic defects in MOFs.

The scope of this mini-review will span both porous and dense MOFs, recognizing that the ease of defect formation may be different. We shall also consider the correlation of such defects, along with the relationship between defects and disorder (section 4), and in particular the extreme case of disorder that is amorphization (section 5). Defects and disorder caused by extraframework guests or solvent molecules, which are ubiquitous in porous MOFs, and extended defects such as dislocations and stacking faults, which are covered in reference 1, are avoided.

The following two sections will give examples of defects on the cation and anion sub-lattices, respectively.

Cation defects in metal-organic frameworks 2.1 Examples with metal cation substitution

There are two contrasting synthetic approaches to the formation of substitutional cation defects in MOFs. One approach aims to crystallize the materials from starting solutions containing the two cations in the desired proportions. ), where crystallization from a solution containing the two cations, in the presence of the appropriate tartrate ligand, is possible.

(1-x)Ca 2+ (aq) + xSr 2+ (aq) + C 4 H 4 O 6 2-(aq) Ca 1-x Sr x (C 4 H 4 O 6 ) Scheme (1)
Alkaline earth tartrates can be made in this manner and provide one of the simplest cases of cation substitution in dense MOFs. 3 Calcium and strontium L-tartrates are isomorphous and form a complete solid solution, Ca 1-x Sr x (L-C 4 H 4 O 6 ), which obeys Végard's Law, i.e. the cell volume is a linear function of composition (Fig. 1). Complexities arise where, as in the corresponding strontium-barium system, two end-member structures are different. In the case of the meso-tartrate phases, there is a solid solution over a very limited range of composition, Sr 1-x Ba x (meso-C 4 H 4 O 6 ) for 0<x<0.1, followed by a two phase region (0.1<x<0.2) in which this anhydrous meso-tartarte, Sr 0.9 Ba 0. Reactions such as those illustrated in scheme (1) works well in some instances, but can be problematic when one of the end-member phases crystallizes more rapidly than the solid solution, resulting in an inhomogeneous product. In such circumstances, a second approach involving post-synthetic cation exchange (PCE) might be adopted. There are two variations on this theme. In the first case, the two end-member MOFs are prepared separately and the two solids are then allowed to undergo cation exchange in solution:

(

1-x)M(L) + xM'(L)  M 1-x M' x (L) Scheme (2)
In a second variation on the PCE approach, only one of the end-members is prepared and this is allowed to undergo ion-exchange with a solution containing the second cation:

(1-x)M(L) + xM'(aq)  M 1-x M' x (L) + xM(aq) Scheme (2')
These two PCE approaches has been successfully used to synthesize mixed (Al/Fe) samples of MIL-53 and mixed (Zr/Ti) samples of UIO-66, respectively. 4 The use of cation solid solutions has been used to control adsorption properties in porous MOFs. 5 By exploiting the reactivities of different precursors to form mixed cation systems according to scheme (1), it has been possible to form both homogeneous solid solutions, on the one hand, and phase separated materials on the other. This was achieved in a MOF system based upon manganese and zinc with 5-nitroisophthalate and 4,4′-bipyridyl as linkers. Classical solid solutions are obtained when using sodium 5-nitroisophthalate as a precursor because it reacts very rapidly, but phase-separated products are found when the 5-nitroisophthalic acid is used (Fig. 2). The two different samples showed strikingly different adsorption behavior with methanol. A second approach to the realization of mixed-metal MOFs leverages the ability to substitute one cation for another into an existing MOF, a process called transmetalation, 6 that can create new functionality. It has been exploited quite extensively for optical materials where the levels of substitution that are needed are usually quite small. Rare-earth ions, especially Eu 3+ and Tb 3+ , have been inserted at low levels (typically <3%) into Y 3+ , Gd 3+ and Bi 3+ containing host frameworks to create photoluminescent materials that are suitable for use as phosphors. In an early example in glutarate (C 5 H 6 O 4

3-

) frameworks, Eu 3+ was substituted for Gd 3+ in Gd(C 5 H 6 O 4 ).nH 2 O phase and the photoluminscent lifetime studied as a function of the level of water in the MOF channels. 7 One of the attractions of MOFs for photoluminescence applications is that the organic ligands themselves can be used as sensitizers to absorb the excitation, as in the case of the Eu 3+ -and Tb 3+ -doped sodium/bismuth 1,4-benzenedicarboxylates, which show very good quantum efficiencies (Fig. 3). 8 Transmetalation, sometimes also called "post-synthetic metal exchange", 9 has also been used in the area of gas separation and storage to control adsorption and transport properties in porous MOFs. One such example is the use of Ti-exchanged UiO-66 [Zr 6 O 6 (OH) 4 (BDC)] to increase gas permeability in mixed matrix membranes. 10 It has also been shown that metal exchange can be used to obtain materials with topologies unreachable through solvothermal synthesis. One example is the case of MOFs based on the N,N'-bis(3,5-dicarboxyphenyl)pyromellitic diimide (H 4 BDCPPI) linker. Solvothermal synthesis yield materials with nbo and pts nets respectively, but metal exchange from the copper-based material allowed Prasad et al. to obtain a Zn 2 (BDCPPI) framework which retained pts topology. 11 However, many systems that are superficially similar may show complex phase behaviour in reality. Transition metal succinates, for example, are notoriously fickle and yield complex results. In the case of the cobalt and nickel succinates (

Amine cations in hybrid perovskites: substitution and disorder

Cations that can undergo substitution are not restricted to metallic ions. The hybrid perovskites, which some may consider MOF-type materials, of general formula [AmH]M II X 3 , where Am is an organic amine cation and X can be a wide range of anions, e.g. formate, azide, halide, borohydride, provide important examples (Fig. 5). The [AmH]Pb II I 3 frameworks are of great current interest because of their outstanding performance as semiconductors in solar cells. 13,14 The most widely studied system is [CH 3 NH 3 ]Pb II I 3 , and it is possible to tune the band gap by partially substituting the methylammonium cation by formamidinium, thereby increasing the efficiency to greater than 14%. 15 This fine-tuning is normally achieved during the initial synthesis according to scheme (1), rather than by PSE.

Halide perovskites also exhibit disorder of a different nature: order-disorder transitions associated with hydrogen bonding between the amine hydrogens and the iodide anions.

[CH 3 NH 3 ]Pb II I 3 itself, for example, is cubic above 330 K, tetragonal between 330 K and 160 K, and orthorhombic below 160 K, due to progressive ordering of the methylammonium cations on lowering the temperature. 16 However, unlike the analogous formate perovskites, the low temperature, orthorhombic phase of [CH 3 NH 3 ]Pb II I 3 is centrosymmetric and is therefore not a ferroelectric. 17 Aside from the lead halide systems, the most widely studied class of the hybrid perovskites are the formates of general formula [AmH]M II (HCOO) 3 , 20 which show a range of interesting multiferroic behavior. 21 Order-disorder transitions associated with hydrogen bonding again (Fig. 5) play a central role in determining the ferroelectric properties of these interesting systems. 22 It has been found recently that the formation of amine cation solid solutions of the type [AmH 1- x Am'H x ]]M II (HCOO) 3 ,can be used to tune the ferroelectric phase transition, though there are outstanding questions concerning the distribution of the two amines within individual grains and crystals. 23

Mixed valence and cation vacancies in Metal-Organic Frameworks

Classical mixed valence behavior is rare in MOFs, and those cases that are known tend to have rational compositions where valence ordering can be achieved. The perovskite-related metal formate composition [NH 2 (CH 3 ) 2 ]Fe II Fe III (HCOO) 6 is typical and has interesting magnetic properties due to the charge ordering of Fe(II) and Fe(III). 24 Similarly, it appears to be difficult to obtain non-stoichiometric MOFs where charge balance might be achieved by forming simple cation vacancies (vac). Thus, while mixed valence Fe 1-x vac x O, which adopts the NaCl structure, is well known and achieves charge balance by combining cation vacancies with Fe(II) and Fe(III), there are no well-documented examples of analogous MOFs, e.g. Fe 1-x vac x L 2-. Nor does it seem possible to create cation vacancies by partial replacement of, say, a monovalent cation by a divalent one to form frameworks of the form Li 2-x Mg x/2 vac x/2 (L). This is unfortunate because strategies such as this are used in lithium solid electrolytes to enhance cation mobility by creating vacancies into which the Li + ions can move.

Ligand defects in metal-organic frameworks

The replacement of one organic ligand by another in MOFs is often more challenging than cation substitution because it is generally more difficult to match the coordinative and spatial requirements of the substituted ligand than it is to match a simple cation. In this respect, porous MOFs are likely to be more accommodating towards ligand substitution and there has been extensive work in this area. Dense MOFs, on the other hand, have less space to accommodate differences in size.

Examples with anion or ligand substitution in porous MOFs

As with cation substitutions, partial anion or ligand substitution can be carried out during the initial synthesis or post-synthetically. We shall focus first on the latter category, and in particular on reactions that do not involve the replacement of the main linkers. Not surprisingly, it can be straightforward to replace ligands that are pendant rather than bridging. Thus, early work on layered phosphonates showed that it was possible to replace up to 60% of the phenylphosphonate ligands by phosphite without changing the layer spacing but nevertheless modulating the intercalation chemistry. 25 More recently, HKUST-1, a MOF that is formed from copper(II) paddlewheel dimers linked by 1,3,5-benzenetricarboxylate ligands, showed that the axial water molecules on the copper dimers could be replaced post-synthetically by pyridine, while the pyridine form could not be obtained by direct reaction. 26 The removal of water ligands to create under-coordinated cation centers can be particularly advantageous for binding dihydrogen in hydrogen storage applications. 27 Some of the first systematic studies of ligand functionalization in MOFs involved work by Cohen on post-synthetic modification (PSM) of 1,4-benzenedicarboxylate linkers in a series of common MOFs. 28,29 The linker itself was not replaced, but aromatic substituents such as -NH 2 were exchanged for -CONH 2 by means of PSM (Fig. 6). Alternatively, additional functionality can be added to the aromatic linkers by introducing other reactants into an activated MOF. For example, Cr(CO) 6 will react with benzene-containing linkers to attach Cr(CO) 3 groups which can then undergo further reactions and create gas binding sites by decarbonylation. 30 Figure 6. A schematic representation of post synthetic modification in porous MOFs. Reproduced from Ref. 29 with permission from The Royal Society of Chemistry.

In a recent and very interesting variation on the above theme, the complete organic linker was exchanged in a process termed "Solvent Assisted Linker Exchange" (SALE). 31 It was applied to the zeolitic imidazolate framework, ZIF-8 [Zn(C 4 H 5 N 2 ) 2 ] The use of SALE enabled the exchange of the organic linker (2-methylimidazole) for a smaller one (imidazole, C 3 H 3 N 2 ). The resultant framework, SALEM-2, possessed very different sorption properties due to the opening of the pore-connecting apertures within the framework. Further functionalization using n-butyl lithium created active sites for BrØnsted base catalysis that could not be achieved in the parent ZIF-8 (Fig. 7). Here again, as in transmetalation, post-synthetic linker substitution allows the synthesis of novel MOF materials with desirable function-property combinations, which could not be attained through direct solvothermal synthesis. Indeed, it has been shown, by computational studies on the family of zeolitic imidazolate framework (ZIF) materials, that the thermodynamic stability of various polymorphs does not uniquely determine their experimental feasibility and accessibility through solvothermal synthesis. 32,33 Figure 7. Conversion of ZIF-8 to SALEM-2, followed by its functionalization with n-butyl lithium. Reprinted with permission from O. Karagiaridi et al,, J. Am. Chem. Soc.134, 18790 (2012). Copyright 2012 American Chemical Society.

While the above examples have focused on cases where the modification of the MOF is done post-synthesis, there has been extensive work on systems where mixed ligands are incorporated during the initial synthesis (Scheme 1). One of the most detailed studies concerns the substitution of multiple combinations of substituted benzene dicarboxylate (BDC) -based ligands into the MOF-5 structure by Yaghi and co-workers. 34 The scope is illustrated in Figure 8, which shows some of the many combinations of these so-called multivariate MOFs (in one case, as many as 8 different substituted linkers were introduced). As with other strategies for ligand substitution, one of the objectives is to create novel functionalities within a known structure type. In this particular work, it was possible to substantially enhance the CO 2 adsorption capacity by appropriate choice of the substituents on the 1,4-BDC linkers. Work by Kitagawa and others focused specifically on flexible frameworks, and endeavored to tune the flexibility in order to optimize sorption performance. 35,36 In one example, two doubly interdigitated frameworks -[Zn(5-NO 2 -ip)(bpy)(0.5DMF•0.5MeOH)] and [Zn(5-MeOip)(bpy)](0.5DMF•0.5MeOH), where ip = 5-nitroisophthalate and bpy = 4,4'-bipyridyl were prepared, giving two closely related structures with similar lattice parameters. However, the guestfree structures were very different after desolvation. The former shows a porous to non-porous transition on degassing (and gate-opening on adsorption), whereas the latter remains in the open form. Ligand-based solid solutions were then prepared and the two substituents appeared to be homogeneously distributed in the as-synthesized samples. The adsorption properties of the degassed samples were then explored and it was found that the gate-opening transition could be fine-tuned by adjusting the composition of the solid solutions, giving rise to enhanced performance for the adsorption of CO 2 from CO 2 /CH 4 mixtures.

Examples of anion substitution in dense MOFs

Recent work on lithium succinates and related systems 37 illustrate what can be achieved in this area. The three dense MOFs -lithium succinate, lithium hydroxysuccinate (i.e. malate) and lithium methylsuccinate -turn out to be isostructural, in spite of the different sizes of their ligands. There is no solvent-accessible volume, but the different ligands can still be accommodated; indeed, the Young's modulus of the succinate is only 20 GPa, which is comparable to many porous MOFs. Mixed samples were synthesized by mechanical milling in order to avoid the problems with differential crystallization rates, and complete solid solution was found throughout the ternary phase diagram (Fig. 9a). This was confirmed Le Bail refinement of the 1/3, 1/3, 1/3 composition using high resolution synchrotron X-ray diffraction. Surprisingly, even the tetrafluorinated succinate ligand can be incorporated into the succinate structure and a complete solid solution can be obtained (Fig. 9b). A 50:50 sample in the lithium succinate-tetrafluorosuccinate solid solution was analyzed in greater detail in order to confirm that the two ligands are homogeneously distributed. Both full Rietveld refinement with synchrotron PXRD data (Fig. 9b), infrared analysis, and spin diffusion solid-state NMR measurements confirmed that the two ligands were well mixed at the molecular level. 37 Interestingly, the flexibility of lithium succinate and related phases provides a rare example of solid solution formation by post-synthesis modification in a dense MOF. Thus, in the specific case of lithium hydroxysuccinate (i.e. malate), it is possible to perform a topotactic dehydration across the C-C bond to form the corresponding fumarate phase. 38 In fact, 80% of the water can be removed from a crystal without loss of crystallinity, and the process has been followed by in situ single crystal X-ray diffractometry. Density functional theory (DFT) calculations show that the fumarate thus formed is a metastable phase, that is 12.7 kJ/mol less stable than the normal form.

Other types of anion defects in MOFs

The above sections have dealt with conventional ligand defects where one ligand, typically a bridging anion, is replaced by another one. There are, however, other possibilities, such as replacing a bridging ligand by two monodentate ones, and this type of defect has been incorporated into porous, pillared diphosphonates, including the widely studied zirconium diphosphonates. 39 For example, in the case of Al 2 [O 3 PC 2 H 4 PO 3 ](H 2 O) 2 F 2 , it has proved possible to replace some of the ethylene diphophonic acid pillars by two phosphite groups to form Al 2 [(O 3 PC 2 H 4 PO 3 ) 1-x (HPO 3 ) 2x ](H 2 O) 2 F 2 solid solutions. Up to 32% of the pillars can be replaced and the distribution of the defects appears to be homogeneous. Such tuning enables the adsorption and desorption properties of the MOF to be modulated. 40 The same strategy has been used to replace a dicarboxylate such as 1,4-BDC by monocarboxylate anions, thus removing the bridge between the cations and creating two dangling ligands. 41,42 This has been reported for UiO-66(Zr or Hf), a MOF that contains octahedral zirconium (or hafnium) clusters that are linked by 1,4-BDC ligands to form a highly porous structure. It has recently been shown that pairs of trifluoroacetate (TFA) groups can replace individual 1,4-BDC linkers and that such substitution results in enhanced Lewis acidity for catalytic reactions. 43 In an interesting extension of this theme, it has also been demonstrated that the removal of linkers by such substitution can enhance the mechanical stability of UiO-66, as shown by the improved resistance to amorphization during milling. 44 This rather counter-intuitive behavior is ascribed to the strengthening of the zirconium-carboxylate bonds due to the local electron withdrawing effect of the TFA.

Domain structures in MOFs

To a first approximation, the various defects described above are generally thought to be randomly-distributed throughout a given framework. In conventional inorganic solids, such a picture is only sometimes true. For example, Schottky defects do not correlate strongly in NaCl but can be so robustly ordered in transition metal oxides that in the case of NbO they even break the lattice symmetry. 45 Understanding the extent to which defects are correlated in MOFs may prove important for a number of reasons. Wherever defects are associated with transport properties (e.g. acid sites for proton conductivity, or vacancy sites for gas storage), percolation will depend critically on the existence and nature of correlated defect positioning. Moreover, the dynamical and elastic properties of MOFs are also sensitive to the presence of domain structures, which influence directly the underlying phonon dispersion. And, although a significant challenge, the controlled arrangement of multiple defects might eventually enable design of protein-like "active sites" for catalytically-active MOFs. 46 Perhaps the clearest indication that MOFs can support correlated defects comes from the structural chemistry of the UiO-66 family. 47 The parent compound is usually prepared under basic conditions from a mixture of ZrCl 4 and H 2 bdc, but addition of monocarboxylate "modulators" leads to a defective framework structure in which some of the bdc linkers have been replaced by modulator pairs [Fig. 10]. 48 Varying the modulator chemistry and concentration determines the density of defects introduced. Because these defects must always occur in pairs, it has always been known that there must be at least local correlations between defect sites. However, most of the experimental and computational approaches used to characterise these defects are insensitive to the longer-range defect correlations associated with domain formation. 48,49 Even detailed singlecrystal X-ray diffraction studies have determined only the total defect concentration and the geometry of individual defect sites. 50 This is because conventional single-crystal analysis is insensitive to long-range correlations if those correlations are not sufficiently strong to force periodic patterning (e.g. of defects). Instead the experimental methods most sensitive to defect correlations are diffuse scattering measurements -X-ray, neutron, or electron; powder or singlecrystal and electron microscopy. 51 These are precisely the methods which have informed much of our understanding of collective defect structures in conventional inorganic materials such as zeolites and transition-metal oxides. 52,53 We note that electron microscopy is the least translatable of these methods to the study of MOFs since so many systems are terminally sensitive to the electron beam. 54 Structured diffuse scattering characteristic of defect nanodomains was first observed in the Hfcontaining analogue UiO-66(Hf), prepared using high concentrations of formic acid modulator. The X-ray diffraction patterns of these samples include a series of diffuse scattering peaks at positions forbidden by the symmetry of the parent UiO-66 structure (space group F43m). 55 That this diffuse scattering is associated with defects rather than solvent ordering as originally suggested is evident in their persistence on thermal desolvation. Intriguingly, defect structure models which consider only linker/modulator substitution cannot reproduce the observed diffuse scattering intensities. Instead a model in which vacancies at the entire [Hf 6 O 4 (OH) 4 ] cluster site are correlated to form defect nanoregions with reduced symmetry (space group P43m) quantitatively accounts for the diffuse scattering [Fig. 11]. 56 There is a strong analogy to the Schottky defect structure of the transition-metal monoxides with the cluster vacancy arrangement in defect nanoregions mapping directly onto the Nb vacancy sites in NbO. 57 This model was supported by electron diffraction, X-ray pair distribution function and anomalous X-ray diffraction measurements, the latter showing conclusively that defects in UiO-66(Hf) involve modulation in Hf concentrations. 56 Cluster vacancies are not unique to the Hf-containing UiOs. Surface area measurements for a number of defective Zr-containing UiO-66 variants are consistent only with the existence of [Zr 6 O 4 (OH) 4 ] cluster vacancies. 41 Likewise, doping of UiO-66(Zr) with Ce gives rise to a series of defective frameworks whose diffraction patterns contain the same characteristic diffuse scattering peaks observed for UiO-66(Hf). 58 As in the Hf MOFs, this scattering persists on thermal desolvation suggesting that it also arises from nanodomains of correlated cluster vacancies. An extreme example of correlated defects in this structural family has recently been discovered in a 2sulfoterephthalate-bridged Zr-containing UiO-66 analogue. 59 The structure of this sulfonated MOF is characterised by a (~40 Å) 3 unit cell in which 3/16 of the Zr cluster sites are vacant. The vacancy ordering observed experimentally can be considered a periodic intermediate lying between the defect-free UiO-66 parent and the P43mdomain structure of UiO-66(Hf), where the cluster vacancy fraction is 1/4 [Fig. 12]. Already there are indications that these correlated defect structures in UiO frameworks result in interesting physical and chemical properties. In Ce-doped UiO-66(Zr), linker vacancies are coupled to Ce 4+/3+ reduction, which in turn promotes catalytic performance in methanol oxidation. 58 A similar conclusion was reached in Ref. 41 , where defect sites promoted uptake of catalytically-active Cr 3+ centers during post-synthetic modification. The fractal-like network of exposed defect sites in the sulfonated MOF of Ref. 59 is thought to be responsible for its extremely high proton conductivity. And, finally, the concentration of correlated cluster vacancies in defective UiO-66(Hf) samples was shown found to influence their thermal expansion behaviour, which can include the desirable property of "colossal" negative thermal expansion. 60 It is not yet clear whether strongly-correlated defects are unique to the UiO family of MOFs. Certainly the strong X-ray scattering contrast of absent vs present [M 6 O 4 (OH) 4 ] clusters means that defect correlations have a clearer experimental signature for this system than is likely for other defective MOF families. Nevertheless there are at least three reasons why the UiO family might be particularly prone to defect inclusion and then to the existence of correlations between those defects to form domain structures. First, the parent network has a very high connectivity, which means that linker/cluster vacancies can be incorporated while maintaining mechanical stability. Second, the unit cell parameters calculated for defect-free and defect structures differ by only 0.05%, and so defects can be incorporated without significant lattice strains. 59 Third, the core [M 6 O 4 (OH) 4 (OR) 12 ] building unit is susceptible to symmetry-lowering distortions that may play a role in driving defect correlations. The idealised T d point symmetry of the cluster lowers spontaneously to D 2d (≡ tetragonal 42m ) in the crystal structures of non-bridging cluster packings (e.g. M = Hf, R = CH 3 ; Refs. 56,61 ) in a way that is reminiscent both of symmetry lowering in the transition-metal acetate paddle-wheels and indeed of Jahn Teller effects in general [Fig. 13]. 62 This susceptibility to symmetry lowering suggests that modulator substitution at the four equatorial sites of a [M 6 O 4 (OH) 4 (OR) 12 ] cluster during MOF growth allows local relaxation. Then, because this local distortion can only be propagated on the NbO-type defect lattice, [START_REF] Wells | Structural Inorganic Chemistry[END_REF] the experimentally-observed domain structure emerges naturally. Our understanding of the microscopic driving forces responsible for defect correlations in MOFs would be improved enormously by an increased diversity of experimentally-characterised systems for which domain structures occur. The increased application of diffuse scattering and electron microscopy methods to the study of MOFs will be critical in this sense. Indeed, experience suggests that the presence of structured diffuse scattering in MOF single crystal diffraction studies is probably more widespread than has been reported in the literature. A recent study of the relationship between correlated defect structures relevant to MOF networks and the corresponding diffuse scattering patterns [START_REF] Overy | [END_REF] demonstrates that the signature of domain structures in the parent Bragg reflections can often be subtle, but the existence of characteristic diffuse scattering may prove invaluable in diagnosing correlated disorder of the type discussed here. A number of known MOF structures, such as the intriguing example of "partial interpenetration" in NOTT-202, 65 may benefit from reanalysis in this context.

Amorphous solids, liquids and melt-quenched glasses

Arguably the clearest examples of disordered materials, those which lack any long range order (i.e. those which are amorphous) are easily identifiable from the absence of Bragg peaks in their X-ray diffraction patterns. In the context of MOFs and coordination polymers, such structures can, like cation substitution in solid solutions, be formed through (i) alteration of initial synthetic conditions, or (ii) through post synthetic introduction of disorder.

Metal-organic frameworks

Fast crystallization kinetics lie behind the propensity of MOF precursors to self-assemble into highly ordered networks during synthesis. 66 Highly disordered products, such as amorphous solids, are hence difficult to form directly, though differential formation of crystalline and amorphous states of a zeolite-related MOF of composition Zn(C 4 H 3 N 2 O) 2 can be effected by altering the rate of addition of a structure directing agent to the initial metal and organic containing solution. 67 More commonly, heating or mechanical impact are used to irreversibly introduce disorder into crystalline hybrid materials.

The application of mechanosynthesis to crystalline, non-solvated zinc imidazolate frameworks (ZIFs) results in formation of stable amorphous network arrangements. 68 However, a recent example on the in-situ ball-milling of ZIF-8 in an aqueous acidic environment, 69 found not only the expected amorphization, but subsequent recrystallization of the amorphous network upon continued milling, to a previously unidentified topology. The metastable nature of some amorphous MOF phases is also evidenced in the crystalline-amorphous-crystalline transition sequence upon heating of ZIF-4, 70 a three-dimensional framework of Zn(C 3 H 3 N 2 ) composition formed of tetrahedral Zn 2+ nodes connected by bidentate imidazolate ligands. Here, after formation of an amorphous solid at 300 °C by heating of the crystalline phase, recrystallization to the dense 'ZIF-zni' MOF is observed at 450 °C. In this case, knowledge of the chemical composition of the amorphous phase (the transition sequence not being accompanied by mass loss), allowed total scattering data on the amorphous phase to be modeled by Reverse Monte-Carlo techniques. 70 Furthermore, whilst MOFs are usually observed to decompose to oxides at high temperatures, the dense ZIF-zni was observed to undergo melting at 580 °C; 71 the first example of the melting of a three-dimensional MOF. This observation was rationalized in terms of the accessibility of the intrinsic melting point of ZIF-zni, made possible by the raising of the point of thermal decomposition to over 600 °C, by excluding air from the heating process.

Cooling of the liquid back to room temperature yielded a glass, which contained both tetrahedral Zn 2+ centers and Zn-Zn pairs separated by bidentate ligands, reminiscent of the crystalline phase (Fig. 14). The different network connectivity of crystal and glass phases suggested a reconstructive transition during melting and cooling, due to Zn-N dissociation in the disordering mechanism. Such dissociation would, in turn, infer ionic liquid formation upon melting, though no investigations into this phase were performed. An alternative explanation, analogous to that used to explain the melting (at much higher temperatures) of inorganic zeolites, involves dihedral angle changes.

1D and 2D coordination polymers

Lower dimensionality hybrid structures have also been observed to melt. The one dimensional coordination polymer, [Zn(HPO 4 )(H 2 PO 4 ) 2 ].2C 3 H 5 N 2 , consists of chains of four-fold coordinated Zn 2+ centers linked by phosphate ligands, along with charge-compensating protonated imidazole species. At the same time as the example presented above, work on this system revealed a solidliquid transition at 150 °C, in a process set off by the excessive disordering and eventual dissociation of one Zn-O bond in the Zn 2+ coordination sphere. 73 Cooling led to glass formation, with dynamic mechanical analysis, 31 P nuclear magnetic resonance spectroscopy, pair distribution function (PDF) and X-ray absorption spectroscopy studies (XAS) on the crystalline and glass solids confirming the absence of Zn-O association in the liquid state. Furthermore, the glass itself was deduced to have a one-dimensional structure, like that of the crystal. Work on another 1D polymer incorporating octahedral Zn 2+ centers, [Zn 3 (H 2 PO 4 ) 6 (H 2 O) 3 ].C 7 H 7 N 2 , also led to liquid formation, though at a lower temperature of 100 °C . In this case however, significantly faster recrystallization rates of the resultant ionic liquid prevented glass formation at room temperature.

In possessing similar immediate inorganic coordination environments in both crystal and glasses, the above systems share common features with other glass formers, including zeolites. 74 A system of intermediate two dimensional connectivity, [Zn(H 2 PO 4 ) 2 (HTr) 2 ] n , 75 which melts at 184 °C, does however undergo network connectivity changes on the solid-glass transformation. In the structure, each organic ligand equatorially bridges Zn 2+ ions to form layers, with an octahedral coordination environment completed by non-bridging phosphate ions in the axial positions. The process again involved bond dissociation, however, unusually, reformation of an extended network was not observed upon glass formation. PDF and XAS studies were used to derive a molecular model for the end glass state, a transition driven by the preference of Zn 2+ ions for a 4-fold coordination environment.

Concluding Remarks

The concepts of defects and disorder are inextricably linked with one another. Point vacancies, substitutional defects and long-range disorder may all be introduced into MOFs pre-, or post synthetically. Whilst at present still system dependent, notable studies of changing both the physical and chemical behavior of known frameworks using the methodologies presented here exist. Such work hints that perhaps we might make equal, if not greater, advances in the field by looking to change the properties of existing structures in a controlled manner, as opposed to synthesizing new structures and characterizing their properties.

Figure 1 .

 1 Figure 1. (a) The X-ray patterns of Ca 1-x Sr x (L-C 4 H 4 O 6 ) show systematic shifts as a function of composition. (b) the unit cell volume obeys Végard's Law. (c) the structure of Ca (L-C 4 H 4 O 6 ) viewed along the a axis. Ca -blue, Ored, C -black, H -white. Adapted with permission from L. Appelhans et al., J. Amer. Chem. Soc.131, 15375 (2009). Copyright 2009 American Chemical Society.

Figure 2 .

 2 Figure 2. (a, b) Different distributions can be seen in the X-ray powder patterns. (c) Schematic showing the formation of solid solutions and phase-separated products. Adapted with permission from T. Fukushima et al,, J. Am. Chem. Soc.134, 13341 (2012). Copyright 2012 American Chemical Society.

Figure 3 .

 3 Figure 3. (a) Unit cell of sodium/bismuth 1,4-benzenedicarboxylate, which, when doped with Eu 3+ -and Tb 3+ -shows (b) excellent luminescence in the red and green, respectively, when excited in the ultra violet. Adapted with permission from A. Thirumurugan, J.-C. Tan and A. K. Cheetham, Cryst. Growth & Design, 10, 1736 (2010). Copyright 2010 American Chemical Society.

  two end-members have different compositions and structures, Co 5 (OH) 2 (C 4 H 4 O 4 ) 4 and Ni 7 (OH) 2 (H 2 O) 2 (C 4 H 4 O 4 ) 6 .2H 2 O, respectively, when crystallized under the same conditions. At intermediate Co/Ni ratios, yet other compositions and structures are obtained: M 4 (OH) 2 (H 2 O) 2 (C 4 H 4 O 4 ) 3 .2H 2 O and M 7 (OH) 2 (H 2 O) 2 (C 4 H 4 O 4 ) 6 .3 . 5H 2 O for 25% and 75% nickel, respectively (Fig.4), with disordered cations in each case.12

Figure 4 .

 4 Figure 4. The cobalt/nickel succinates form four different structures with entirely different compositions when crystallized under the same conditions. Reprinted with permission from C. Livage et al, Angew. Chemie Intl. Ed. 46, 5877 (2007). Copyright 2007. John Wiley and Sons.

Figure 5 .

 5 Figure 5. (a) Structure of the fully ordered hybrid perovskite, [(CH 3 ) 2 NH 2 ]Cu II (HCOO) 3 . Reproduced from Ref. 18 with permission from The American Physical Society (2013). (b) the hydrogen bonding between the amine cation and the iodide anions in the A-site cavity of [CH 3 NH 3 ]Pb II I 3 . Reproduced from Ref. 19 with permission from The Royal Society of Chemistry (2015).

Figure 8 .

 8 Figure 8. Various permutations of 1,4-BDC derivatives in the MOF-5 structure type. From H. X. Deng et al, Science, 327, 846 (2010). Reprinted with permission from AAAS.

Figure 9 .

 9 Figure 9. (a) Ternary phase diagram shows solid solutions from lithium succinate and related phases. (b) The succinate-tetrafluorsuccinate system obeys Vegard's Law throughout its composition range. Reprinted with permission from H. H. M. Yeung et al, Angew. Chemie Intl. Ed. 52, 5544 (2013). Copyright 2013. John Wiley and Sons.

Figure 10 .

 10 Figure 10. The structure of non-defective UiO-66 is comprised of [ZrO 4 (OH) 4 ] clusters connected by terephthalate linkers. The metal-containing clusters of the resulting framework are arranged on a face-centred cubic lattice. (b) Inclusion during framework synthesis of monocarboxylate modulators, such as formate as shown here, can lead to correlated linker vacancies where a single terephthlate linker is replaced by two monocarboxylates in an opposing geometry.

Figure 11 .

 11 Figure 11. (a) Reproduced from reference 56 . The X-ray powder diffraction pattern of d-UiO-66(Hf) contains a series of weak diffuse reflections that are forbidden by the symmetry of the parent structure. (b) These reflections are evident in selected area electron diffraction patterns of individual d-UiO-66(Hf) crystallites. (c) The fundamental unit of the domain structure in this material has cluster vacancies at the corner of the unit cell and has primitive cubic symmetry. (d) These primitive cells coexist with regions of the parent face-centred cubic structure, facilitated by the close match in unit cell metric. (d) Because the face-centred cubic lattice can be understood in terms of four interpenetrating primitive cubic lattices, there are four possible orientations for the defect nanodomains. In the bulk d-UiO-66(Hf) structure all four of these occur (here shown using different colours). This model of the domain structure was used to calculate the diffraction pattern shown in panel (a).

Figure 12 .

 12 Figure 12. (a) The face-centred cubic structure of defect-free UiO-66. (b) The 2-sulfoterephthalate UiO-66 derivative reported in reference 59 has a body-centred cubic structure with a lattice parameter twice that of the parent structure shown in (a). Cluster vacancies order at the edges of this enlarged unit cell. (c) The primitive cubic structure of the defect domains in d-UiO-66(Hf), which is related to the cell shown in (b) on removal of the clusters at the corners of that unit cell (shown here in blue).

Figure 13 .

 13 Figure 13. (a) Representation of the Zr 6 cluster geometry found in the zirconium formate compound Zr 6 O 4 (OH) 4 (HCOO) 8 . 61 The equatorial oxygen atoms belong to dangling formate anions, and are disordered over closely-related sites. The two crystallographically-distinct Zr atoms are shown in different shades of green. (b) The central Zr6 octahedron geometry in this cluster, showing the spontaneous symmetry-lowering of Zr...Zr distances.

Figure 14 .

 14 Figure 14. Dimensionality of coordination polymers and MOFs known to undergo structural melting. (a) [Zn(HPO 4 )(H 2 PO 4 ) 2 ].2C 3 H 5 N 2 . (b) [Zn(H 2 PO 4 ) 2 (HTr) 2 ] n . 72 (c) [Zn(C 3 H 3 N 2 ) 2 ]. Adapted with permission from Umeyama, D.; Horike, S.; Inukai, M.; Itakura, T.; Kitagawa, S. J. Am. Chem. Soc. 2015, 137, 864., and Umeyama, D.; Horike, S.; Inukai, M.; Itakura, T.; Kitagawa, S. J. Am. Chem. Soc. 2012, 134, 12780. Copyright 2015 and 2012 American Chemical Society respectively.

Acknowledgements

The authors would like to thank Ras Al Khaimah Center for Advanced Materials (AKC, TDB), Trinity Hall (TDB), and the ERC (ALG, Grant 279705).

Notes

The authors declare no competing financial interests.


Lydia Montandon
email: lydia.montandon@atos.net

Ioana Ghergulescu
email: ioana.ghergulescu@adaptemy.com

Marilena Bratu
email: marilena.bratu@fpse.unibuc.ro

Diana Bogusevschi
email: diana.bogusevschi@dcu.ie

Renata Rybarova
email: renata.rybarova@stuba.sk

Multi-dimensional Approach for the Pedagogical Assessment in STEM Technology Enhanced Learning

The concept of the NEWTON project is based in an identification of the challenges and barriers in creating engagement and enthusiasm amongst STEM students and recognises the need to stimulate and embrace the type of change that innovative technologies can bring. Thus, NEWTON focuses on employing novel technologies to increase the quality of the learner experience, improve learning processes and increase learning outcomes. One of the major challenges of the project is to validate the platform impact and the effectiveness of the teaching scenarios in terms of learner satisfaction, improvement of the learning and teaching experience, and the evaluation of underlying technology through European-wide real-life pilots. In this paper, we describe briefly the criteria to be taken into account for technological learning systems evaluation from a pedagogical view point. After setting out the different dimensions of learning to be evaluated, we describe the iterative approach adopted to ensure that there is a continuous feedback loop created thereby ensuring that the results of all pilot activities are shared and are informing future iterations of the NEWTON solution. Finally, we provide a few details of the work plan designed to guide all stakeholders involved in the project pilots. This work is dedicated to teachers and teaching organizations willing to engage learners in STEM education.

Introduction

Technology enhanced learning is widely seen as the future of education, with personalised learning journeys created through adaptive software, taking into account the uniqueness of every student. These advances are not meant to replace the teacher in the classroom, but rather provide them with tools that will enable them to teach students at different levels, while keeping everyone engaged and up to speed. Technologies will change the role of the teacherand this has to be part evaluationbut a change in role can be positive as part of a wider drive towards more dynamic, interactive and experiential forms of learning.

The concept of the NEWTON project is based in an identification of the challenges and barriers in creating engagement and enthusiasm amongst STEM students and recognises the need to stimulate and embrace the type of change that innovative technologies can bring. Although the rationale of bringing such novel technologies into the classroom is clear, we also recognise the need to:

link the use of technologies with the application of new and existing pedagogies in a way that ensures effective learning and that goes beyond the mere 'novelty' of adding a new piece of technology to a learning process demonstrate how any new technology can adapt to the curricular demands of different learning institutions and support the delivery of learning content that builds required knowledge and skills understand better how the use of novel technologies impacts on the role of the teacher, on their preparation prior to learning, their interactivity with students during learning and the evaluation and assessment of students after learning identify and analyse the practical implications of implementing new novel technologies within a learning environment including an assessment of the impact on resources, time, cost and so on

The NEWTON project is unique in that the solution is an integrated, gamified system that utilises a number of novel technology components, such as game-based learning, gamification, Virtual and Augmented Reality, Personalisation and Adaptation, Mulsemedia, etc.

NEWTON1 focuses on employing novel technologies to increase the quality of the learner experience, improve learning processes and increase learning outcomes. Furthermore, NEWTON aims to implement personalisation and adaptation for content, delivery and presentation. One of the major goals of the project is to validate the platform impact and the effectiveness of the teaching scenarios in terms of learner satisfaction, improvement of the learning and teaching experience, and the evaluation of underlying technology through Europeanwide real-life pilots. Special attention is be given to the evaluation of the pilot studies, as neither subjective nor objective methods alone are enough to assess all the properties of any given approach, including effectiveness, efficiency, satisfaction, etc. Thus, a framework that combines subjective and objective methods to evaluate the pilot studies, their learning effectiveness, learning efficiency, and learner satisfaction is proposed.

To support the evaluation, we offer a set of theoretical and practical recommendations for all the partners involved in piloting NEWTON proposed technologies and in pedagogical assessment of the impact of these technologies in school activities, both for typically developed students and for students with special educational needs. The objective of this theoretical framework complemented by a practical work plan has been to amalgamate different elements of the NEWTON project that relate to the effective evaluation of the project impact as a tool for teaching and learning.

Our methodology criteria

Specifically, we have identified important criteria to be taken into account for technological learning systems evaluation from a pedagogical view point. These criteria are described below:

the nature, size and scale of all of the NEWTON pilots and the feasibility of assessment and evaluation within the context of each the interaction between the various NEWTON technologies, on the one hand, and the design of the content and experiments to be undertaken within the pilots on the other hand the relationship between the NEWTON 'platform' (the learning management system we have developed) and the NEWTON 'components' (the various technologies that will be 'plugged into' the NEWTON platform) the nature of the data that will be generated through use of the NEWTON platform and how we will be able to use this data in order to inform our evaluation the way in which the evaluation and assessment tools that we develop may be useful or otherwise in addressing the impact of NEWTON on learners with special educational needs the requirement to address not only the value of the NEWTON approach in terms of promoting learning but also to explore the wider impact of using the NEWTON technology on learner motivation, affective state, teacher role and so on

Our methodology dimensions

In our theoretical framework and practical guidelines and work plan, we have set out the main dimensions to be evaluated: learning outcomes, learner satisfaction, affective state dimension, psychological profile development (an additional dimension that will be evaluated only for students with special educational needs; on this type of learners all the other dimensions mentioned above will be also assessed) and teacher role and satisfaction. For each of these dimensions we present definitions of metrics and methods (objective and subjective) to be used in assessment process and what the toolbox recommendations are. The impact on teacher role is a critical aspect of introducing any new technology into the classroom and in this context we have paid particular attention to the involvement of teachers in the evaluation process. For the evaluation made by teachers we have referred at the following aspects: level of enjoyment/frustration by the teacher in the use of the technology; degree to which the technology supports the teacher in their activity; degree to which the technology helps the teacher to achieve specified learning outcomes.

The work plan for pilot evaluation establishes that once usability and technical performance are addressed for the different learning scenarios of the project, the pedagogical assessment of pilots will be undertaken at three levels:

1. Project General Questions 2. Specific Research questions associated to each particular pilot (Pilot hypothesis), to be negotiated with each pilot. 3. Iterative Learning. We also consider a 3 rd level related to the implementation of the pilots. It will be of utmost importance for subsequent pilots to learn from the experience gained in previous pilots. For instance, were the questionnaires understandable and of right size? Was the initial presentation to teacher relevant? Was the training to operate the technology sufficient? Was the information gathering processes efficient? Etc. This learning may be useful beyond the scope of the project, for future deployment of the NEWTON platform, technologies and pedagogical approaches.

Assessment objectives at the didactic level can be designed in the context of the educational process for verifying the quality of the training activity after three pedagogical indicators:

1. School progressidentify if the proposed technologies are implementable in the school infrastructure, if the institution is able to implement efficiently these technologies and how higher technical facilities contribute to improve school activity. 2. School effectiveness -Reporting the results to the general and specific objectives of the curriculum (in terms of competencies) and to the concrete objectives designed by the teacher (in this case we have the normative assessment). In NEWTON one of the objectives of pedagogical assessment is to verify how the use of modern pedagogical approaches and new technologies influences the development of competences in children. 3. School efficiency -reporting of school results to the existing resources in the educational process, (in this case, we have managerial assessment). In case of NEWTON pedagogical assessment, from this point of view, one of the main objectives is to highlight the benefits that NEWTON brings to school results by improving the resources that school has.

Conduct the assessment

Analyse the results

Take actions

Action Research

The approach we are taking to assess is iterative and should ensure that there is a continuous feedback loop created thereby ensuring that the results of all pilot activities are shared and are informing future iterations of the NEWTON solution. We need to guarantee that there is a clear coherence between the type of pilot and the type of evaluation run. Although our aim is to create standardisation across the evaluation process, we must also recognise that we cannot run exactly the same evaluation for all pilots and that some aspects of the evaluation will be dependent on size, scale and context. Specifically, longer pilots will give us the opportunity to explore more deeply some aspects of learner progress whereas shorter pilots or 'one-offs' may be better focused on learner experience with the technology.

In order to ensure that our approach is iterative, we will broadly follow the model set out in Figure 1 above. It illustrates the iterative action research approach recommended in the evaluation process. In each iteration of the evaluation loop, the researcher will start by defining the evaluation goal, the research question, and the expected learnings of the evaluation. The researcher will design the methodology from the NEWTON evaluation toolbox based on the evaluation goal and research question(s). After conducting the assessment and analysing the results, the researcher (or the research team) will recommend and suggest actions to be taken. These outcomes will be shared within a communal online environment with other Pilot Leaders and other researchers working with pilots.

As shown in Figure 2, some of the steps to put in place the pilots' assessment have been defined and are planned. For example, initial engagement with pilot institutions, initial research questions, steps of the intervention.

We will also be able to generate similar data in relation to teacherswe will be able to evaluate how specific NEWTON elements fit with particular teachers from different backgrounds, contexts and situations. Not only will this data help us to continuously improve the NEWTON platform and components, but it will also provide us with a wealth of data that we can report back on, thereby ensuring that the NEWTON project makes a significant contribution to further an understanding of state of the art and, ultimately, to standardisation.

Concluding Remarks

The theoretical framework and complementary work plan represent a guide for all partners involved in piloting. We have included the main steps to be followed in order to gather relevant data from each and every pilot, regardless of the type, size or nature of the pilot. The work plan starts with clarifications on the terms we use (smallscale pilot, large scale-pilot, direct learning, indirect learning etc.) and with the presentation of the main objectives and methodology of pedagogical assessment. The main roles and responsibilities are established in order to ensure that all activities are coordinated and performed in alignment with the pedagogical assessment objectives. All participants in the pilots will be assigned a specific role and associated responsibilities. The work plan then presents the different phases and activities that will take place during the assessment and underlines the importance of a strong relationship with the partner institution that will host NEWTON pilots. Teachers are active partners in the pedagogical assessment and they will be involved in every step of this process.

Finally, the proposed methodology considers the support of a 'Pedagogical Assessment Committee', which is a group of NEWTON experts who will act as a guide, partner and critical friend to all NEWTON pilot leaders, ensuring that the assessments undertaken in each pilot study are in line with the overarching strategy set out in the methodology and that all results are collated, analysed and shared effectively across the project. All partners will be familiarised with the methodology and key elements of the work plan will be summarised and shared with participating institutions. The first pilots using the methodology have started in February 2018 in Ireland. They have taken on board the guidelines and have designed an assessment process aligned with the proposed approach. The NEWTON research community expect to able to share the first informal results by mid-June.

Figure 1 :Determine

 1 Figure 1: Action research process to be integrated in the evaluation process

http://www.newtonproject.eu/

Acknowledgements

This research is supported by the NEWTON project (http://www.newtonproject.eu/) funded under the European Union's Horizon 2020 Research and Innovation programme, Grant Agreement no. 688503.