Stéphane Le Dizes
email: ledizes@irphe.univ-mrs.fr

Rotor Generated Vortices: Structure and Stability

Eduardo Duran-Venegas 1 , Stéphane Le Dizès 1 1 Aix Marseille Univ, CNRS, Centrale Marseille, IRPHE, Marseille, 13013, France

ABSTRACT

The structure of the vortical wake generated by a rotor is analyzed in the framework of a Joukowski model using a vortex filament approach. The different operational regimes of an helicopter in vertical flight are considered. The stability of the vortical wake is also analysed by looking at the spatio-temporal evolution of linear perturbations introduced in the rotor plane and compared to available theoretical results. A global instability is discovered in rapid descent flight when the wake is in the Vortex Ring State (VRS).

We study the structure and stability of the vortical wake generated by a two-blade Joukowski rotor of radius R b , rotating at angular velocity Ω, in an axial wind of velocity V ∞ . In this rotor model, each blade is assumed to emit a free tip vortex of circulation Γ and a hub vortex of opposite circulation centered on the rotor axis. Considering a fixed vortex core size a and neglecting viscosity, the vortex dynamics is computed by a free vortex method [START_REF] Leishman | Principles of Helicopter Aerodynamics[END_REF] using the Biot-Savart law. The problem is defined by three non-dimensionalized parameters: the tip speed ratio λ = R b Ω/V ∞ , the vortex strength η = Γ/ΩR b 2 and the vortex thickness ε = a/R b . For a small and fixed vortex thickness ε = 0.01, we first show that stationary solutions (in the rotating frame) can be obtained for almost all values of λ and η except in a small parameter region indicated in gray in figure 1. In this figure are indicated the different flight regimes of a helicopter associated with each solution. Of particular interest is the so-called Vortex Ring State (VRS) occurring during rapid descent that cannot be described by the general momentum theory [START_REF] Sørensen | General momentum theory for horizontal axis wind turbines[END_REF]. In this regime, vortices are present on both sides of the rotor plane as illustrated in figure 2. The stability of the various solutions is also addressed by analyzing the linear response to a Dirac perturbation applied at the rotor tip. For most flight regimes, the flow is found to be convectively unstable: the perturbation grows but is advected away from the rotor plane. The property of the wave packet far away has been compared to the theoretical predictions for uniform helices [START_REF] Gupta | [END_REF] and a very good agreement has been observed [see figure 3].

-0. Fig. 3 Growth rate versus wavenumber normalized by the far-wake helix pitch h for λ = ∞, η = 0.02 and ε = 0.01. Symbols: numerics. Line: Gupta & Loewy theory [START_REF] Gupta | [END_REF].

In the VRS regime, a different behavior is observed: the perturbation continues to grow in the neighborhood of the rotor and a well-defined global mode emerges everywhere. The rotor wake has become globally unstable.

Fig. 1

 1 Fig. 1 Helicopter flight regimes in the (1/λ, η) plane for a two-blade rotor for ε = 0.01.

Fig. 2

 2 Fig. 2 Vortex-Ring-State structure for λ = -15, η = 0.04 and ε = 0.01.


A Rafiee

M Vinches

Application of Geostatistical Characteristics of Rock Mass Fracture Systems in 3D Model Generation

The mechanical behaviour and the state of stability of a jointed rock mass depend on how the fracture systems were propagated in this medium. In fact, obtaining an accurate description of the fracture spatial distribution is one of the main problems to be solved in rock mass stability studies. The purpose of this paper is to apply the geostatistical method to find the spatial correlation between joint systems. Geostatistics enable us to quantify the spatial characteristics of fracture patterns, which can not be derived by ordinary statistical method or structural analysis. The combination of the traditional method used in structural analysis and of the geostatistical method is used in order to get a better insight into the fracture distribution in the rock mass. The results obtained, are then applied to generate 3D numerical jointed rock mass models.

Introduction

The rock mechanics and fluid transport properties of the rock mass depend mainly on the characteristics of the fracture field. Fractures are only accessible at their intersections with outcrops, boreholes or drifts: one of the first steps in fractured rock mass studies is to make assumptions about the fracture geometry, and to simulate the fracture field.

The modelling of a fractured rock mass is complex and depends on multiple parameters. Therefore, because of existing constraints in field surveys and incomplete data, it is necessary to extract as much information as possible from the data which has been collected. This is one of the reasons for the application of sophisticated methods to obtain the fundamental spatial characteristics of fracture systems. This type of study can be performed on different scales: on large faults, on fractures and on microjoints. For each category two separate features can be studied:

1-The fracture network: each element is considered as relatively simple (i.e., a planar fracture), and the study is concerned with the imbrications of the various elements.

2-The single fracture: generally an open fracture is not a pair of parallel planes; there are voids and contact areas that depend on stress and have a high influence on the mechanical behaviour of the fracture.

The fractal and geostatistical approaches are used to study the single fracture [START_REF] Belem | Fractal analysis of shear joint roughness[END_REF][START_REF] Ehlen | Fractal analysis of joint patterns in granite[END_REF][START_REF] Hamdi | A fractal description of simulated 3D discontinuity networks[END_REF][START_REF] Huang | Applicability of fractal characterization and modeling to rock joint profiles[END_REF][START_REF] Mandelbrot | The Fractal Geometry of Nature[END_REF][START_REF] Mandelbrot | Self-affine fractals and fractal dimension[END_REF][START_REF] Mandelbrot | Fractal character of fracture surfaces of metals[END_REF][START_REF] Mcwilliams | Estimation of shear strength using fractals as a measure of rock fracture roughness[END_REF][START_REF] Sayles | The spatial representation of surface roughness by means of the structure function: A practical alternative to correlation[END_REF][START_REF] Thomas | Structure fractal de l'architecture de champs de fractures en millieu rocheux[END_REF][START_REF] Xie | Fractal effects of surface roughness on the mechanical behavior of rock joints[END_REF][START_REF] Xie | Direct fractal measurement of fracture surface[END_REF][START_REF] Yang | Adirectional method for directly calculating the fractal parameters of joint surface roughness[END_REF]. The scope of this work, the ultimate aim of which is the study of the mechanical behaviour of the fractured rock mass, is focused on the study of the fracture network. In such studies, because we have very limited data in comparison with the size of the model that will be simulated, deterministic modelling is often impossible. In this case, stochastic modelling is needed. Much work has been carried out to characterize a fracture network by the integration of the statistical properties of the fracture system such as distribution of orientation, trace length, spacing and aperture in the models [START_REF] Hadjigeorgiou | Defining In-Situ Block Size[END_REF][START_REF] Dershowitz | Characterizing rock joint geometry with joint system models[END_REF][START_REF] Pr | Pattern analysis and simulation of joints for rock engineering[END_REF][START_REF] Lanaro | A random field model for surface roughness and aperture of rock fractures[END_REF][START_REF] Baecher | Statistical description of rock properties and sampling[END_REF][START_REF] Billaux | Three-dimensional statistical modelling of a fractured rock mass -an example from the Fanay-Augères Mines[END_REF][START_REF] Cruden | Describing the size of discontinuities[END_REF][START_REF] Dershowitz | Interpretation of fracture spacing and intensity[END_REF][START_REF] Hamdi | A methodology for rock mass characterisation and classification to improve blast results[END_REF][START_REF] Kulatilake | Estimation of mean trace length of discontinuities[END_REF][START_REF] Kulatilake | Application of probability and statistics in joint network modeling in three dimensions[END_REF][START_REF] Lyman | Rock fracture mean trace length estimation and confidence interval calculation using maximum likelihood methods[END_REF][START_REF] Mauldon | Estimating the fracture trace length and density from observations in a convex window[END_REF][START_REF] Pahl | Estimating the mean length of discontinuity traces[END_REF][START_REF] Xu | Three-dimensional simulation of natural rock granulometry[END_REF][START_REF] Priest | Discontinuity spacing in rock[END_REF][START_REF] Zhang | Estimating the mean trace length of rock discontinuities[END_REF][START_REF] Zhang | Estimating the intensity of rock discontinuities[END_REF]. The fractal method has been applied in some cases for studying the fracture network. Geostatistical techniques have also been used to study the regionalized characteristics of the fracture network [START_REF] Ayalew | Characterizing weathered rock masses-a geostatistical approach[END_REF][START_REF] Chilès | Fractal and geostatistical methods for modelling of a fracture network[END_REF][START_REF] Chilès | Three-Dimensional geometric modelling of a fracture network[END_REF][START_REF] Chilès | Caractérisation et simulation multi-échelles de réseaux de fractures de réservoirs pétroliers[END_REF][START_REF] Chilès | Stochastic modelling and simulation of fracture networks in petroleum and geothermal reservoirs[END_REF][START_REF] Gentier | Infuence of fracture geometry on shear behavior[END_REF][START_REF] Gentier | Analyse et simulation d'une surface de fracture rocheuse[END_REF][START_REF] Hestir | Some techniques for stochastic modelling of threedimensional fracture networks[END_REF][START_REF] Massoud | Modélisation de la petite fracturation par les techniques de la géostatistique[END_REF]. The high complexity of this network requires the results from all of these methods to produce a realistic model.

The approach proposed here combines the geostatistical method and the usual structural geological analysis. First, the semivariogram analysis is used for all of the data sets, in this case for the fracture density, and afterwards the same method is used for the data grouped by structural analysis. The fracture density, and the fracture orientation characteristics are often the predominant factors in the stability of rock slopes. These two factors contribute to the definition of the vulnerability of the rock mass. The use of this combination of methods will be illustrated within the framework of a real case study.

As previously mentioned, we use the orientation data sets measured within the framework of road widening in the Pyrenees region. This 600 meter long project, on a main road crosses mountainous territory (Fig. 1) [START_REF]Laboratoire Régional des Ponts et Chaussées de Toulouse[END_REF]. The uphill and downhill slope of the present road will have to be cleared of the blocks likely to be destabilized by vibrations.

The road cuts through both migmatites and micaschists. Discontinuity orientation data are obtained by means of compass measurements of discontinuities that intersect with the scanline positioned on the exposed rock faces (Fig. 1b). The fracture measurements were carried out on nine scanlines: six in the migmatite and three in the micaschist. In this study, we chose the scanline on which a significant number of fracture measurements was collected for geostatistical analysis. Scanline number 1, in the migmatite with 392 measured fractures, is 90 m long. The adequate scanline for a geostatistic analysis depends upon several parameters, particularly the fracture density and the scale of the study.

The grouping of discontinuities into principal discontinuity sets for this scanline is performed by the spectral clustering method that is a recent approach proposed by R. Jimenez-Rodriguez and N. Sitar [START_REF] Jimenez-Rodriguez | Spectral method for clustering of rock discontinuity sets[END_REF]. We developed a code in the R environment on the basis of their algorithm, see details of this algorithm in [START_REF] Jimenez-Rodriguez | Spectral method for clustering of rock discontinuity sets[END_REF].

In Fig. 2, we present the computed partitions of the spectral clustering in the lower hemisphere projection when four principal families are considered for all the data sets.

Calculating the direction of the average unit normal vectors for each discontinuity set and partitioning the data in four families was performed on the fractures that are not very distant from the reference point of each family. Consequently here, certain fractures have been assumed not to belong to any of these four families. Fig. 3 shows the results of the spectral clustering for 315 fracture orientations located in the vicinity less than 25° from the average direction of each family. As a result, the pole points labelled with the * symbols were not taken into account for the average point calculations. They represent, in our case, nearly 20% of the complete data set. The choice of 25° for the vicinity angle was made by comparing the results obtained for different vicinity angles. In fact, in order to calculate more adequately the representative points for each fracture set, it is necessary to eliminate the points located away from the centre of each set. However, using 25 degrees in this elimination step, a large amount of data is not lost as would have been the case for 5°, 10°, 15° and 20° vicinity angles. The high vicinity angle also leads to having more marginal points in the set, and consequently decreases the pertinence of the representative point. Therefore, in Fig. 3 by choosing the vicinity angle of 25°, we tried to eliminate the marginal points and also to keep a reasonable number of points for the next steps of data processing.

Table 1 shows that the average normal vectors for each discontinuity set and the number of fractures belonging to each family based on the calculation displayed in Fig. 3. The average of the normal set spacing in cm for each fracture set is also presented in Table 1 (the set spacing measured along a line that is perpendicular to the mean orientation of a particular set).

Statistical characteristics of the fracture density

For each set, the fracture spacing is measured by the real distance separating successive intersections, on the scanline, of fractures belonging to this set. The discontinuity spacing is widely used as a measure of the quality of a rock mass in several engineering classifications.

The inverse of the mean spacing is the average density of intersections along the scanline. The frequency histogram for the normal spacing values of four joint sets (Fig. 4) appears to follow the general shape of the exponential distribution. This distribution may be expressed as

x e x f   - = ) ( (1)
where

) (x f
is the frequency of a discontinuity spacing x, and  is the average number of discontinuities per metre. It should be noted that the exponential distribution for discontinuity spacing distribution was recommended by Priest and Hudson [START_REF] Priest | Discontinuity analysis for rock engineering[END_REF][START_REF] Priest | Estimation of discontinuity spacing and trace length using scanline surveys[END_REF].

Discontinuity frequency (density) is one of the fundamental measures of the degree of fracturing of the rock mass. Frequency can be expressed in terms of the number of discontinuities that are observed or predicted to occur in a unit volume, a unit area or a unit length of a sample from a given rock mass [START_REF] Priest | Discontinuity analysis for rock engineering[END_REF][START_REF] Priest | Estimation of discontinuity spacing and trace length using scanline surveys[END_REF]. In this study, frequency is used as a variable for geostatistical analysis as explained in the next section. A histogram for the discontinuity frequency for all four sets in 2m long segments is presented in Fig. 5.

Geostatistical approach

Geostatistical approaches can be used to investigate the spatial behaviour of a variable. In our case, this variable can be a characteristic of a fractured rock mass such as spacing, fracture length and fracture frequency. This can lead to a better understanding of the spatial distribution and structure of the variable. The semi-variogram γ(h) is a function describing the degree of spatial dependence of a spatial random field or stochastic process Z(x). In this study, the fracture density is considered as the variable Z(x), and its semi-variogram is calculated by numbering the fractures observed in the defined segments along the scanline. The semi-variogram indicates how different the values become as the distance h between two points increases. In fact, for the fracture density case, the semi-variogram can be defined as a measure of half of the average of the square difference between the number of fractures (in equal segment size) a distance h apart.

The semi-variogram of fracture density is defined as:

2 1 ) ) ( ) ( ( 2
1 ) (  = - + = N i i i x Z h x Z N h  (2)
where x and x+h refer to points in an n-dimensional space where n could be 1, 2 or 3. For the fracture density case, using a scanline, n is considered equal to 1. The parameter N represents the number of pairs for the distance h [START_REF] Armestrong | Basic Linear Geostatistics[END_REF][START_REF] Journel | Mining Geostatistics[END_REF].

The semi-variogram summarizes the dissimilarity or represents the statistical mean of the square of the estimation error of Z (here the fracture density) between two points, a distance h apart. We subdivide a surveying line into equal segments of length b. Then for distances h=b, 2b, 3b..., we compute the semi-variogram of the number of intersections per segments. Experimental semi-variograms often increase continuously with distance; however, the semi-variogram is not restricted to such monotonic forms: decreasing segments or cyclical patterns can be observed. Non-monotonic semi-variogram structures are identified as "holeeffect" structures [START_REF] Journel | Anisotropic hole-effect modeling[END_REF]. Hole-effect structures most often indicate a form of periodicity, which is a common and legitimate spatial characteristic in geology. Ignoring these non-monotonic structures may result in unrealistic heterogeneity models that do not reproduce the observed patterns of variability.

In addition, as the segment size increases and progressively reaches values similar to the length of the characteristic periodic interval, the periodic structure becomes less apparent. The segment size of 2m was chosen in this study, for the geostatistical analysis, by considering the loss and weakness of the periodic structure for segment sizes higher than 2m and the frequency distribution (Fig. 5) which is nearly normal for this segment size. It should be noted that the frequency distribution for three other segment sizes were far from the normal distribution.

Because of the high influence of the variable distribution on the experimental semi-variogram calculation, the data of the 2m long segments were preferred [START_REF] Armestrong | Basic Linear Geostatistics[END_REF][START_REF] Journel | Mining Geostatistics[END_REF].

In order to better understand the origin of this cyclical pattern, experimental semi-variograms were separately calculated for all four fracture sets. Fig. 7 shows the fracture density semivariograms obtained for the four discontinuity sets. As can be observed in these semivariograms, three of them have nearly the same behaviour, but the variogram of the second set (Fig. 7b) presents a form similar to the variogram obtained for all fractures (Fig. 6). This remark can explain that this particular behaviour of the fracture density variogram for all the discontinuities, illustrated in Fig. 6, derives only from the fractures belonging to the second set.

As with the variogram in Fig. 6, we have the same hole-effect behaviour for the variogram of the fracture density for the second set with the same periodic distance of 12m.

In addition, in order to demonstrate that this similarity corresponds only to the second set, the semi-variogram of all the discontinuities without those of the second set is computed. Fig. 8 shows the semi-variogram calculated from these data. There is no similarity between this variogram and the variogram illustrated in Fig. 6. As a result, the discontinuities of the second set are the source of the hole-effect behaviour of the semi-variogram in Fig. 6.

3D model generation

After the statistical and geostatistical analysis explained in the previous sections, the results obtained are used to generate a 3D fractured rock mass model. This model is simulated by considering three distinct phases: fracture surface generation set by set, intersection of the different sets with a hierarchical order, and finally identification of individual blocks.

Fracture surface generation set by set

Two steps are necessary for this phase. The first step will generate the fracture sets that do not have an obvious spatial structure: the generation of these sets relies only on their statistical characteristics. The second step consists in identifying the fracture sets presenting a quasiperiodic structure in their semi-variogram and generating the fracture sets using their revealed geostatistical characteristics.

Concerning the first step, we generate in our example the discontinuities of sets 1, 3 and 4.

It should be mentioned that the hierarchical method is used only during the intersection phase (see below), where the primary geometry is divided into separated blocks by the previously stochastically generated fracture surfaces. Consequently, it is not necessary to apply the hierarchical procedure for this phase of fracture surface generation.

These sets do not show a particular structure in their semi-variograms. As a result, their fracture systems are generated by considering only their statistical characteristics: mean value and standard deviation derived from their spacing distribution. For this purpose, we developed a code written in the Visual Basic language in the AutoCAD® software. The flow chart of the model process is shown in Fig. 9a for the model generated by considering the simple statistical characteristics. These three sets are not necessarily generated in the direction of the scanline.

However, for the second set this issue was taken into account by considering its observed structure in this direction. The position of first fracture surface in the space for each set is generated using the uniform law. The rest of the fractures are then generated parallel to this surface by taking into account the statistical characteristics of the spacing distribution.

A jointed rock mass is simulated by generating a joint pattern that is statistically based on the joint spacing, the orientation data and the persistence of each fracture set. It should be noted that in this study, fracture is initially considered as an infinite planar surface, taking into account the average orientation parameter (Table 1) and the spacing distribution. However, this infinite planar surface can then be pivoted during the intersection phase by a given tolerance concerning its dip and strike. As a result, each fracture surface can take different orientations inside its tolerance range, whenever it is called by the code in the intersection phase. The joint spacing parameter for this study is assigned an exponential distribution by equation ( 1) and is obtained by a random generator algorithm. Then the rejection method is used to ensure that these randomly generated spacing values fall within the desired distribution [START_REF] Press | Numerical recipes in C[END_REF].

For orientation data, the results shown in Table 1 are The data generated for calculating the model semi-variogram is then employed in the Auto Cad code, as previously explained, to generate a fracture system by considering the number of fractures per segment and the statistical parameters of spacing distribution for this set.

Therefore, this set is generated segment by segment in order to respect its semi-variogram attributes. It should be noted that because of the limited number of fractures per segment, the statistical parameters used in this step for the second set are obtained from its whole data, but simultaneously the number of fractures per segment corresponds to the number determined by semi-variogram analysis.

Intersection of fracture sets with a hierarchical order

Persistence is a necessary parameter for the generation of the fractured model in the intersection phase. Persistence (joint continuity) is defined as the probability that any joint cuts a block that lies in its path [START_REF] Kalenchuk | Characterizing block geometry in jointed rockmasses[END_REF][START_REF] Meyer | Geologic Stochastic Modelling and connectivity assessment of fracture systems in the Boston area[END_REF][START_REF] Einstein | Modern developments in discontinuity analysis the persistence discontinuity problem[END_REF]. It can be considered as a substitution parameter for joint length. Persistence can have a value of between 0 and 1. In addition, it is the persistence parameter that determines the state of the fracture termination when intersecting with pregenerated ones. For a persistence value near to 1, there will be more fractures that go through pre-existing fractures. For values near to 0, the new fractures end when intersecting with preexisting ones. The approximate persistence value for each fracture set can be determined by comparing the fracture length distribution obtained from the field survey with the distribution resulting from the generated test model. It should be noted that this comparison must be carried out on the same scanline direction. This part of the program is a very time-consuming but necessary task, in order to obtain a realistic model.

Once all the fracture sets are generated, on the basis of notes taken during the field measurement, the order of sets is defined [START_REF] Lee | Hierarchical fracture trace model[END_REF]. Indeed, there are three possibilities for ordering including hierarchical, random or hybrid methods. In this case, the hierarchical method (1 st , 2 nd , 3 rd and finally 4 th set) is used for the model generation. 2 respectively for 1 st , 2 nd , 3 rd and 4 th fracture sets. It should be remarked that these persistence values have been determined by a trial-and-error approach to obtain fracture trace length distributions in the scanline direction, similar to the in-situ measured distributions.

Identification of individual blocks

Because the aim of this code is to use the generated model for numerical simulation, another possibility is added to the code to avoid having blocks smaller than a determined volume. In other words, the code user can determine the threshold value for the minimum block volume.

By considering the existing abilities in the Auto Cad environment and several auxiliary codes developed in this software, each generated block is considered as a distinct entity. A series of codes are also developed for data acquisition from the generated model. The developed programs enable us to obtain data such as block volume distribution, area and orientation of the block facets and block vertices coordinates. The block volume distributions shown in Fig. 11 are examples of their application. In this model the volume threshold value is 0.25 m 3 and the final model has 19227 stochastically generated blocks.

Conclusion

Understanding the discontinuity spatial distribution of a rock mass has important applications in the field of rock engineering. As a result, applying statistical and particularly geostatistical techniques is useful to reveal the spatial characteristics of fracture systems. These characteristics generally vary according to a large number of parameters such as the rock mass type, or the tectonic situation. Therefore, the different combinations of statistical techniques should be used by taking into account the diverse geological and tectonic conditions. In this study, we have illustrated the application of the geostatistical technique combined with the traditional methods to reveal the main characteristics of the fracture spatial distribution. The importance of separating the fractures into distinct sets was demonstrated by illustrating their different spatial behaviour. One of the main goals of this study is to show the usefulness of a geostatistical analysis in order to distinguish the spatial characteristics of the discontinuity distributions that are beyond the capabilities of the simple statistical and traditional methods.

The results are used to generate a realistic 3D rock mass model. Such a model can then be used in different applications such as rock mass stability analysis, planning the exploitation system, or size estimation of exploitable rock blocks depending on the project goals.

Fig. 6

 6 Fig.6illustrates the average experimental semi-variograms for the fracture densities with

  used to generate four fracture sets. It should be mentioned that the code user can give the tolerance angle for strike and dip value by taking into account Fig. 3. The orientation data are assumed to follow a normal law and the program takes the mean and standard deviation of the fitted distribution for each fracture set as input parameters. The second step of the fracture model generation concerns the fracture set whose semivariogram has a special behaviour. For this type of set, the characteristics of the density semivariogram in addition to its statistic features are taken into account in the program. The model is generated by considering the 2m long segments and by obtaining similar behaviour in the semi-variogram, for the numerically generated fractures. This step is performed by using a special code developed in the R environment which uses geostatistical tools. In fact, the characteristics such as segment size, range, maximum and minimum number of fractures per segment and the standard deviation of the density distribution are used to generate the model with the same statistic and geostatistic features. Besides, the user of the code can compare the semi-variogram of the model with the semi-variogram of the raw data, and accept or reject it. In case of rejection, a new simulation is performed until a good enough fitting of the numerical model and the experimental results is obtained. The fracture density semi-variogram obtained in this way for the second set is displayed in Fig. 10. It presents the same periodic forms 12m apart and similar other parameters (apparent sill and drift).

Fig. 11 illustrates

 11 the results of a 3D model generation in which four discontinuity sets are incorporated. This figure shows how fracture sets are included in the model in a hierarchical way: first with two sets, then with the inclusion of the two others. The persistence values used in this model are 0.45, 0.35, 0.3 and 0.

Fig. 1 .

 1 Fig. 1. Left: Panoramic view of the sector of Pallat, showing the existence of danger due to rock mass instability-Right: close-up of the fracture system at the road side outcrop, and measuring scanline.

Fig. 2 .

 2 Fig. 2. Spectral clustering in four principal sets of the 392 discontinuity orientations measured in the migmatite rock mass, projected in the lower hemisphere system-Pallat ravine.

Fig. 3 .FrequencyFig. 4 .Fig. 5 .

 345 Fig. 3. Spectral clustering in four principal sets of 315 discontinuity orientations projected in the lower hemisphere system-by considering the vicinity of 25°. The points shown in * symbol belong to no family and are considered as outliers.

Fig. 6 Fig. 7 .

 67 Fig. 6 Average semi-variogram of fracture densities for the segment sizes of 0.5m, 1m, 1.5m and 2m.

Fig. 8 .

 8 Fig. 8. Semi-variogram of the fracture density for 2m segments from all the discontinuities measured on the scanline, without the discontinuities belonging to the second set.

Fig. 9 Fig. 10 .

 910 Fig. 9 Flowchart of 3D fractured rock mass modelling: (a) 3D model without semi-variogram structure (Auto CAD) and (b) 3D model with hole-effect structure in the semi-variogram (R environment).

Fig. 11 .

 11 Fig. 11. Simulated rock mass in a (30m*30m*30m) cube and corresponding block volume histogram: (a) stochastically generated jointed rock mass by two fracture sets including first and second sets (Y axis is the North direction); (b) stochastically generated jointed rock mass with all four fracture sets; (c) the histogram of block volume distribution for the rock mass model shown in (a); the histogram of block volume distribution for the final rock mass model exhibited in (b).

Table 1

 1 Direction of average unit normal vectors for each discontinuity set, obtained from Fig.3, number of fractures and real spacing distance (in cm) for each set.

	Set		Number of fractures	Dip	Direction of dip	Real spacing average(cm)
	1	+	117	8	193	17.65
	2	∆	88	8	94	96.9
	3	○	78	17	308	88.15
	4	×	32	51	305	140

Acknowledgments:

The authors wish to thank M. Gasc-Barbier, D. Virely ,J. Guittard and the personnel of the the "Laboratoire Régional des Ponts et Chaussées de Toulouse" who made their measurements data and the technical report of the case used in this paper available.


Laetitia Ricci1
email: l.ricci@chru-nancy.fr

Jean-Baptiste Lanfranchi3

Fabienne Lemetayer2

Christine Rotonda4

Francis Guillemin1

Joël Coste5

Elisabeth Spitz2

Qualitative methods used to generate questionnaire items: a systematic review 1 Qualitative methods used to generate questionnaire items: a systematic review

Keywords: Items generation, Qualitative methodology, Systematic review, Content validity

A systematic review of articles using qualitative methods to generate questionnaire items identified in MEDLINE and PsycINFO from 2000 to 2014 was carried out. Articles were analyzed for 1) year of publication and journal domain; 2) qualitative data collection methods;

3) method of data content analysis; 4) professional experts' input in item generation; 5) debriefing of the newly developed items. 371 articles were included and results showed 1) an acceleration of published articles; 2) individual interviews and focus groups were common ways of generating items and no emergent approach was identified; 3) the content analysis was usually not described (43% of articles); 4) experts were involved in eliciting concepts in less than a third of articles; 5) 61% of articles involved a step of further submission of newly developed items to the population of interest. This review showed an insufficient reporting of qualitative methods used to generate new questionnaires despite previous recommendations.

The development and use of questionnaires are common in health research and the use of qualitative methods to generate items enriches the quality of questionnaire items [START_REF] Mckenna | Development and validation of the living with chronic obstructive pulmonary disease questionnaire[END_REF]. Content validity is a fundamental element of the questionnaire validity. Content validity is the degree to which items of a questionnaire are relevant to and representative of the targeted construct for a particular assessment purpose [START_REF] Mckenna | Development of the PsAQoL: a quality of life instrument specific to psoriatic arthritis[END_REF]. The most appropriate way to collect data to ensure content validity is by conducting qualitative research with direct communication with the population of interest. Experts recommend that the generation of questionnaire items include qualitative methods involving members of the population of interest to ensure that the questionnaire fully reflects their perspective and that items are acceptable, comprehensive, and relevant to their condition [START_REF] Mckenna | Development of the PSORIQoL, a psoriasis-specific measure of quality of life designed for use in clinical practice and trials[END_REF].

This methodology drives item generation via the population of interest rather than the expectations of researchers. It ensures that all important aspects were explored and that no important aspect was omitted [START_REF] Doward | The development of patient-reported outcome indices for multiple sclerosis (PRIMUS). Multiple Sclerosis[END_REF].

Strategies to ensure rigor in qualitative research were well documented in the early 2000s in the BMJ (Mays & Pope, 2000;[START_REF] Pope | Qualitative research in health care. Analysing qualitative data[END_REF]. More recently, several papers [START_REF] Brod | Qualitative research and content validity: Developing best practices based on science and experience[END_REF][START_REF] Lasch | PRO development: rigorous qualitative research as the crucial foundation[END_REF]Patrick et al., 2011a[START_REF] Patrick | Content Validity-Establishing and Reporting the Evidence in Newly Developed Patient-Reported Outcomes (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 2-Assessing Respondent Understanding[END_REF] specifically provided recommendations for establishing content validity during the development of new questionnaires. All these papers provide the same recommendations involving a series of steps with 1) a review of the literature, 2) concept elicitation interviews or focus groups, 3) qualitative data analysis, 4) item generation, and 5) cognitive interviews to elicit thoughts about the questionnaire items in the population of interest to identify problems and support the content validity of the questionnaire.

However, since the introduction of recommendations for qualitative methods to generate items for questionnaires [START_REF] Brod | Qualitative research and content validity: Developing best practices based on science and experience[END_REF][START_REF] Lasch | PRO development: rigorous qualitative research as the crucial foundation[END_REF]Patrick et al., 2011a[START_REF] Patrick | Content Validity-Establishing and Reporting the Evidence in Newly Developed Patient-Reported Outcomes (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 2-Assessing Respondent Understanding[END_REF] and in the health research literature [START_REF] Mays | Qualitative Research: Rigour and qualitative research[END_REF], 2000;[START_REF] Pope | Qualitative research in health care. Analysing qualitative data[END_REF], we have lacked a comprehensive overview of current practices regarding the qualitative methods actually used in the development of questionnaires. This review focused on 1) how items are generated from a qualitative perspective and whether there are methods other than individual interviews or focus groups to collect qualitative data and 2) whether there was an increase over time in the use of methods eliciting content from the population of interest for item generation.

Methods

Search strategy

A systematic search strategy was used to identify eligible articles published from January 1, 2000 to December 31, 2014, written in English. We chose publications from 2000 on to capture the progression of reporting of qualitative methods.

-For MEDLINE the query (including MeSH Terms) was: -For PsycINFO, the search terms equation was:

[qualitative AND (Scale OR measure OR questionnaire OR survey OR test) AND items] in Select a Field which corresponds to All Fieds. We applied a restriction to academic journals and journals, so we excluded dissertations and books. We used the thesaurus of PsycINFO to select terms. For "qualitative" and "item", no new term appeared. "Questionnaire" was indexed to "Scale" and "Measure" and we added "Survey" and "Test".

Article selection

Articles were included if they described the development of new questionnaires with qualitative methods to generate items. We excluded articles with no qualitative approach described for item construction, and those concerning revision, modification or translation of existing questionnaires.

The titles and abstracts of identified articles were screened by two reviewers (Laetitia Ricci, Jean-Baptiste Lanfranchi) who used these pre-specified inclusion criteria. With any disagreements between reviewers, the full text of articles was read to reduce the possibility of rejecting relevant articles. The full text of all selected articles was read and duplicates were removed. This report followed the PRISMA statement for reporting systematic reviews [START_REF] Liberati | The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration[END_REF].

Analysis of selected articles

General information

The year of publication was used to analyze articles. Journals were classified into six domains: clinical, health sciences, nursing, psychology and psychiatry, social sciences and others. The distribution by journal domain was determined.

Qualitative data collection methods from the population of interest

The degree of structuring individual interviews is a common way to classify the research interview [START_REF] Brédart | Interviewing to develop Patient-Reported Outcome (PRO) measures for clinical research: eliciting patients' experience[END_REF]: in structured interviews, all interviewees are asked the same question in the same order. In semi-structured interviews the interviewer follows a guide but is able to follow topical trajectories in the conversation that may stray from the guide when appropriate. Finally, in unstructured interviews, the discussion guide is not standardized. Focus groups are a form of group interview that explicitly use group interaction as a part of the method: instead of the researcher asking each person to respond to a question in turn, people are encouraged to talk to one another [START_REF] Kitzinger | Qualitative Research: Introducing focus groups[END_REF]. Written answers to open ended questions is a method to collect data that focus more on precise topics [START_REF] Hanson | Qualitative Research Methods for Medical Educators[END_REF]. We identified methods other than interviews, focus groups or written answers to collect qualitative data. The percentage use of individual interviews, focus groups and written answers in the articles was calculated. For individual interviews, the characteristics of interviews, as reported by the authors, was described. Moreover, we provide a description of other methods to collect qualitative data.

Methods for content analysis

The variable amount of information concerning the qualitative content analysis represented three modalities: 1) no information given by the authors, 2) only content analysis methodology cited, and 3) qualitative content analysis methodology specified and detailed.

The distribution of this variable was determined. Multiple coders of the same material were checked. If the authors did not explicitly describe multiple coding, inter-coder agreement or how coders resolved disagreements, we considered no multiple coding. If the coding transcript was performed by two researchers, this did not mean that they both worked on the same transcripts, so we considered no multiple coding. The use of software for qualitative research methods was identified. We give the percentage use of these two last indicators in articles. Furthermore, we give an overview of software used. Computer-Assisted Qualitative Data Analysis Softwares (CAQDAS) support qualitative research. Advantages of using qualitative data analysis software include having improved the rigor of analysis (Cope, 2014).

Saturation means that sufficient data to account for all aspects of the phenomenon have been obtained [START_REF] Morse | The Significance of Saturation[END_REF]. It is achieved when concepts and sub-concepts cannot be further specified with additional interviews or focus groups. We identified the percentage of articles describing saturation.

Professional expert input

We considered as professional experts any professionals defined as such by authors in articles. Experts provided input in several methodological approaches: interviews, focus groups, consultations, Delphi, brainstorming. The Delphi method, used for consensusbuilding, involves a series of questions (J. [START_REF] Skulmoski | The Delphi Method for Graduate Research[END_REF]. We calculated the percentage of articles describing experts involved in concept elicitation and the distribution of methods for expert contribution.

Debriefing of the newly developed items

The debriefing session for the newly developed items represents the final possibility to adjust the tool before validating the psychometric proprieties of the questionnaire. Newly developed items can be submitted to the population of interest, experts or both. Individual interviews can be used for debriefing. The cognitive interview is a way to perform an individual interview with participants from the population of interest defined as the administration of draft survey questions while collecting additional verbal information about the survey responses. This technique is used to evaluate the quality of the response or to help determine whether the question generates the information the author intends [START_REF] Beatty | Research Synthesis: The Practice of Cognitive Interviewing[END_REF]. The thinkaloud procedure is a form of cognitive interview in which interviewers encourage participants to verbalize thoughts while answering questions. In numeric debriefing, participants rate each item on a Likert scale for several variables (relevance, comprehensiveness of content, clarity of presentation, ease of administration etc.). The Content Validity Index (CVI) is a method of numeric debriefing that uses ratings of item relevance [START_REF] Polit | Is the CVI an acceptable indicator of content validity? Appraisal and recommendations[END_REF] and the Content Validity Ratio (CVR) is an index of inter-rater agreement for questionnaire items used with dichotomous ratings on items. Focus groups or the Delphi can also be used. The distribution of methods used in this debriefing step was described separately for the population of interest and for experts.

Assessment criteria of the use of recommendations

From recommendations and after a first round of reading articles to identify the potentially exploitable elements, we identified six quality binary criteria (presence/absence) for assessing report of qualitative research methods: semi-structured individual interviews, focus groups, description of qualitative content analysis, multiple coding, saturation, and submission of further newly developed items to the population of interest. We summarized the presence of these criteria of the use of recommendations.

Results

Search strategy results

The search strategy yielded 2449 citations. After duplicates were removed, 2183 articles remained. Following analysis of abstracts, 1662 articles were excluded and from 521 articles remaining for full text assessment, 150 were excluded; finally 371 articles were eligible.

General information

The number of articles included in the review increased steadily each year from 2000 to 2014 (Figure 2, supplemental file). Only 4 articles were identified in 2000 but 57 in 2014.

Concerning journal domain, articles were published in clinical (n = 140, 38 %), Health sciences (n = 88, 24%), Psychology and Psychiatry (n = 61, 16%), Nursing (n = 43, 12%), Social sciences (n = 20, 5%) and Others (n = 19, 5%). Written answers to open ended questions 16 4

Qualitative data collection from the population of interest

Note. Two articles used two types of individual interviews (semi-structured interviews followed by structured interviews, and unstructured interviews followed by semi-structured interviews). For presentation reasons, only the first interview strategy was counted.

In 266 articles (72%), individual interviews from the population of interest were described for collecting qualitative data (Table 1). The semi-structured interview was by far the most frequently reported interview method reported (n = 172, 65%). In 103 articles (39%), no information concerning the type of structure of the research interviews was available. The focus group method was used for 150 articles (40%). Finally, only 57 articles (15%) described mixed individual interviews and focus groups for the population of interest. In 16 articles (4%), written responses to open ended questions were collected.

A total of 6 articles reported at least one other approach to collect qualitative data;

spontaneous writing was identified in one article, with personal letters written by patients analyzed. Anorexic patients were asked to write two letters, one addressed to a friend and the other to an enemy [START_REF] Serpell | The development of the P-CAN, a measure to operationalize the pros and cons of anorexia nervosa[END_REF]. In one study transcripts of the full course of psychotherapy were analyzed [START_REF] Carter | Patient change processes in psychotherapy: Development of a new scale[END_REF]. A diary approach was used alone [START_REF] Wu | Development and validation of the effectiveness of [corrected] auditory rehabilitation scale[END_REF], combined with observational research [START_REF] Tol | Developing a function impairment measure for children affected by political violence: a mixed methods approach in Indonesia[END_REF] or with written answers to open-ended questions [START_REF] Wu | Development and validation of the effectiveness of [corrected] auditory rehabilitation scale[END_REF]. An observational research [START_REF] Jap | The development of indonesian online game addiction questionnaire[END_REF] was also identified. In observational research, authors conducted field observations in natural surroundings. The observer is immersed and a member of the setting in which data are collected.

No emerging approach was identified. With the exception of patient letter analysis and transcripts of the course of psychotherapy, all methodologies were already mentioned by [START_REF] Pope | Qualitative research in health care. Analysing qualitative data[END_REF]: "qualitative research can produce vast amounts of data. These may include (…) detailed fieldnotes of observational research, a diary or chronological account".

Methods for content analysis

In 118 articles (32%), no information was given concerning the qualitative content analysis, and in 95 articles (25%) only the methodology was cited. Consequently, the methodology of content analysis of qualitative data was described in 158 articles (43%). Only 24% of the articles (n = 89) referred definitely to a multiple coding of the same material in the analysis of verbatim or any material collected. In 80 articles (22%), authors described the use of computer-assisted qualitative-data analysis software programs: NVivo (n = 46, including 11 articles that used NUD*IST, the previous version of NVivo), Atlas.ti (n = 26), Ethnograph (n = 6), Leximancer (n = 1) and Kwalitan (n = 1). Overall, 269 articles (73%) did not take saturation concept into consideration. For 108 articles (29%), experts were involved in concept elicitation for questionnaire development. Experts' involvement at this stage most of the time represented consultation for their opinion (n = 51, 47%). Hence, methods to collect experts' contribution were standardized only in 57 articles (53%).

Professional expert input

Debriefing for newly developed items

After the concept elicitation stage, 226 articles (61%) described a step of submitting new developed items to the population of interest and in 147 articles (40%) to experts. For 94 articles (25%), the draft was submitted to both the population of interest and experts. Overall, 278 articles (75%) described a refinement step for the newly developed items from the population of interest or from experts, or both. Concerning the population of interest debriefing, 14 articles described a double feedback mechanism with the population of interest. The individual qualitative interview was the most common way to refine newly developed items (n = 125). In 70 articles, the authors used the precise terms "cognitive interview" or "debriefing cognitive" and in 14 articles, authors clearly spoke about the "think-aloud" debriefing methodology. From 2009, authors recommended the cognitive interview method to debrief newly developed items [START_REF] Brod | Qualitative research and content validity: Developing best practices based on science and experience[END_REF][START_REF] Lasch | PRO development: rigorous qualitative research as the crucial foundation[END_REF]Patrick et al., 2011a[START_REF] Patrick | Content Validity-Establishing and Reporting the Evidence in Newly Developed Patient-Reported Outcomes (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 2-Assessing Respondent Understanding[END_REF]. Between 2000 and2008, 15 articles reported the use of a cognitive interview (12% of articles over the period), 69 articles between 2009 and 2014 (28% of articles over the period). So, an increase in cognitive interviews was observed from 2009. Furthermore, when a debriefing stage was mentioned, 62 articles (27%)

gave no concrete information about the design of consulting the population of interest.

Concerning the professional experts' debriefing, 4 articles described a double feedback mechanism with experts. Input from experts in this debriefing phase was mainly numeric (n = 65, including 24 with CVI, plus 3 with CVR). Furthermore, for 66 articles (45%), the debriefing with experts represented a consultation with no information reported about the precise design of the debriefing.

Assessment criteria of the use of recommendations

Table 4 Presence of criteria of the use of recommendations for the main steps of the qualitative research method

Step of qualitative research method Criteria of the use of recommendations For qualitative interviewing, the level of use of methods of semi-structured individual interviews and focus groups is consistent with recommendations (65% and 40% of articles, respectively). However, 213 articles (57%) provided no description of qualitative data analysis. Multiple coding and assessment of saturation were not common (24% and 27% of articles, respectively). In 226 articles (61%), new items were further submitted to the population of interest.

General Discussion

This systematic review addressed how qualitative methods were used to generate items for questionnaires in light of recommendations [START_REF] Brod | Qualitative research and content validity: Developing best practices based on science and experience[END_REF][START_REF] Lasch | PRO development: rigorous qualitative research as the crucial foundation[END_REF]Patrick et al., 2011a[START_REF] Patrick | Content Validity-Establishing and Reporting the Evidence in Newly Developed Patient-Reported Outcomes (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 2-Assessing Respondent Understanding[END_REF]. With a large number of articles analyzed (n = 371), the number of published articles accelerated over the study period, individual interviews and focus groups were common ways of generating questionnaire items but no emergent approach was identified, the content analysis methodology was usually not described, experts were involved in eliciting concepts in less than a third of articles, and more than half of the studies involved a step of further submission of newly developed items to the population of interest.

This analysis showed an insufficient reporting of qualitative research methods used to generate questionnaires despite previous clear, useful and suitable recommendations. Thus, recommendations seem to be often ignored probably due to a lack of dissemination among authors who engage in a process of items generations.

A tendency in the development of new questionnaires to be directly extracted from statements of the population of interest was pointed out. However, this effect may be artifactual owing to better access to publication for this kind of research. Moreover, articles were mainly published in clinical journals and health science journals. Individual interviews and focus groups were the two predominant methods to collect the perspectives of the population of interest. Semi-structured interviews were by far the most frequently reported interview method, which agrees with recommendations highlighting the development of a clear openended interview guide to avoid the undue influence of the researcher, interviewer, or interview guide bias [START_REF] Lasch | PRO development: rigorous qualitative research as the crucial foundation[END_REF]. However, 103 articles (39%) gave no information about the choice of interview strategy even though qualitative research should have a strong and documentable scientific basis for content validity development [START_REF] Brod | Qualitative research and content validity: Developing best practices based on science and experience[END_REF].

We found a low level of the combination of individual interviews and focus groups with the population of interest in concept elicitation, but the literature contains no clear-cut guidelines on combining individual interviews and focus groups for the question of interest. Focus groups and individual interviews should be considered complementary because they may provide different information [START_REF] Brod | Assessing the impact of growth hormone deficiency and treatment in adults: development of a new disease-specific measure[END_REF]Mays & Pope, 2000). However, to our knowledge, these considerations are based on intellectual arguments and are not evidence-based. One study found that item generation based on participants' viewpoints expressed during interviews was more valid than item generation based on patient focus groups [START_REF] Rat | Content of quality-of-life instruments is affected by item-generation methods[END_REF]. For the other approaches used to collect qualitative data, we identified no innovative approach, even though our search terms were sufficiently broad so as to not miss new emergent methodologies.

Regarding the lack of information concerning the content analysis methods used, results are consistent with statements by [START_REF] Lasch | PRO development: rigorous qualitative research as the crucial foundation[END_REF] who found that the systematic analysis of qualitative data was usually neither documented nor transparent.

To reach saturation, data are collected until no new information is obtained. So saturation is a key component for the content validity. But only 27% of our articles (n = 102) focused on saturation.

Another aspect of this systematic review concerned professional experts' input in concept elicitation. This issue is discussed and still debated in the literature. Some think that qualitative input from experts is crucial to generate ideas for possible draft items. In fact, some claim that the first thing the researcher needs to do is conduct interviews with practitioners and/or subject-matter experts [START_REF] Mackenzie | Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and Existing Techniques[END_REF].

However, others consider that participants can generate the complete content set of items, feeling that often nothing further is added by experts [START_REF] Boyer | Assessment of psychiatric inpatient satisfaction: A systematic review of self-reported instruments[END_REF]. In this systematic review, experts' involvement at the stage of concept elicitation was not a general practice (29%). As well, their contribution often involved consultation of their opinion, a vague practice that does not meet the basic criteria for medical research.

In 61% of articles, the population of interest was used for debriefing newly developed items, which agrees with recommendations. Moreover, we observed an increase in use of cognitive interviews over the years, which agrees with recommendations issued since 2009.

Descriptions of this debriefing phase lacked information on the debriefing design for the population of interest or experts. Consequently, in these articles, we did not have access to the data required to assess the process of the research.

Finally the key issue would be the quality of items generated. It would be the ultimate outcome criteria for comparing the benefit of various qualitative methods. Unfortunately, no indication of the quality of items generated was given in a single paper that can be compared to items quality generated in another single paper. With rare exception [START_REF] Rat | Content of quality-of-life instruments is affected by item-generation methods[END_REF], we could not identify papers formally comparing this issue across different methods. So we have to acknowledge that reporting the method used is only a preliminary step toward transparency, not evidence-based quality, and that it cannot be used as a surrogate for quality of outcome.

Strengths and limitation

Our search strategy was very careful. Our search terms were encompassing and did not refer to a particular qualitative method; thus, we ensured the detection of potential emergent or innovative methods. Titles and abstracts were double-screened to reduce the risk of rejecting relevant articles.

The assessment of qualitative research [START_REF] Tacconelli | Systematic reviews: CRD's guidance for undertaking reviews in health care[END_REF] within articles was not taken into consideration as required by PRISMA (Moher, Liberati, Tetzlaff, & Altman, 2009). That would have probably increase the robustness of the systematic review findings. But with 371 articles included, formal quality assessment of articles would have been too complicated.

However, our systematic review was not completely exhaustive because it was restricted to articles available at MEDLINE and at PsycINFO, but the risk of lack of exhaustiveness of methods detected was limited by the large number of articles analyzed (n=371).

Conclusion

This overview of qualitative research methods used in concept elicitation for questionnaire development indicated 1) an acceleration of articles published over the years concerning the use of qualitative data from the population of interest for questionnaire development and 2)

individual interviews and focus groups as common practices to generate questionnaire items.

However, a comparison of data collected in our systematic review with recommendations

showed insufficient reporting at all stages of qualitative research (data collection, content analysis and debriefing of the newly developed items). Several papers specifically provided recommendations to ensure rigor in qualitative research but recommendations are not often implemented. Particularly, the qualitative approach was not documented in articles. Although we did neither explore the journal requirements in these domains nor their potential changes over the study period, we think that there is a persistent underestimation of importance of reporting the details of qualitative methods and a lack of demand of reviewers or editors,.

Without descriptions and documentation, replication is not possible, the notion of replication being a priority in science. Authors should better document qualitative research methods used to establish content validity in the development of new questionnaires. Our systematic review would serve to further the science of qualitative research.

[

  qualitative[All Fields] AND (("weights and measures"[MeSH Terms] OR ("weights"[All Fields] AND "measures"[All Fields]) OR "weights and measures"[All Fields] OR "scale"[All Fields]) OR (("research design"[MeSH Terms] OR ("research"[All Fields] AND "design"[All Fields]) OR "research design"[All Fields] OR "test"[All Fields]) AND construction[All Fields]) OR (("surveys and questionnaires"[MeSH Terms] OR ("surveys"[All Fields] AND "questionnaires"[All Fields]) OR "surveys and questionnaires"[All Fields] OR "questionnaire"[All Fields]) AND ("growth and development"[Subheading] OR ("growth"[All Fields] AND "development"[All Fields]) OR "growth and development"[All Fields] OR "development"[All Fields]))) AND ("Items"[Journal] OR "items"[All Fields]) AND (("2000/01/01"[PDAT] : "2014/12/31"[PDAT]) AND English[lang])]

Figure 1 (

 1 Figure 1 (Flow chart of selection of articles) and Full reference list are available in

Table 1

 1 Characteristics of qualitative data collection methods

	N	%

Table 2

 2 Expert contributions

	Unique method of input	N	%
	Opinion consultation	51	47
	Interviews	30	28
	Focus groups	18	17
	Delphi	3	3
	Brainstorming	2	2
	Double method of input	N	%
	Interviews plus focus groups	4	4
	Total	108	100

Table 3

 3 Methods of input from the population of interest and experts in the debriefing stage

	Methods of Input	Population of interest (N)	Experts (N)
	Individual Interviews	125	5
	Cognitive interviews	70	/
	Think aloud	14	/
	Numeric	33	65
	CVI	4	24
	CVR	/	3
	Focus groups	16	5
	Delphi	2	7
	Written feedback	2	2
	No information about the method	62	66
	Note. CVI, Content Validity Index; CVR, Content Validity Ratio

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for profit sectors.

Declaration of conflicting interests

The Author(s) declare(s) that there is no conflict of interest.