Thomas Remen

Lesley Richardson

Corinne Pilorget

Gilles Palmer

Jack Siemiatycki

Jérôme Lavoue

Richardson

Siemiatycki

2018 -CAPS-Canada tool DEVELOPMENT OF A CODING AND CROSSWALK TOOL FOR OCCUPATIONS AND INDUSTRIES

Keywords: agreement, computer-coding assistant, job-exposure matrix

teaching and research institutions in France or abroad, or from public or private research centers.

 [START_REF] Conk | Occupational Classification in the United States Census: 1870-1940[END_REF][START_REF] Woollard | The classification of occupations in the 1881 census of England and Wales[END_REF]. They started as listings of occupations without any hierarchical structure, and tended to reflect social strata rather than tasks performed. In the past 50 years, such classifications have grown increasingly detailed based on the tasks and responsibilities of workers. It has long been the practice to use such classifications to code the jobs of workers in epidemiologic studies and to conduct analyses of disease risks in different occupations and industries [START_REF] Mannetje | The use of occupation and industry classifications in general population studies[END_REF][START_REF] Arheart | Occupational vs. industry sector classification of the US workforce: which approach is more strongly associated with worker health outcomes?[END_REF]. Such procedures are relatively inexpensive, and they can be useful for hypothesis-generation, but the occupation code is a fairly crude indicator of occupational exposure, which is usually of greater interest [START_REF] Siemiatycki | Discovering carcinogens in the occupational environment: a novel epidemiologic approach[END_REF][START_REF] Siemiatycki | Risk Factors for Cancer in the Workplace[END_REF].

While many consider expert assessment to be the most valid approach for retrospective exposure assessment in case-control studies [START_REF] Mcguire | Assessment of occupational exposures in communitybased case-control studies[END_REF][START_REF] Teschke | Occupational exposure assessment in case-control studies: opportunities for improvement[END_REF][START_REF] Fritschi | Validation of expert assessment of occupational exposures[END_REF], it is very costly and time-consuming [START_REF] Mcguire | Assessment of occupational exposures in communitybased case-control studies[END_REF]. An alternative method of exposure assessment is to use a job exposure matrix (JEM). Most JEMs use standard occupation (or industry) classification as the job axis, but there are many such classifications, both national and international, from which to choose. Thus, an important component of occupational study design is the choice of the occupation (or industry) classification system to use for the job histories collected.

Job coding into a standard occupation or industry classification system is both important and imperfect. The classification system may be imprecise or it may not align perfectly with the job description from records or subjects, and there are subjective judgements about how to fit one set of words against another. All this leads to loss of validity and inter-rater reliability of coding [START_REF] Mannetje | The use of occupation and industry classifications in general population studies[END_REF][START_REF] Pilorget | Evaluation of the quality of coding of job episodes collected by self questionnaires among French retired men for use in a job-exposure matrix[END_REF]. In some situations, as for exposure assessment through the use of a JEM or for participation in multi-center efforts, the investigator may need to code the jobs into more than one classification system. The coding can be done manually based on original job descriptions in the questionnaire. However, coding hundreds or thousands of jobs into a new occupation or industry classification is costly. Alternatively, the job codes can be translated by the use of crosswalks. National or international bodies that create such occupation/industry classification systems sometimes provide official crosswalks to other systems or between older and newer versions. However, these cannot be applied directly as they contain many links that are not univocal, i.e. one-to-many, many-to-one, or many-to-many. For instance, the translation of the ISCO-1968 code 7-76.10 "Baker, General" provides three different possibilities for ISCO-2008: 3122 "Manufacturing supervisors Official"; 7512 "Bakers, pastry-cooks and confectionery makers" and 8160 "Food and related products machine operators". Such links have to be resolved individually based on expert opinion as to which possibility is the "best" or "most likely", and this in turn leads to measurement error in subsequent exposure assessment. The objectives of this paper are threefold: we aim to present CAPS-Canada and its development, describe how the system of crosswalks was implemented, and assess the contribution of its different functions through an inter-rater reliability study.

The University of Montreal

METHODS

A. Development of the CAPS-Canada tool

The current version of CAPS-Canada includes two major functions: a coding assistant, and a crosswalk assistant. From these official classifications provided by national or international agencies, the following items of information were extracted in separate fields for each occupation or industry code: main title, job CAPS-Canada includes a search engine based on keywords (entered by the coders) that functions as follows: each time a keyword (or its inflected form) is found in one of the fields of a record, a score is attributed to the record. The score varies depending on where the keyword is found in the various fields of information available in the classification; a weight of 1.5 is allocated for keywords found in the main title while a score of 1.0 is allocated for keywords found in the other fields such as « definition" or "synonyms" (this was in part based on the idea that occupations with the keyword in the title would be likelier candidates, in part a practical decision to avoid a long list of potential codes with the same score). Only one score is attributed for each keyword; if a keyword is found in several fields, only the maximum score is attributed. The final score for a record is the sum of the scores across all keywords. All records with a non-null score are presented, ranked according to their score. It is possible to customize the parameters: users with administration privileges can include or exclude any field from the indexing, and also attribute new weights to each field, separately for each classification.

A.2. Crosswalk assistant CAPS-Canada also includes a crosswalk assistant for translation of a code from one classification to another. This tool was built by starting with "official" crosswalks (defined below), and supplementing those with two data sources from our studies, which we label "empirical"; and "expert" links.

"Official" links National or international agencies that have created occupation or industry classifications sometimes published crosswalks to other systems. We used such tables to establish links among the classifications included in CAPS-Canada, and refer to them as "official" links. These links simply provide, for a given occupation (or industry) code in a starting classification, the corresponding list of potential codes in the selected target classification. The links are sometimes one-to-one matches of codes, but most links point to multiple possible codes in the target classification (as shown in Table 1). Where no official crosswalk exists between a given pair of classifications, it was possible to establish indirect linkages via multiple existing crosswalks. For example, there is no direct official crosswalk between ISCO-1968 and US-SOC-2010 codes, but there were official crosswalks between ISCO-1968and ISCO-1988, between ISCO-1988and US-SOC-2000, and between US-SOC-2000 and US-SOC-2010. The sequence of crosswalks used to establish a link between ISCO-1968 and US-SOC-2010 is illustrated in Figure 1. Given the sequence of crosswalk steps, the links are rarely univocal, with frequent occurrences of one-to-many or many-to-one associations, as illustrated in Appendix A.

Figure 1: Description of the several steps used for "official" crosswalks between ISCO-1968 and US-SOC-2010 classifications "Official" links constitute the default value in CAPS-Canada, when there are no available "empirical" or "expert" links (see below).

If a link for a code in one classification leads to a large number of potential codes in the target classification, the value of such a link in helping a user translate the initial code is questionable.

Hence, links for starting codes for which there were more than 10 possible codes in the target classification were not kept in CAPS-Canada. Information on how frequently this occurred for "official" links in our study is provided in Table 1.

"Empirical" links "Empirical" links are those that we created using a database of jobs that had previously been "manually" coded in several systems. The jobs coded came from subjects who had participated in four case-control studies conducted by our team in Greater Montréal, Canada [START_REF] Gerin | Obtaining occupational exposure histories in epidemiologic case-control studies[END_REF][START_REF] Labreche | Occupational exposures to extremely low frequency magnetic fields and postmenopausal breast cancer[END_REF][START_REF] Ramanakumar | Risk of lung cancer following nonmalignant respiratory conditions: evidence from two case-control studies in Montreal, Canada[END_REF][START_REF] Study | Brain tumour risk in relation to mobile telephone use: results of the INTERPHONE international case-control study[END_REF][START_REF] Lavoue | 0382 CANJEM: a general population job exposure matrix based on past expert assessments of exposure to over 250 agents[END_REF][START_REF] Kirkham | Impact of aggregating exposure information from cases and controls when building a population-based job-exposure matrix from past expert evaluations[END_REF]. Each participant in these studies was asked to provide a detailed description of each job they had held in their working lifetime. Each job description was initially coded by a team of experts using Canadian occupational classification (CCDO-1971) and Canadian industrial classification (SIC-1980). More recently, the jobs were coded in the other five classifications mentioned above by the 2 coders (DR and EA) that participated to this work. All of coding was performed "manually" by expert coders. In total, 31 673 jobs were coded in each of the seven classification systems.

To simplify the explanation, let's consider that there are two occupation classification systems A and B, and we wish to create an empirical crosswalk from A to B. For simplicity we will refer to the job code i of classification A as A-i. Suppose there were 50 jobs in our database that were assigned in classification A to code 1 (i.e. A-1). Suppose that in our database, of those same 50 jobs, using classification B, 40 had been assigned code B-1 and 10 had been assigned code B-2. Therefore, starting from job code A-1, the most probable corresponding code in target classification B was B-1 (80%), followed by B-2 (20%). This constitutes an empirical link from classification-code A-1 to classification-codes B-1 and B-2. Such a calculation was performed between every pair of codes across all pairs of classification systems.

Two additional restrictions were used in the creation of empirical links : (i) the starting code involved at least ten jobs in our database in order to ensure a minimal accuracy of the frequencies, and (ii) when a starting code is associated with more than ten codes in the target classification, the cumulative relative frequency of the 10 most frequent target codes is greater than or equal to 80% (i.e. the 10 most frequent target codes represent at least 80% of the jobs).

As an illustration, in our database, the CCDO-1971 code 1130-126 -"General Manager, finance (bank. & finance)" was assigned to 12 jobs. It is linked to the two following ISCO-1968 codes: (i) 2-19.50 -"Budgeting and Accounting Manager" for 83% and (ii) 2-11.10 -"General Manager" for 17% of 12 jobs. By contrast with "official" links which provide all possible target classification codes for each starting classification code, the empirical links display the probability of each target classification code for each starting classification code.

"Expert" links "Expert" links consist of a one-to-one association based on expert opinion.

Each job description in our epidemiological database was initially coded in CCDO-1971 andSIC-1980. As mentioned above, two coders (those involved in the current study) later added the five other classifications. At this stage, the two coders identified some connections where, for a starting CCDO-1971code (or SIC-1980 code), they always assigned the same occupation (or industry) code in another classification even if several possible codes existed. Such connections were discussed between them and, when consensus was reached, saved as "expert" links.

For example, although the CCDO-1971 code 3131-130 "Nurse, general duty (medical)" is possibly connected to seven different ISCO-1968 codes, the coders decided by consensus to always use code 0-71.10 "Professional nurse (medical)". Thus the translation of CCDO-1971 code 3131-130 to ISCO-1968 code 0-71.10 is an "expert" link.

For one code in a starting classification, an official link would be available most frequently in CAPS (an official link would always exist in the official crosswalk, but it might not be shown in CAPS if the number of potential target codes is greater than 10). In a smaller number of cases, empirical links would be available where enough jobs were present in our data to estimates probabilities (in addition to the two restrictions mentioned above). Finally expert links are limited to the starting classifications CCDO-1971 andSIC-1980, and by the relatively few situations where experts felt sufficiently confident to create a univocal association).

Contribution of "empirical" and "experts" links

As mentioned above, "Empirical" and "experts" links are not available for all possible pairs of codes in all pairs of classification systems. Thus it is not an option to build CAPS-Canada on only "empirical" and "experts" links. These are used when available, but the default when they are not available is the "official" link. In order to assess the marginal benefit of "empirical" and "experts" links as additions to the official links, we compared the use of the full crosswalk assistant (including "official"; "empirical" and "expert" links) with the use of "official" links only. We evaluated, for each possible pair of starting / target classifications, for what proportion of jobs in our database the crosswalk from the starting to the target classification would lead to useful information (i.e. no more than ten possibilities in the target classification). We were also interested in the ability of the crosswalk systems to be used to perform fully automated recoding, which require univocal links only. For this purpose we estimated for what proportion of jobs in our database the crosswalk from the starting to the target classification would provide only univocal links. For this calculation, we considered "empirical" links as univocal when, for a given starting code, a single target code represented at least 80% of all links in our database.

Hierarchy of systems when multiple systems are available in crosswalk assistant Given that CAPS-Canada combines three different systems of crosswalks (based on "official"; "empirical"; and "expert" links), more than one can be available for a particular starting code. In such a situation, only the crosswalk based on the most informative link was retained by CAPS-Canada. As "expert" links are univocal, when they exist, they override any other type of link.

Otherwise, when an "empirical" link is available, it overrides any "official" one.

B. Inter-rater reliability study

We conducted a study to assess how the use of CAPS-Canada influenced the agreement between our two coders (DR and EA) in comparison with that of conventional manual coding. Further, in addition to evaluating the performance of the full CAPS-Canada tool with all the crosswalk systems, we were interested in assessing the added value of including "empirical" and "expert" links to facilitate the user recoding choice. Therefore, three different approaches were compared with each other:

(i) Manual approach : manual coding based on published version of the classifications

With this first approach, the coders only had access to the electronic versions (Portable document Format or PDF) of the classifications and official crosswalks provided by the different national or international agencies.

(ii) Partial CAPS : CAPS coding assistant + "official" crosswalk assistant REMEN T -version du 10/04/2018 -CAPS-Canada tool With this second approach, the coders had access to the coding assistant and to the crosswalk restricted to "official" links.

(iii) Full CAPS: the full CAPS-Canada tool including all crosswalk types

This final version included the full coding and crosswalk assistants, including "official", "empirical" and "expert" links.

We used the inter-rater reliability of coding between two coders as the measure of quality of the job coding. As there is usually no gold standard when coding occupation or industry, and we were unable to have a group of coders create a consensus list of codes, inter-rater reliability between our two coders appeared a useful, albeit imperfect, proxy. It should, however, be remembered that perfect inter-rater reliability is not realistically attainable. Job coding comes with intrinsic variability stemming from incomplete task descriptions, task descriptions incompatible with a classification's structure, and personal interpretation of the coders.

Data source for the reliability study

The jobs were extracted from a population-based case-control study of lung cancer conducted from 1996 to 2001 (one of the 4 studies mentioned above). The study population consisted of males and females aged 35-75 years residing in Greater Montréal who were Canadian citizens. More details are available elsewhere [START_REF] Ramanakumar | Risk of lung cancer from residential heating and cooking fuels in Montreal, Canada[END_REF].

There was a total of 2,740 subjects in the study, and a total of 13,992 jobs. For the present trial, a random sample of 1,000 jobs was extracted.

Job coding

The sample of 1,000 jobs was randomly divided into 3 equal (333, 333 and 334) sized subgroups for each coding approach. The first subgroup was coded with the manual approach; the second by the partial CAPS approach sample, and the third by the full CAPS approach. All the jobs, using all three approaches, were independently coded by two expert coders based on job titles and tasks descriptions. They were tasked with coding the 1,000 jobs into the four occupation and three industry classification systems mentioned previously.

Note that with both (partial and full) CAPS approaches, the coders usually started to code one combination of occupation and industry classifications which they then attempted to convert to the other systems. They often used CCDO-1971 / SIC-1980 as starting classifications since they are very detailed.

All jobs were coded at the highest hierarchical level for each classificiation (e.g. 7-Digit CCDO-1971).

For the calculations involved in the inter-reliability study, codes for lower resolutions (e.g. 4-and 3-Digit CCDO-1971) were obtained by truncation.

Data analysis for the reliability study For each classification system and each level of resolution available in the respective system, we computed the inter-rater reliability (IRR, also called inter-rater agreement) in job coding. While percentage of agreement is an unsatisfactory measure of inter-rater reliability when there are few categories to be assigned and the probability of chance agreement is high, given that there are hundreds of occupation codes in these classification systems, the probability of chance agreement is very low. Consequently the percentage of agreement is in fact a good marker, and we assessed IRR by estimating the proportion of jobs coded identically by the two coders.

Results will be presented overall and by coding approach.

RESULTS

The analyses presented in this paper were performed for each of the four occupation classifications and each of the three industry classifications available in CAPS-Canada. However, we will focus on the occupation classifications in this paper, with a brief mention of industry specific patterns in the discussion (results related to industry classifications are available in supplementary material in the online edition).

A. Development of the CAPS-Canada tool

Crosswalks between classification systems

Table 1 shows the proportion of jobs in our database for which the CAPS-Canada crosswalks can provide "useful" information (i.e. no more than ten possibilities in the target classification), or for which the automated recoding could be implemented (i.e. univocal links only). To further illustrate the contribution of the CAPS-Canada crosswalks, the table shows the same results when restricted to "official" links.

The use of four different occupational classification systems led to the creation of 12 possible directional pairs of occupation classifications. Depending on the directional pair considered, using the full crosswalk assistant (all links combined), "useful" information was available for 85% to 99% of jobs (median 95%). In comparison, "official" links were available for far fewer jobs ranging from 0.5% for the ISCO-1968 to CCDO-1971 conversion to 93% for the US-SOC-2010 to NOC-2011 conversion, with a median of 61%. Retricted to univocal links only, the full crosswalk assistant could lead to automated recoding for 18% to 81% of jobs (median 56%). In contrast, "official" links could lead to automated recoding for 0% to 34% of jobs (median 3%) (Table 1).

When considering the relative contribution of the different link types to the overall proportion of jobs covered, expert and empirical links (which both override official links when available), generally represented the overwhelming majority of the information. Detailed results are presented in Appendix B.

B. Inter-rater reliability study

IRR -All approaches combined

Table 2 presents IRRs in job coding between the two coders for each classification used. They ranged from 35.7% (7-digit CCDO-1971) to 66.5% (3-digit CCDO-1971), with a median of 53.7%. When considering the classifications at their highest resolution level, lower IRRs were observed for classifications containing the highest number of codes. The lowest IRR value (35.7%) was observed for the 7-digit CCDO-1971 covering 7,907 different codes and the highest (53.9%) for the 4-digit NOC-2011 covering only 500 different codes. The IRR also consistently increased with lower resolutions of the classification system.

IRR by coding approach

Figure 2 shows for each classification the IRR for each coding approach used (manual, partial CAPS, full CAPS). The use of the full CAPS approach was associated with a consistent gain in IRR across all classifications and resolution compared to the manual approach. Across the 10 combinations of classification/resolution, the absolute gain in IRR ranged from 2.8% to 12.8% (median 6.7%). These patterns were similar for the industry analysis. Use of the "partial" CAPS approach (excluding empirical and expert links) was associated with an increased IRR for CCDO-1971 (absolute gain ranging from +3.0% to +5.1% depending on resolution) and ISCO-1968 (+2.1% and +2.4% for the 5-and 3-digit resolutions) classifications, but with decline for US-SOC-2010 (from -0.6% to -1.5% depending on resolution), and NOC-2011 (no change for the 4-digit and -3.0% for the 3-digit resolutions) classifications. In the industry analysis, a small increase was observed for ISIC-1971, but no increase or slight decrease for SIC-1980 and NAICS-2012.

DISCUSSION

The free online CAPS-Canada tool was developed to assist in coding occupations and industries. It combines a method based on a keyword search in the various fields of information (title, description, synonyms...) available in the documentation of each classification with a system of crosswalks between classifications. In this paper, we present a description of its development, as well as the contribution of crosswalks based on "empirical" and "expert" links to support translation into different classifications. Finally, we present an evaluation of its influence on inter-rater reliability between coders.

With the growing use of JEMs for assessment of occupational exposures, the ability to translate a set of coded occupations and industries into other systems becomes critical. Official crosswalks generally include many equivocal links due to differences in the philosophy behind the coding structure, or because of different levels of resolution. As illustrated in table 1, this means that their practical use for recoding from one classification to another is very limited in most cases. Recoding manually the individual jobs would be prohibitive in terms of cost. Some custom crosswalks were developed by experts determining for each starting code the closest equivalent in the target classification [START_REF] Koeman | JEMs and incompatible occupational coding systems: effect of manual and automatic recoding of job codes on exposure assignment[END_REF]. Creating such systems involves significant costs and can lead to coding errors and loss of specificity. Koeman et al. assessed the extent of these errors based on a limited set of 200 jobs [START_REF] Koeman | JEMs and incompatible occupational coding systems: effect of manual and automatic recoding of job codes on exposure assignment[END_REF]. The authors compared the codes attributed by 2 recoders with codes attributed automatically by custom crosswalks translating from a Dutch occupation classification into ISCO-1968 andISCO-1988 codes. Similar agreement was observed between the expert-crosswalk and each recoder and between the 2 coders. The non-official crosswalk system we created, based on the coding of >30,000 jobs in 7 classiffications, offers an interesting alternative to reasearchers wishing to translate their initial coding into other systems.

With a restricted list of likely codes provided for a median of 95% of jobs, and over 50% of them potentially automatically coded, it becomes both viable and efficient to avoid recourse to a custom crosswalk, especially if such a crosswalk isn't already available. Some limitations inherent to the addition of the expert and empirical crosswalks must be acknowledged. Considering that empirical links correspond to at least 80% relative frequencies in our source database (so one might expect up to 20% of errors) and expert links depend on expert judgement, it means that crosswalk translations for occupations or industries that were infrequent or unusual in our database might be unreliable. However, these risks of errors appear limited in comparison to what we know about the accuracy of manual coding (discussed below).

The generalizability of this tool to users in other countries can be addressed from two viewpoints, coverage and validity. The default "official" links underlying CAPS-Canada are not dependent on the Montreal situation, and are thus universally valid, or as valid as official crosswalks can be. The supplementary information that we built into CAPS does come from an urban Canadian population that was active from the 1930s to 2000s. We are confident that, given the large number of jobs that we evaluated in the course of those studies that underlay the creation of empirical and expert linkages, the occupations and industries covered are sufficiently wide-ranging to cover occupations that might occur in countries of similar state of industrialization as Canada. As for validity of the expert and empirical links outside our own study population, this also seems to us like a phenomenon that would be quite stable across urban industrial populations.

There have been few previous studies of agreement between coders in assigning occupation or industry codes. An overview of some re-coding trials shows IRR ranging from 44% to 89% for occupation coding and from 59% to 98% for industry coding at the 4-5 digit level [START_REF] Mannetje | The use of occupation and industry classifications in general population studies[END_REF]. The moderate-to-good IRRs observed in our study at the finest resolution are in the same magnitude, ranging from 43% [5-digit ISCO-1968] to 53% [4-digit NOC-2011] for occupations. As expected, the higher the resolution of the occupational code, the greater the opportunity for disagreement, the lower the IRR values of inter-rater concordance.

All the analyses presented in this manuscript on occupation classifications were also performed with the industry classifications included in CAPS-Canada. The corresponding results can be found in an online supplementary document. The patterns observed for industry classifications were similar to those observed for occupation classifications, albeit with consistently higher inter-rater reliability for analyses of industry classifications (for comparable numbers of codes).

When comparing the different approaches used for coding, consistent improvement was associated with the use of the full CAPS tool in comparison to the PDF approach. Compared to the manual approach, the contribution of the keyword search was inconsistent and differences were within the confidence intervals. The strongest signal was observed when coding with CCDO-1971, for which it is plausible that even a simple keyword search and classification browsing functionality would alleviate the considerable complexity of a system with more than 7,000 codes. Most of the gain therefore occured through the addition of the expert and empirical crosswalks. This might seem counterintuitive, especially given the increase was significant for CCDO-1971, the classification system usually used first for the coding task. Discussions with the coding experts revealed that they used the conversions even during the selection of the first code, as a means to navigate across classifications in different directions to make sure the various codes were all consistent with the task descriptions.

The observed improvement therefore likely represents an improved general confidence caused by easily linking codes between systems. Because the three methods were tested sequentially, one might also suspect a gradual improvement in coder's ability. However the whole study was performed in three months, and the two coders already had more than a year's past experience in job coding. As noted in the methods, inter-coder reliability is an imperfect metric to measure quality of coding. Hence, the observed increase in IRR might mean CAPS-Canada merely makes the coders' mistakes more consistent. While this might be of concern for more automated coding tools in which coder input is limited, we believe it is unlikely that such a phenomenon would explain our results.

Ultimately we cannot entirely exclude it as a possible explanation.

Computer-based tools are recognized as cost-efficient alternatives to manual coding or recoding [START_REF] Patel | Performance of automated and manual coding systems for occupational data: a case study of historical records[END_REF][START_REF] Russ | Computer-based coding of free-text job descriptions to efficiently identify occupations in epidemiological studies[END_REF]. Indeed, even if they do not improve the quality of coding, one study reported a time reduction in coding of 13-23% [START_REF] Bushnell | An Evaluation of Computer-Assisted Occupation Coding: Results of a Field Trial[END_REF]. In our study, because of technical difficulties during the trial associated with server slowdowns, we were unable to formally assess the time-efficiency of the three tested approaches. The coders, however, when asked to subjectively assess their perception of gain in time, provided an estimate of approximately 40% decrease when the tool functioned as intended.

A wide range of computer-based tools have been developed during the last decades to assist users in job coding. Some automatically interpret job descriptions, while some provide assistance either to professional coders or directly to subjects from a study [START_REF] Ossiander | A computer system for coding occupation[END_REF]NIOSH, 2012;[START_REF] Russ | Computer-Based Coding of Occupation Codes for Epidemiological Analyses[END_REF][START_REF] De Matteis | Occupational self-coding and automatic recording (OSCAR): a novel web-based tool to collect and code lifetime job histories in large population-based studies[END_REF]. For instance, the NIOSH Industry and Occupation Computerized Coding System (NIOCCS) is a web-based software tool designed to translate occupation and industry desriptions into standardized codes. Records that meet user-specified autocoding confidence criteria are automatically coded, while the others are transferred into a computer-assisted coding module (NIOSH, 2012). SOCcer performs a sophisticated analysis of the words in the text format job descriptions, comparing them to the words in the classification, in order to provide a list of recommendations [START_REF] Russ | Computer-based coding of free-text job descriptions to efficiently identify occupations in epidemiological studies[END_REF]. All these systems are, however, specific to a single occupation and/or industry classification systems. CAPS-Canada is simpler than most of these tools, as it can be seen as an advanced computerization of the official classification documentation.

Hence, the coding assistant included in CAPS requires a coder to interpret job and task descriptions and choose keywords, and then select the proper code among the ranked possibilities. However, it is important to note that CAPS-Canada covers 7 occupation and industry classifications, including international systems, and was built to allow easy integration of additional classifications (SIC-1970 andISCO-2008 are now available) and crosswalks between classifications (official conversions, or user-provided conversions). Finally, the empirical and expert conversions included in CAPS-Canada, while directly available in the online tool, can be requested from our group for automated batch conversion of existing coded datasets.

CONCLUSION

CAPS-Canada combines coding and crosswalk assistants that improve upon manual coding and the use of official crosswalk tables. Such a tool would be useful for professionals involved in job-coding and it is of particular interest for those who are required to code jobs into more than one occupation or industry classification or to recode jobs from one classification to another. For each possible pair of starting / target classifications was evaluated for what proportion of jobs in our database the crosswalk from the starting to the target classification would lead to:

Contribution of the authors

α "< 10 codes proposed" = usefull information provided

  occupation and industry were first developed in the last decades of the 19 th century to serve the needs of population censuses

  Hospital Research Centre has partnered with the French National Institute for Health Surveillance (ex-InVS now Santé Publique France) and the French Center for Research and Development in Medical Informatics (CREDIM) to develop a tool to assist in coding of occupations and industries. The tool is known by its French acronym, CAPS 1 . Initially created by Santé Publique France to support occupation coding efforts in France, CAPS was further developed between 2012-2015. Two versions of the CAPS tool were developed: the French version (http://www.caps-france.fr/) which covers International, European and French occupation and industry classifications, and the Canadian version (http://www.caps-canada.ca/) which covers International, Canadian and U.S. classification systems. Both versions are bilingual (English and French), integrate a computer assisted coding tool that allows finding the most suitable code by 1 Assisted Coding of Occupations and Industries. keyword search and include a crosswalk assistant based on official crosswalks between classifications. The Canadian crosswalk function is augmented by an additional system of crosswalks to reduce the number of equivocal links between classifications.

A

  .1. Coding assistant CAPS-Canada allows coding into seven classifications, four being related to occupations: the International Standard Classification of Occupations of 1968 (ISCO-1968), the Canadian Classification and Dictionary of Occupations of 1971 (CCDO-1971); the National Occupational Classification of 2011 (NOC-2011) from Canada and the US Standard Occupational Classification of 2010 (US-SOC-2010); and three related to industries: the United Nations Industrial Classification of All Economic Activities of 1971 (ISIC-1971), the Canadian version of North American Industry Classification System of 2012 (NAICS-2012) and the US Standard Industrial Classification of 1980 (SIC-1980).

  REMEN T -version du 10/04/2018 -CAPS-Canada tool description, associated titles (includes synonyms, examples, sub-occupation title, or even user defined titles).

Figure 2 :

 2 Figure 2: Inter-rater reliability (and 95% confidence interval) in occupation coding according to the

β

  "direct link to 1 code" = univocal links

α

  Number of different codes for the classification β Number of intermediate tables used between the starting and target classifications for "official" crosswalks γ Median and values of the 5 th and 95 th percentile of the distribution of numbers of possible codes in the target classification linked with each code in the starting classification

Figure S1 :

 S1 Figure S1: Inter-rater reliability (and 95% confidence interval) in industry coding according to the

Table 1 :

 1 Proportion of jobs in a database of occupational histories (31,673 jobs) for which the CAPS-Canada crosswalk assistant allowed linkage between pairs of occupation classification systems

			Proportion of jobs linkable with	Proportion of jobs linkable with
	Given	Target	the full crosswalk assistant	only "official" links
	classification	classification	< 10 codes	direct link to	< 10 codes	direct link to
			proposed α	one job β	proposed α	one code β
	CCDO-1971	ISCO-1968	85.1%	72.7%	15.1%	1.0%
	CCDO-1971	NOC-2011	99.0%	81.0%	93.9%	13.2%

Table 2 :

 2 Inter-rater reliability in occupation coding between the two coders for each classification

			used
			Number of	Inter-rater reliability
		Resolution level
			corresponding codes at	in job coding [95% CI] -
		(number of Digit)
			the resolution level	All approaches combined
		7	7,907	35.7% [32.7%-38.7%]
	CCDO-1971	4	500	53.5% [50.4%-56.6%]
		3	81	66.5% [63.5%-69.5%]
		5	1,504	43.2% [40.1% -46.3%]
	ISCO-1968
		3	282	55.9% [52.8% -59.0%]
		6	840	48.1% [45.0% -51.2%]
	US-SOC-2010	5	461	53.9% [50.8% -57.0%]
		3	97	62.9% [59.9% -65.9%]
		4	500	53.2% [50.1% -56.3%]
	NOC-2011
		3	140	58.8% [55.7% -61.9%]

  The category of links was ranked from the most to the less informative. When a link is available for a starting code, links at the lower levels were not sought

	US-SOC-2010	ISCO-1968	97.3%	0%	94.3%	3.0%	35.8%	0%	34.0%	1.8%
	US-SOC-2010	NOC-2011	98.9%	0%	95.4%	3.5%	73.8%	0%	68.0%	5.8%

α β The sum can vary due to rounding

Table S1 :

 S1 Proportion of jobs in a database of occupational histories (31,673 jobs) for which the CAPS-Canada crosswalk assistant allowed linkage

		between pairs of industry classification systems
			Proportion of jobs linkable
					Proportion of jobs linkable
			with the full crosswalk
	Given	Target			with only "official" links
			assistant
	classification	classification
			< 10 codes	direct link
					< 10 jobs	1 job
			proposed α	to one job β
	ISIC-1971	NAICS-2012	78.7%	17.3%	18.2%	2.3%
	ISIC-1971	SIC-1980	71.9%	21.0%	19.3%	1.6%
	NAICS-2012	SIC-1980	100.0%	74.4%	94.4%	47.7%
	NAICS-2012	ISIC-1971	99.4%	90.4%	87.2%	35.5%
	SIC-1980	ISIC-1971	99.0%	90.9%	77.8%	32.6%
	SIC-1980	NAICS-2012	99.6%	80.0%	93.0%	40.9%

Table S2 :

 S2 Inter-rater reliability in industry coding between the two coders for each classification used

		Number of	Inter-rater reliability in
	Resolution level
		corresponding codes	job coding [95% CI] -All
	(number of Digit)
		at the resolution level	approaches combined
	4	159	78.5% [75.9% -81.1%]
	ISIC (1971)
	3	71	81.1% [78.6% -83.6%]
	4	860	64.1% [61.1% -67.1%]
	SIC (1980)
	3	318	71.3% [68.4% -74.2%]
	6	922	63.2% [60.2% -66.2%]
	5	711	67.3% [64.3% -70.3%]
	SCIAN (2012)
	4	323	71.5% [68.7% -7 4.3%]
	3	102	78.2% [75.6% -80.8%]

Table S3 :

 S3 Correspondances between industry classifications based on "official" crosswalks

Table S4 :

 S4 Relative contribution of the different types of link covered by the crosswalk assistant The category of links was ranked from the most to the less informative. When a link is available for a starting code, links at the lower levels were not sought

			Proportion of jobs in a database of occupational histories (31,673 jobs) for which crosswalks provide
						usefull information
	Origin	Target
				≤ 10 jobs proposed			Direct link to 1 code
	system	system
			All links	By type of link (exclusively) α	All links	By type of link (exclusively) α
			combined β	"Expert" "Empirical" "Official"	combined β	"Expert" "Empirical" "Official"
	ISIC-1971	NAICS-2012	78.7%	0%	78.5%	0.1%	17.3%	0%	17.2%	0.0%
	ISIC-1971	SIC-1980	71.9%	0%	71.8%	0.1%	21.0%	0%	21.0%	0.0%
	NAICS-2012	SIC-1980	100.0%	0%	95.3%	4.7%	74.4%	0%	71.3%	3.1%
	NAICS-2012	ISIC-1971	99.4%	0%	95.3%	4.1%	90.4%	0%	87.1%	3.4%
	SIC-1980	ISIC-1971	99.0%	2.4%	94.0%	2.6%	90.9%	2.4%	86.8%	1.8%
	SIC-1980	NAICS-2012	99.6%	9.6%	86.8%	3.2%	80.0%	9.6%	68.5%	1.9%

α β The sum can vary due to rounding

Annals of Occupational Hygiene

Acknowledgement

The authors thank Loïc Garras, Carine Prévot and Julien Lemonnier for their contribution in the development of the CAPS coding tool. The authors are grateful to Dora Rodriguez (DR) and Elmira Aliyeva (EA), the two coders in charge of the coding task, and Ana Gueorguieva for her contribution in the development of the crosswalks.

Conclusion: CAPS-Canada is an attractive alternative to manual coding and is particularly relevant for coding a job into multiple classifications or for recoding jobs into other classifications.

Keywords: agreement, computer-coding assistant, job-exposure matrix CP, GB and JL developed the coding assistant. JL, LR and JS were involved at different stages of the creation of the crosswalk assistant. TR performed the analyses and drafted the manuscript. All coauthors participated to the writing and approved the final manuscript.

Support

The development of the coding assistant was funded by the French National Institute for Health


Mohammad Mohammadamini

Driss Matrouf

Jean-Francois Bonastre

Romain Serizel

Sandipana Dowerah

Denis Jouvet

Compensate multiple distortions for speaker recognition systems

Keywords: additive noise, early reverberation, full reverberation, x-vector, denoising autoencoder

The performance of speaker recognition systems reduces dramatically in severe conditions in the presence of additive noise and/or reverberation. In some cases, there is only one kind of domain mismatch like additive noise or reverberation, but in many cases, there are more than one distortion. Finding a solution for domain adaptation in the presence of different distortions is a challenge. In this paper we investigate the situation in which there is none, one or more of the following distortions: early reverberation, full reverberation, additive noise. We propose two configurations to compensate for these distortions. In the first one a specific denoising autoencoder is used for each distortion. In the second configuration, a denoising autoencoder is used to compensate for all of these distortions simultaneously. Our experiments show that, in the co-existence of noise and reverberation, the second configuration gives better results. For example, with the second configuration we obtained 76.6% relative improvement of EER for utterances longer than 12 seconds. For other situations in the presence of only one distortion, the second configuration gives almost the same results achieved by using a specific model for each distortion.

Introduction

In recent years, deep neural networks became the most commonly used approach for speaker modeling [START_REF] Snyder | X-Vectors: Robust DNN Embeddings for Speaker Recognition[END_REF][START_REF] Arsha Nagrani | Voxceleb: Large-scale speaker verification in the wild[END_REF]. Although DNN-based speaker recognition systems outperform their previous statistical systems such as i-vector, still their performance degrades in the presence of acoustic distortions, such as reverberation and additive noise. Using a huge amount of data makes the DNN based systems robust against noise and reverberation but in severe conditions the performance can degrade drastically [START_REF] Novotny | On the use of X-vectors for Robust Speaker Recognition[END_REF]. Therefore, explicit denoising and dereverberation can make these systems more robust [START_REF] Mohammadamini | Data augmentation versus noise compensation for x-vector speaker recognition systems in noisy environments[END_REF][START_REF] Rozenberg | Siamese x-vector reconstruction for domain adapted speaker recognition[END_REF].

Approaches based on speech signal enhancement [START_REF] Novotný | Analysis of DNN Speech Signal Enhancement for Robust Speaker Recognition[END_REF] and denoising techniques at the speaker modeling level are often proposed to reduce the impact of noise and reverberation on speaker recognition systems [START_REF] Mohammadamini | Denoising x-vectors for Robust Speaker Recognition[END_REF].

In the previous work [START_REF] Mohammadamini | Data augmentation versus noise compensation for x-vector speaker recognition systems in noisy environments[END_REF][START_REF] Mohammadamini | Denoising x-vectors for Robust Speaker Recognition[END_REF] we proposed several Denoising Autoencoders (DAE) to reduce the effect of additive noise in x-vector space. The proposed DAEs estimate the clean x-vector from the noisy version. In this paper we want to extend the application of proposed denoising techniques to other distortions like early and full reverberation. Another contribution of our work is proposing two configurations for five environments, where there is one or more of these distortions: additive noise, early reverberation, full reverberation. Using data augmentation with noise and reverberation in training the speaker embedding network and denoising techniques in signal level, feature level and speaker modeling level are common approaches to make the system robust against noise and reverberation. It is important to specify that both strategies are important and none of them could be replaced by another one. Indeed, the proposed DAE tries to learn the relation between the x-vector affected by a given distortion and its clean version. Thus, it is a direct learning of the distortion effect, which would make the denoising system more efficient than multi-condition training [START_REF] Mohammadamini | Data augmentation versus noise compensation for x-vector speaker recognition systems in noisy environments[END_REF]. The DAE uses more specific information about the distortion than what would be used in a data augmentation approach. Moreover, the DAE we propose is trained in x-vector space, which makes its training very quick, with respect to the x-vector extractor training.

In this article the additive noise and reverberation are simulated. The artificial reverberation is applied using image model [START_REF] Scheibler | Pyroomacoustics: A Python Package for Audio Room Simulation and Array Processing Algorithms[END_REF]. Obviously, the use of test data from a real environment is preferable. Unfortunately, we do not have such a database at the moment. However, the simulation model we used has proven that is effective in generating real data [START_REF] Wang | Distant-talking speaker identification using a reverberation model with various artificial room impulse responses[END_REF].

The paper is organized as follows. Section II reviews related works. Section III describes the two proposed configurations (specific and general DAE) for handling different distortions. Section IV is devoted to experimental setup, and Section V presents and discusses the results.

II.

Related works

Despite significant advances, the performance achieved by speaker recognition systems degrades dramatically when they are used in real applications where channel mismatch as well as environmental additive noise and reverberation are present separately or simultaneously. Many efforts have been put into modeling and compensating these distortions throughout the years by working on different levels. Reducing the negative effect of noise and reverberation could be treated in signal level or higher-levels (i.e., speaker modeling level).

In signal level, several works have been done to compensate for additive noise and/or reverberation. In [START_REF] Zhao | Robust speaker identification in noisy and reverberant conditions[END_REF] a system proposed for conditions in the presence of additive noise and reverberation. In this work, firstly the additive noise is removed through a binary masking estimated by a neural network. For reverberation the speaker model trained in reverberant condition. Also, some works are done at signal level to make the x-vector framework robust against noise and reverberation. In [START_REF] Sefik Emre Eskimeza | Front-end speech enhancement for commercial speaker verification systems[END_REF] LSTM and CNN networks are used to denoise the speech signal. The LSTM network is trained to reconstruct the clean log magnitude spectrum. The CNN network is used to denoise the short-time Fourier transform (STFT) magnitude spectrogram. The proposed LSTM with 20% to 30% relative improvement of EER, has given better results in comparison to other methods used in this research. In [START_REF] Novotný | Analysis of DNN Speech Signal Enhancement for Robust Speaker Recognition[END_REF] a denoising autoencoder is used for joint compensation of additive noise and reverberation in the xvector framework. In this research, the DAE reconstructs the clean version of the magnitude spectrum. In the best case a 30% percent improvement of EER has been reported. In [START_REF] Taherian | Deep Learning Based Multi-Channel Speaker Recognition in Noisy and Reverberant Environments[END_REF] several masking-based beamformers used for denoising and dereverberation. The MVDR Rank1 beamformer gave the best results for the real RIRs, and the GEVBAN has given the best results with simulated RIRs. A DNN that supports acoustic beamforming and dereverberation is used in the frontend of x-vector framework [START_REF] Yang | Joint optimization of neural acoustic beamforming and dereverberation with x-vectors for robust speaker verification[END_REF]. The joint training of this network and x-vector network has improved the performance of speaker recognition with 40% percent improvement of EER for simulated RIRs and 25% improvement for real RIRs.

There are some works in speaker modeling level to compensate for a specific distortion. In [START_REF] Mohammadamini | Denoising x-vectors for Robust Speaker Recognition[END_REF], statistical i-map, and three derivations of denoising autoencoders presented to suppress the effect of additive noise in the x-vector domain.

In [START_REF] Mohammadamini | Data augmentation versus noise compensation for x-vector speaker recognition systems in noisy environments[END_REF] the importance of using denoising techniques alongside data augmentation was explored to make the xvector system robust against additive noise. The 66% relative improvement of EER in the x-vector domain shows that noise compensation in speaker modeling level is very effective for speaker recognition systems. Siamese x-vector reconstruction was introduced in [START_REF] Rozenberg | Siamese x-vector reconstruction for domain adapted speaker recognition[END_REF] to compensate for additive noise and sampling rate mismatch in the x-vector framework. It is shown that this technique is more effective to compensate sampling rate mismatch rather than additive noise. In the previous generation of speaker recognition systems (i.e., i-vector) the mapping from distorted to clean vectors is explored broadly. i-MAP [START_REF] Ben Kheder | Fast i-vector denoising using MAP estimation and a noise distributions database for robust speaker recognition[END_REF] and joint i-MAP [START_REF] Ben Kheder | Probabilistic Approach Using Joint Clean and Noisy i-Vectors Modeling for Speaker Recognition[END_REF] are two statistical techniques used for noise compensation in the i-vector framework.

However, there is no specific research to compare the state-of-the-art techniques in signal level with speaker modeling level in the same experimental setup, the results reported by compensation techniques in speaker modeling level from previous works are more promising and working in speaker modeling level is easier than signal level. Also, the reviewed works done in speaker modeling level focused on one distortion. In the current paper we extend this approach for situations where there are different distortions.

III. System configuration

Our study will focus on 5 conditions and their combinations: Clean (D), additive Noise (N), Early reverberation (E), Full reverberation (F) and additive noise with Full reverberation (FN).

The reverberation is the sum of sound reflections arrived at a single point inside an acoustical enclosure. Early reflections which are called early reverberation arrive between 50-100ms after the arrival of the direct signal. The full reverberation is the next echoes that arrives to listener with longer delays [START_REF] Hu | Effects of early and late reflections on intelligibility of reverberated speech by cochlear implant listeners[END_REF].

In the next sections, the DAE(N), DAE (E), DAE (F), DAE (FN), DAE (N+E+F+FN) stands for experiments that the input of denoising autoencoder is noisy x-vector data, early reverberated x-vector data, the full reverberated xvector data, the simultaneously noisy fully reverberated xvector data and finally a combination of x-vectors for all distortions. The output of the DAE is always the clean xvector data.

In this paper we compare two approaches in handling multi-acoustic distortions. In the first approach, after the xvector network, a specific denoising autoencoder is used for each distortion. For clean x-vectors the scoring is done without passing them through a DAE. The details of this configuration are depicted in Figure 1. In this configuration, we assume that the type of distortion is known. We will show when the type of distortion is unknown, using a classifier can help to detect the kind of distortion automatically.

In the second configuration depicted in Figure 2, the compensation for different distortions is done by using one single DAE. In this configuration like the specific compensation, there is clean speech, distorted speech with additive noise, early reverberation, full reverberation, additive noise and full reverberation. With this system, it is not necessary to have previous information about the kind of distortion. As it is shown in results, the denoising autoencoder learns to compensate all those distortions simultaneously. In the case of clean speech, we show that, without any change in the system and without having previous information about the environment, the denoising part does not have a negative effect on clean x-vectors.

IV.

Experimental setup

The x-vector network introduced in [START_REF] Snyder | X-Vectors: Robust DNN Embeddings for Speaker Recognition[END_REF] was used to create x-vectors for train and test data. The x-vector network has been trained with Voxceleb1, Voxceleb2 [START_REF] Arsha Nagrani | Voxceleb: Large-scale speaker verification in the wild[END_REF] and one million utterances From Voxceleb 1 and Voxceleb 2 augmented with different parts of Musan corpus (Noise, Babble, Speech) and real RIRs [START_REF] Snyder | MUSAN: A Music, Speech, and Noise Corpus[END_REF]. The x-vector extractor has been used to create clean, noisy and reverberant data to train DAE. The data simulation procedure is described in 3.1.

For each experiment firstly the reverberation and/or noise were added to Voxceleb2. Then pairs of distorted/clean x-vector were produced to train DAE. In the same manner, the data distorted for the test. In all cases the enrollment data is clean; because we assume that in real applications it is possible to have clean data for enrollment.

In the experiments, we use deep stacked DAE introduced in [START_REF] Novotny | On the use of X-vectors for Robust Speaker Recognition[END_REF]. The architecture (number of layers and neurons) is the same in all experiments. In the backend the cosine metric is used for scoring.

The FABIOL corpus was used for test and enrollment. In FABIOL corpus there are 6992 files from 130 speakers. Form 130 speakers in FABIOL, for 100 speakers there is a small number of utterances. We used these speakers just in enrollment. The utterances belonging to the remaining 30 speakers are randomly separated for the test and enrollment. In enrollment there are 3576 files and 3244 files are used for the test. Because the duration of files in FABIOL corpus spans from very short to long, the test files are separated into seven groups for each two seconds.

V.

Results and discussions

The results obtained in the presence of each distortion are summarized in Table 1. When we are in a clean environment without noise or reverberation, we show that the use of DAE(F+N) gives almost the same (sometimes better) results as the baseline system. This is an interesting property of the proposed approach. If the x-vector belongs to the clean class, the scoring could be done directly. But for general configuration presented in Fig 2 , we don't care about the cleanness of the environment and even in the case of noise free and non-reverberant environments the test xvector will be passed through the DAE. When we apply the DAE trained on noisy x-vectors on clean x-vectors, the output is still almost identical to the input. It means that for noise and reverberation free environments the system could be used without any modification.

In the case of additive noise, both specific models and general models were tested. As it is shown the results obtained by specific models are a little better. For example, for utterances longer than 12 seconds, the EER obtained by specific model is 1.91 but the EER obtained by general model is 2.23. For early reverberation and full reverberation distortions, the results obtained by general model for short segments are better than the specific model but the results obtained for longer utterances is almost the same. When there is additive noise and reverberation, in all cases the results achieved by the general model are better. For specific models the experiments are done directly without using a classifier. But to prove that it is possible to detect the type of distortion, we trained a feedforward neural network. The accuracy of the trained network is 81%. Even if we had a distortion classifier with 100% accuracy, the results show that it's better to use a general DAE instead of using a specific DAE for each distortion.

VI. Conclusion

In this paper we proposed two configurations for robust speaker recognition in the environments where there are several distortions. The systems act efficiently in environments with early reverberation, full reverberation, additive noise, additive noise and reverberation. To solve this problem, we proposed two configurations. In the first configuration we used a specific DAE for each distortion. In the second configuration, one DAE is used to learn all of these distortions simultaneously. The second configuration is simpler and gives the same or even better results than specific compensation for each distortion. We also showed that the speaker recognition performance doesn't decrease (with respect to the baseline) when the test data is clean, which is a nice property of the proposed denoiser.

Figure 1 .Figure. 2

 12 Figure 1. Using specific models for each distortion

Table 1 .

 1 The results obtained for different distortions (EER)

	Environment	Duration (seconds)	[0,2]	[2,4]	[4,6]	[6,8]	[8,10]	[10,12] [12,]
	Dry signal (D)	D	9.89	5.08	3.62	2.44	1.77	1.85	1.65
		DAE (FN)	9.01	5.06	3.31	2.40	1.75	1.85	1.65
		N	14.58	11.16	8.83	9.12	8.05	7.97	6.63
	Additive noise (N)	DAE (N)	10.48	6.08	3.56	3.27	2.26	1.91	1.91
		DAE (N+E+F+FN)	10.75	6.93	4.45	4.47	3.53	2.77	2.23
		E	18.60	10.54	7.08	4.58	3.98	4.16	4.02
	Early reverberation (E)	DAE (E)	13.08	6.48	4.15	2.50	2.23	2.31	2.29
		DAE (N+E+F+FN)	11.57	5.67	3.85	2.96	2.58	2.77	2.29
		F	20.01	11.96	8.01	6.15	4.88	5.09	5.23
		DAE (F)	9.05	5.03	3.26	2.48	2.21	2.31	2.23
	Full reverberation (F)
		DAE (N+E+F+FN)	9.29	4.86	3.62	2.49	2.21	2.31	2.05
		FN	27.32	24.34	20.77	18.68	19.38	19.93	17.61
	Additive noise and Full	DAE (FN)	14.24	9.93	7.37	4.97	3.96	3.68	4.53
	reverberation (FN)
		DAE (N+E+F+FN)	13.66	9.73	6.52	4.54	3.61	3.24	4.09

Acknowledgement

This work has been supported by ROBOVOX ANR project.