Patrick Saint-Dizier
email: stdizier@irit.fr

LELIO: An Auto-Adaptative System to Acquire Domain Lexical Knowledge in Technical Texts

Keywords: Lexical Acquisition, Constrained Natural Language, Technical Text Authoring

In this paper, we investigate some language acquisition facets of an auto-adaptative system that can automatically acquire most of the relevant lexical knowledge and authoring practices for an application in a given domain. This is the LELIO project: producing customized LELIE solutions. Our goal, within the framework of LELIE (a system that tags language uses that do not follow the Constrained Natural Language principles), is to automate the long, costly and error prone lexical customization of LELIE to a given application domain. Technical texts being relatively restricted in terms of syntax and lexicon, results obtained show that this approach is feasible and relatively reliable. By auto-adaptative, we mean that the system learns from a sample of the application corpus the various lexical terms and uses crucial for LELIE to work properly (e.g. verb uses, fuzzy terms, business terms, stylistic patterns). A technical writer validation method is developed at each step of the acquisition.

Aims and Challenges

Technical documents form a linguistic genre with specific linguistic constraints in terms of lexical realization, syntax, typography and overall document organization, including business or domain dependent aspects. Technical documents cover a large variety of types of documents: procedures, equipment and product manuals, various notices such as security notices, regulations of various types (security, management), requirements and specifications (Hull et ali. 2011). These documents are designed to be easy to read and as efficient and unambiguous as possible for their users. They must leave little space for personal interpretations. For that purpose, they tend to follow relatively strict controlled natural language (CNL hereafter) principles concerning both their form and contents [START_REF] Fuchs | First-Order Reasoning for Attempto Controlled English[END_REF], [START_REF] Kuhn | A Survey and Classification of Controlled Natural Languages[END_REF] (Aurora et ali. 2013). These principles are described in documents called authoring guidelines. These are general purpose statements, norms in e.g. aeronautics, chemistry, or considerations proper to a company. Guidelines contain some details about e.g. the terms which can be used (for example, a limited subset of verbs) or not and syntactic structures which are allowed or preferred. Stylistic considerations may also be introduced, for example concerning the order of complements or subordinate clauses (e.g. condition -main clause -purpose). These considerations have motivated the development of the LELIE project [START_REF] Barcellini | Risk Analysis and Prevention: LELIE, a Tool dedicated to Procedure and Requirement Authoring[END_REF], [START_REF] Saint-Dizier | Challenges of Discourse Processing: the case of technical documents[END_REF], which is a system that detects several types of errors in technical documents, whatever their authoring and revision stages are. Lelie produces alerts related to these errors on terms, expressions or constructions that need various forms of improvements, at all levels: lexical, syntactic, semantics and discourse. LELIE deals with errors which are not detected by usual text editors such as MS Word. These errors may indeed be perfectly acceptable expressions in non-technical language. For each application domain, and for each company, LELIE must be customized. The major part of the customization is the development of the lexical resources and their features, while the error detection rules remain relatively stable over domains. Acquiring and structuring the lexical knowledge of an application domain is a very much costly and time consuming activity, and an error prone process. This is a major limitation to an accurate and efficient customization of applications. The lexicon proper to an application is about 60% of the total lexicon. In this paper, we investigate, within the framework of LELIE, the different facets of an auto-adaptative system that can acquire most of the relevant lexical knowledge for an application in a given domain. This is the LELIO project: producing customized LELIE solutions to an application domain or in accordance with a company authoring practices. Technical texts being relatively restricted in terms of syntax and lexicon, results obtained show that this approach is feasible and relatively reliable, with some form of human validation. By auto-adaptative, we mean that the system:

(1) learns from a sample of the application corpus the various lexical terms and uses crucial in LELIE (e.g. verb uses, fuzzy terms, business terms) and (2) automatically generates the application oriented lexicon, which is paired with the domain independent lexicon, defined in the kernel of LELIE. This topic is very vast and has been investigated at the level of the re-usability of resources. This notion of autoadaptative system has been developed in dialogs, machine translation [START_REF] Scherrer | Adaptive String Distance Measures for Bilingual Dialect Lexicon Induction[END_REF] and to a lesser extend in lexical tuning in opinion analysis. To the best of our knowledge, nothing has been done in the type of context we consider. Foundational ergonomics principles are investigated in [START_REF] Ganier | Considering users and the way they use procedural texts : some prerequisites for the design of appropriate documents[END_REF], and in [START_REF] Weiss | Writing remedies. Practical exercises for technical writing[END_REF](White et ali. 2009) from a more linguistic and pedagogical perspective. In this paper, we address this variable part of the resources which must be acquired. We concentrate in this paper on two main cases: the acquisition and the character-ization of fuzzy terms in an application and the acquisition of business terms. We develop in section 5 a style facet of technical authoring by an induction mechanism of the main stylistic practices which can then be used as a guideline or a norm. The techniques developed in this work are based on lexical and syntactic considerations. These techniques are simple but seem to produce relatively satisfactory results.

The LELIE Prototype

The LELIE prototype (Version V2.1, code freely available CC BY NC) detects errors related to the non-observation of a recommendation in the authoring guidelines that is considered in various types of technical texts: procedures, specifications or requirements (Hull et ali.,. 2011), [START_REF] Grady | System Requirements Analysis[END_REF]. For example, according to most CNL principles and guidelines, the use of passives, modals or negation must be avoided in instructions [START_REF] Kuhn | A Survey and Classification of Controlled Natural Languages[END_REF]. Specific errors have also been observed for non-native speakers of the language the use [START_REF] Garnier | Correcting errors in N+N structures in the production of French users of English[END_REF]. LELIE also allows to specify business constraints such as controls on style and the use of business terms. The errors detected by LELIE are typical errors of technical texts, they are not errors in ordinary language. Error detection in LELIE depends both on the textual genre and on the precise discourse structure that is observed: for example modals are the norm in requirements [START_REF] Grady | System Requirements Analysis[END_REF] but not in instructions. Titles may contain deverbals which are not frequently admitted in instructions or warnings [START_REF] Grady | System Requirements Analysis[END_REF]. LELIE is parameterized and offers several levels of alerts depending on the a priori error severity level. LELIE and the experiments reported below have been developed on the logic-based <TextCoop> platform [START_REF] Saint-Dizier | DISLOG: A logic-based language for processing discourse structures[END_REF], 2014). The output of LELIE is the original text with annotations. These results show that there is on average about one error every 4 lines of text, not counting errors related to business term. This error rate is very high. The alerts produced by the LELIE system have been found useful by most technical writers that tested the system. In this table, we can see that the main resources which are domain dependent in LELIE are: verbs (and deverbals), fuzzy terms (of various categories) and business terms (e.g. for sentence complexity measures).

The development and test corpora

The modeling of the lexical knowledge acquisition proper to a domain has been realized from a number of corpora from various technical domains, authoring guidelines and genres (procedures, specifications, business rules, requirements, etc.). In a first stage, our corpora are in English, however, a similar task is planned for French. Most of the observations made for English will probably remain identical for other European languages. However, some analysis elements may differ due to the specific syntactic and morphological features. The corpora we considered are the following: In total, our corpus has 290 pages with about 105 000 words. Aeronautics and computer science use relatively strict authoring guidelines, while medicine, pharmacy and management have more shallow guidelines and a large diversity of word uses are observed. These two areas are of much interest for the discourse and style analysis reported in section 5.

Acquisition of fuzzy terms

A first experimentation of the principles developed above is devoted to the case of fuzzy lexical items which is a major type of error, very representative of the need of lexical knowledge acquisition. Roughly, a lexical item is fuzzy if it denotes a concept whose meaning, interpretation, or boundaries can vary considerably according to contexts, readers or conditions, instead of being fixed once and for all. Fuzzy lexical items include several categories of adverbs (manner, temporal, location, and modal adverbs), adjectives (adapted, appropriate) determiners (some, a few), prepositions (near, around), a few verbs (minimize, increase) and nouns. These categories are not homogeneous in terms of fuzziness, e.g. fuzzy determiners and fuzzy prepositions are always fuzzy whereas e.g. fuzzy adverbs may be fuzzy only in certain contexts. The degree of fuzziness is also quite different from one term to another in a category.

The context in which a fuzzy lexical item is uttered may have an influence on its severity level. For example 'progressively' used in a short action (progressively close the water pipe) or used in an action that has a substantial length (progressively heat the probe till 300 degrees Celsius are reached) has two different severity levels because the application of 'progressively' may be more difficult to realize in the second case. In the case of this adverb, it is not the manner but the underlying temporal dimension that is fuzzy. Finally, some usages of fuzzy lexical items are allowed. This is the case of business terms that contain fuzzy lexical items which should not trigger any error. For example, low visibility landing procedure in aeronautics corresponds to a precise notion, therefore 'low' must not trigger an alert in this case. The equivalent, non-business expression landing procedure with low visibility should probably originate an alert on 'low', but there is no consensus among technical writers. A business term is a kind of black box, but the difficulty, as shall be seen in section 5, is to precisely define what a business terms is.

A fuzzy lexical item must be contrasted with an underspecified term or expression. For example, a verb such as damaged in the mother card risks to be damaged is not fuzzy but underspecified because the importance and the nature of the damage is unknown; similarly for heat the probe to reach 500 degrees because the means to heat the probe are not given but are in fact required to realize the action.

In LELIE, a relatively large resource of lexical items which can be fuzzy in technical documents has been defined with an a priori severity level from 1 to 3. This resource has been elaborated from various corpora and company resources. Table 4 shows that a domain includes on average less than 100 fuzzy lexical items from our predefined lexicon, among which about 80 items from open classes. This is a relatively small number of terms. The challenge is (1) the acquisition of the fuzzy lexical items not present in this lexicon, which represent about 40% of these items and (2) the identification of those contexts where such an item is really fuzzy (Saint-Dizier 2015). Given these resources, our lexical acquisition system scans a given domain corpus. This corpus is composed of about 200 to 400 pages of technical documents produced in the domain considered for the lexical acquisition, this domain may be restricted to a company or a group of technical writers. The system proceeds by lexical category, as described in the following subsections.

Fuzzy determiners and prepositions

The auto-adaptative system we have defined proceeds as follows concerning closed classes (determiners, prepositions). The system first constructs the set of the items found in the domain corpora and identified as fuzzy lexical items in our fuzzy lexical items resource. These items a priori keep their fuzzy character in the application lexicon, a confirmation of their fuzzy character can be asked to e.g. the application administrator (a senior technical writer in general) together with a confirmation of their severity level. The severity level can be used to filter out items which are moderately fuzzy in the domain. Fuzzy determiners include simple terms and compounds e.g.:most, a majority of, almost all, etc. Fuzzy prepositions and preposition compounds include e.g. near, above, about, below, rather close to, etc.

Fuzzy adverbs

The system then identifies in the domain corpora adverbs, verbs and adjectives, on the basis of a POS tagger. These three categories are identified simultaneously because they are used in syntactic analysis: adverbs are verb or adjective modifiers. Adverbs judged to be fuzzy in our database are in general fuzzy in most contexts and domains. The only variation is the severity level of each context, as illustrated at the beginning of this section, this is developed in [START_REF] Saint-Dizier | Features of an Error Correction Memory to Enhance Technical Texts Authoring in LELIE[END_REF].

Our database contains various types of adjectives, among which: -temporal adverbs, e.g. momentarily, shortly, -manner adverbs, e.g. rapidly, progressively, -quantity adverbs, e.g. sufficiently, enough .

Let us consider the case of adverbs. Most adverbs end by -ly in English. A list of adverbs can then be constructed from the domain corpus being investigated. The adverbs already present in the database of fuzzy lexical items are kept and tagged as fuzzy, and for those which are not in the database (about 55%), the system has to determine if they may have fuzzy uses. Potential fuzzy uses are identified via grammatical induction. The system searches if these adverbs are used in similar contexts than one or more elements of the database in both the domain and our development corpora to guarantee a certain coverage. By context, we have explored at the moment two situations: Adverb + Verb and Adverb + Adjective. Beyond a threshold of 70%, of identical contexts, the adverb is probably fuzzy (e.g. carefully exists in our lexicon while slowly and cautiously do not and are found with approximately the same usages). This threshold has been defined and tuned gradually from corpus observations. It may slightly differ from one domain to another, in particular it may depend on the number of verbs and adjectives used in a domain. These adverbs are inserted into the domain lexicon and tagged as fuzzy a priori or via a confirmation from a technical writer. At this stage the initial fuzzy lexical items database is stable and does not include this new terms. However, these new terms are kept in a 'secondary'

database for further analysis in other domains.

In a second stage, this lexicon of domain fuzzy terms can be enhanced. If these adverbs have an equivalent adjective (e.g. careful) or noun (care, caution), then these terms are also integrated into the fuzzy lexicon. Finally, recurrent expressions, recognized by a simple pattern, e.g.:

[prep, adj(fuzzy), noun],

[prep, intensifier, noun(fuzzy)], containing these fuzzy lexical items are also considered as fuzzy expressions, e.g. with careful precaution, with great caution. These are included into the fuzzy term lexicon as additional fuzzy expressions. At the moment, 14 such patterns have been implemented. This approach is relatively simple, but sufficiently accurate given the language 'profile' of technical texts. An evaluation on a 36 pages test corpus (aeronautics), was carried out, where LELIE tags all expressions it found fuzzy, with their severity level. The result was compared with the same texts manually annotated by three trained technical writers. Manually annotating fuzzy lexical items is a difficult task because annotators often have difficulties to identify all these items unless they read the text several times. A Cohen Kappa test indicates an agreement level of 83% between the three annotators, which belong to the same company: this shows that the task is not so straightforward. In particular, some terms are judged to have a low level of fuzziness may not be tagged. Then, after discussion, a single annotated document was produced with 197 errors found. Our indicative evaluation is based on this resulting document. LELIE tagged 202 fuzzy terms, leading to a precision of 94%, and a recall of 91%. This performance level is acceptable for technical writers which do not want to be bothered too often by inappropriate error messages. A rate of less than 8 to 10% for inappropriate errors is acceptable for them, considering that their level of disagreement on such a task is around 10 to 12%, e.g. [START_REF] Schriver | Evaluating text quality : The continuum from text-focused to reader-focused methods[END_REF].

Fuzzy adjectives

Adjectives are not very frequent in technical documents, an analysis carried out on our corpora indicates a total of 280 adjectives, out of which about 70 are used more than 10 times. This does not include adjectives found in business terms which must not be considered as fuzzy since business terms are considered as closed terms. The main categories of these frequently used adjectives are:

-evaluative or judgment adjectives (e.g. adequate, common, flexible, standard, useful, optimal, typical) which are used for example to qualify a process or an equipment, adjectives such as useless, irrelevant are not judged to be fuzzy since they express a boolean fact, -scalar adjectives in particular related to measures (e.g. hot, cold, long, short, heavy, etc.) with their corresponding antonyms, -temporal adjectives (e.g. recurrent, frequent), and -modal adjectives, e.g. crucial, essential, necessary.

The other types of adjectives found may be vague such as color adjectives but they are not judged to be really fuzzy. These fuzzy adjectives may be modified by an intensifier (very, somewhat) which reinforces their fuzzy character. Finally, adjectives such as damaged are not fuzzy in our approach but underspecified since the type of damage is not explicit. When considering adjective acquisition (identified via a POS tagger) and analysis from new texts, it turns out that about 78% of these adjectives are already present in our database. The 22% are often very specialized adjectives, whose interpretation is domain dependent. These adjectives could be considered as one-word business terms. To detect whether these adjectives have a fuzzy character, a simple method consists in looking for patterns where these adjectives are combined with specific terms that reveal their fuzzy character. The number of new fuzzy adjectives is relatively significant (about 1/3 of the total). In terms of human analysis, 8 among these 11 terms have been judged to be really fuzzy. The 3 others are moderately fuzzy and the expressions in which they occur can be understood. The 7 adjectives judged not to be fuzzy are indeed not fuzzy.

Fuzzy verbs and deverbals

Verbs and deverbals form a single class in technical documentation: when the agent is committed, deverbals are often used instead of passive forms which are not allowed in most CNL guidelines. For example, given: the tester must define five use cases, to omit the agent (assumed to be implicit), a passive form such as: five used cases must be defined, but since passives are not recommended in general, a deverbal can be used: definition of five use cases. even if the modal is not realized, the injunctive character of the statement is still present. Our lexical database contains verbs which are anchored to their various deverbal forms. These are treated in a similar way in what concerns their fuzzy character. Constrained natural language principles recommend to use a restricted set of verbs in a domain. Areas such as transportation and aeronautics have defined lists of recommended verbs and their uses. The set of allowed verbs is lower then 100 in most cases. However, we found areas which do not follow these recommendations where the number of verbs can go up to 600 or even 700 different verbs, not including morphological variations. We then observe many redundancies and fuzzy or vague uses. However, even with a limited number of verbs, fuzzy uses are observed, which are due to the arguments the verb is combined with. For example, clean the screen is judged fuzzy because it is not clear which windows on the screen must be closed. Although these verbs are not totally excluded from technical texts, authors are invited to use more precise verbs whenever possible. In our approach the verbs of these classes are a priori fuzzy: the technical writer is invited in the texts he produces to avoid them. The restricted set of verbs which are recommended and follow CNL principles are a priori not fuzzy. However, they may become fuzzy in combination with some arguments as illustrated above.

To detect these fuzzy uses, we developed a learning mechanism that observes technical authors when they make corrections and induces fuzzy uses (Saint-Dizier 2015).

Finally, for the verbs found in a domain which are neither general purpose nor the recommended ones (when such a set exists), our approach is twofold. First they are tagged as not belonging to the recommended set when it is defined, but this does not necessarily means that these verbs must absolutely be avoided. Next, these verbs are tagged as fuzzy when they are used in several distinct contexts in the domain corpus. Context is characterized by verb arguments, in particular objects, which can be standard language or business terms. The difficulty is to evaluate that two contexts are distinct, not simply closely related terms.

For that purpose, a domain terminology (or ontology) is necessary, on which a distance metrics is applied. Then, a verb is fuzzy if one of these conditions is met:

-the objects of the verb are general purpose terms in the ontology (characterized by a distance of N arcs to reach the leaf node, to be adjusted depending on the domain ontology complexity, which may have a hierarchy of up to 9 nodes), -the objects of the verb may be low level words in the terminology, but they belong to at least 3 different subtrees, where the common ancestor is at least 3 nodes up in the hierarchy, -the objects of the verb include more than a threshold T of different business terms. T depends on the complexity and the number of the business terms. A value around 10 seems to be a good indicator.

Similarly to adverbs and adjectives, we asked a group of three technical writers to annotate the verbs they think are fuzzy in their context in a 36 page long document, including general purpose verbs. 76 occurrences of fuzzy verbs were found in total with a kappa of 83% which is relatively low.

Our system detected 82 fuzzy verbs, the resulting accuracy is 85%. Given the kappa, this result sounds satisfactory. Indeed, even if our system erroneously tags verbs as fuzzy, this may be useful to help technical writers to improve their writing skills and concentrate as mush as possible on the use of verbs which are unambiguous, not polysemous and probably with a low degree of fuzzyness.

Acquisition of Business Terms

Automatically acquiring business terms is a major challenge. This is an important task for several reasons in LELIE, e.g. ( 1) fuzzy terms in business terms must not generate any error message, (2) business terms count for a single unit when the length of a sentence is evaluated, (3) they also enter with a specific weight when the complexity of a sentence is measured. Besides their use in LELIE, it is often crucial for a company or a particular activity to have the list of the business terms which are used, to structure it and to possibly eliminate redundancies or useless uses. The first difficulty is to define what a business term is and what is its structure. For example: flap retraction speeds is probably a business term in aeronautics, but are lexical variations such as: minimum flap retraction speeds, maximum flap 5' retraction speed or 25 degrees flap retraction speed business terms ? If so, this means that quite a large level of lexical variation and syntactic composition may occur within business terms. Finally, is a single word with a specific meaning and uses a business term or just a polysemous word that needs contextual interpretation ? On the same corpus as for fuzzy lexical items, we carried out an additional experiment, with the same annotators, asking them to identify business terms. On 36 pages of text, 1152 business term occurrences have been found, with 291 different terms, union of all the occurrences found by the annotators. The kappa test indicates a level of agreement of only 81% among the three annotators. Again, this shows that this task is difficult. Roughly, the auto-adaptative system uses the following criteria to identify business terms:

1. Detect structures from 3 to 6 terms not containing any closed class term (modals, auxiliaries, determiners, conjunctions, prepositions) and no inflected verb: these are potential candidates for business terms. 6 terms seems to be the largest size for a business term.

All terms, even with some overlap, are kept.

2. Search in the corpora for variations, as illustrated above, and evaluate the degree of lexical variation at any position in the business term considered. Lexical variation is characterized by two types of patterns:

• general purpose patterns, such as the inclusion of an adjective in the pattern: flap retraction speeds → flap retraction maximum speeds. These patterns are implemented in Dislog, which runs on our TextCoop development environment (free resource). Five general purpose patterns have been defined, a single one manages the different positions an adjective may have in a business term.

• specific patterns introducing e.g. numbers, units, adjectives or additional nouns at specific places. These patterns are induced and generalized from the observation of already existing sets of business terms and their variations. For example: flap retraction speeds → 5 degree flap retraction speed. These patterns are much more ad'hoc and need to be developed with care. So far, we have developed 27 such pattern, with relatively high lexical category, e.g.:

[noun] → [number unit noun].

3. evaluate the level of variation between pairs of terms.

If it is too high (according to our evaluation, which still needs to be adjusted: above 6 different forms for a 3 word term, above 8 forms for a 4 word term), then it is probably not a business term, to be confirmed by the administrator. This is a kind of measure of the variability level of an expression.

4. Search in the corpora for expanded versions or these structures, i.e. if a business term is composed of the words A B C D, search via patterns for forms where these words occur in a different order with grammatical realizations, in particular using prepositions and determiners, e.g. for the above example: the minimum speeds for the retraction of flaps. Such an extended reformulation is indeed not allowed in the case of business terms, therefore such terms are eliminated.

5. the terms selected in (1) above that pass the various tests described in (2), ( 3) and ( 4) above are a priori business terms in the domain considered.

Considering the low level of agreement between annotators, it is difficult to produce a very accurate evaluation of the results we obtained. To have a more independent evaluation, the domain we considered (energy) does not have any approved terminology. The results can be summarized as follows:

• number of different business terms recognized by annotators (A): 291,

• number of different business terms recognized by LELIE (B) : 315,

• number of similar business terms (C): 269,

• number of different business terms: 76,

• agreement level computed via the formula: C / ((A+B)/2: 88%.

The result obtained by the auto-adaptative system is relatively good considering the complexity of the task. However, a larger evaluation is ongoing with a larger set of potential business terms, on two domains (aeronautics and bio-pharmacy). New decision criteria than the three ones developed above may probably emerge.

Induction of the most common Discourse Structures

This task has quite a different purpose in the autoadaptative landscape. It aims at automatically identifying the probably implicit authoring behaviors of technical writers, and to induce a set of best practices in terms of discourse organization. This is an important feature of our LELIO system. LELIE and TextCoop (the discourse processing platform on which LELIE runs) can recognize a variety of general purpose discourse structures often found in technical texts: conditions, circumstances, justifications, illustrations, reformulations, elaborations, purposes, etc. and structures specific to technical texts: requirements, instructions and titles, prerequisites, warnings, advice, etc.

The induction of the most frequent discourse structures is carried out by making statistics on the frequency and the position of these structures in sentences (mainly instructions and requirements for the general purpose structures) and paragraphs, and indicates the preferred positions. The most frequent positions are then proposed per structure type as guidelines or as by-default positions. The goal is to make texts more homogeneous, with a higher cohesion, so that they are easier to understand by readers. Results largely depend on the application, however, some general practices are observed.

Perspectives

The research reported in this paper has a direct application in LELIE and in many other authoring tools to acquire large portions of the lexical knowledge that is needed to customize a kernel such as LELIE to an application domain. This task is still often done manually by lexicographers, however, this task is long, costly and error prone. The results given in this paper, probably for the most crucial and complex lexical resources (business terms and fuzzy lexical items) show that it is possible automate this process. Obviously, a human validation will always be necessary, but the induced workload is much lower. Another element, briefly developed in this paper, is the automatic induction of the main discourse structures used in a domain or application or group of technical writers. The goal is to contribute to improving style homogeneity and cohesion. This is a major feature to facilitate the reading of technical texts, leaving the cognitive load for more central tasks.

These experiments remain quite empirical and probably need to be adjusted to different context, including the language and conceptual complexity of the domain and the critical level of the area in terms of risks for example. These experiments constitute the first step of the development of the LELIO platform, a generator of LELIE customized solution, where LELIE is a kernel that needs to be adapted on several levels to be really useful in an industrial context.

Acknowledgments

I thank Juyeon Kang for useful discussions a year ago that contributed to motivating this research. I am grateful to the CNRS, my institution, for letting me pursue this work. I also thanks the companies that contributed to the project via documents or analysis from technical writers. I however regret the large difficulties raised by our private valorisation institution (toulouse tech transfer) and an SME who made this project really difficult in terms of juridical and administrative constraints.

Table 1

 1

	below shows

Table 1 .

 1 Errors found in technical texts for companies A, B and C

Table 2 .

 2 Main fuzzy lexical classes: main distributions.

		An useful measure
	is the average number of fuzzy lexical items per category in an
	application:
	Category	Total number of lexical
		items in lexicon
	manner adverbs	210
	temporal and location adverbs 167
	determiners	24
	prepositions	38
	verbs and modals	252
	adjectives	197
	Since technical texts are a restricted textual genre in terms of
	lexical variation, the number of fuzzy lexical items per category
	remains manageable. The severity level depends on the context,
	however, some evaluation can be given a priori from experience
	(1 is low, 3 is high):
	Category	A priori severity level
	manner adverbs	2 to 3
	temporal and location adverbs in general 2
	determiners	3
	prepositions	2 to 3
	verbs and modals	1 to 2
	adjectives	in general 1
		Table 3. severity level.
	Finally, among these fuzzy lexical items, only a small number
	appear in a given document, but with rather high frequencies and
	in a number of diverse lexical and syntactic combinations:
	Category	Average nb of terms
		per application
	manner adverbs	18
	temporal and location adverbs 11
	determiners	8
	prepositions	9
	verbs and modals	18
	adjectives	21

Table 4

 4

. frequencies in specific documents.

  The main pattern is : [adverb(+evaluative) adjective]. where evaluative adverbs include: almost, very, rather, extremely, highly, etc. which are typical tests for scalar adjectives. To evaluate this analysis, let us consider a 130 page corpus from the financial software domain. The following results are obtained with our approach:

	Category	Average nb of items
	total nb of adjectives found	41
	Fuzzy adjectives from database 23
	New fuzzy adjectives	11
	Not fuzzy	7
	Table 5. adjective distribution.

  -general purpose action verbs, e.g.: accelerate, enlarge, maximize, , -general purpose communication verbs, e.g.: comment, argument, document, revise.

These cases are very difficult to detect without domain knowledge and inference patterns. Let us now concentrate on the detection of fuzzy verbs or deverbals alone. In technical documents, verbs which are fuzzy are mainly: -general purpose epistemic verbs, e.g.: analyze, evaluate, take into account, measure, model, adapt, -general behavior verbs, e.g.: avoid, allow, adopt, inspect, care, permit,


Marie Garnier
email: mgarnier@univ-tlse2.fr

Patrick Saint-Dizier

Error Typology and Remediation Strategies for Requirements Written in English by Non-Native Speakers

Keywords: requirements authoring, error analysis, error correction

In most international industries, English is the main language of communication for technical documents. These documents are designed to be as unambiguous as possible for their users. For international industries based in non-English speaking countries, the professionals in charge of writing requirements are often non-native speakers of English, who rarely receive adequate training in the use of English for this task. As a result, requirements can contain a relatively large diversity of lexical and grammatical errors, which are not eliminated by the use of guidelines from controlled languages. This article investigates the distribution of errors in a corpus of requirements written in English by native speakers of French. Errors are defined on the basis of grammaticality and acceptability principles, and classified using comparable categories. Results show a high proportion of errors in the Noun Phrase, notably through modifier stacking, and errors consistent with simplification strategies. Comparisons with similar corpora in other genres reveal the specificity of the distribution of errors in requirements. This research also introduces possible applied uses, in the form of strategies for the automatic detection of errors, and in-person training provided by certification boards in requirements authoring.

Motivation

In most international industries, English is the main language of communication for technical documents. Product specifications, business rules or requirements form a specific genre in technical documentation (all instances of this genre will hereafter be called 'requirements' for the sake of simplicity). These documents are designed to be easy to read and as efficient and unambiguous as possible for their users and readers. They must leave no space for personal interpretation. In the case of international industries based in non-English speaking countries, the professionals in charge of writing requirements are very often non-native users of English, and these professionals rarely receive adequate training in the use of English for this task. As a result, requirements can contain a relatively large diversity of lexical and grammatical errors. One of the standard ways of simplifying the task of writing requirements and harmonizing results is the use of a chosen controlled natural language for any particular business or industry. However, resorting to the use of controlled languages only offers a limited solution to the linguistic problems encountered by non-native English users: having been designed for native or near-native users of the language, controlled languages do not address the proper use of English grammar and lexicon, nor do they issue warnings about the loss of intelligibility that may occur as a consequence of lexical and grammatical errors (even though they do issue such warnings on other aspects linked to intelligibility). As a result of this situation, requirements may include errors that decrease readability, increase the risk of ambiguity and misinterpretation, and possibly lead to problems of misconception and lack of productivity and efficiency. Recent years have seen a rise in concern for good practices in requirements writing, and the corresponding development of training programs and certifications that go beyond the application of controlled languages (e.g. the International Requirements Engineering board, with the Certified Professional for Requirements Engineering certification). We feel it is now crucial to take into account specific authoring problems encountered by non-native users of English when they produce requirements, in order to guarantee a satisfactory level of language quality. In this project, we propose to document and analyze linguistic errors found in a representative corpus of requirements written in English by native speakers of French, and present the different ways in which the results from this research can be applied in order to improve language use in requirements.

Controlled Languages and the Language of Requirements

Various sectors of industry and business have seen the creation of guidelines for writing documents following controlled languages issued by user consortium and companies, resulting in a large diversity of specifications (e.g. ASD-Simplified Technical English, SLANG, Attempto simplified English [START_REF] Kuhn | A Principled Approach to Grammars for Controlled Natural Languages and Predictive Editors[END_REF], see also [START_REF] Wyner | On Controlled Natural Languages: Properties and Prospects[END_REF] for some examples of implementation). Norms tend to emerge, such as Semantics of Business Vocabulary and Business Rules (SBVR) Structured English for business rules and International Council on Systems Engineering-Object Management Group for requirements [START_REF] Hull | Requirements Engineering[END_REF].

The reader can consult a detailed synthesis and classification of CNL principles and projects in [START_REF] Kuhn | A Survey and Classification of Controlled Natural Languages[END_REF], which also investigates grammars for CNL.

Various companies in the sectors of space and aeronautics carry out further specialized analysis for critical systems. Additionally, RUBRIC, a Flexible Tool for Automated Checking of Conformance to Requirement Boilerplates, is dedicated to requirement control. The authoring principles and constraints developed for requirements and similar genres generally include dictionaries as well as rules limiting the types of grammatical structures that can be used. For example, passive forms and forms expressing the future are prohibited in most norms dealing with requirement authoring (e.g. INCOSE, IEEE830), and the use of modal auxiliaries is highly constrained, with some modals being prohibited while others being only used with a specific meaning (e.g. shall and must). At the lexical level, the number of verbs allowed for one domain is fairly limited, and the included verbs must be unambiguous and precise. The use of deverbal nouns is also not recommended.

Complex formulations are to be rephrased, following precise recommendations [START_REF] Grady | System Requirements Analysis[END_REF], [START_REF] Saint-Dizier | Challenges of Discourse Processing: the case of technical documents[END_REF], [START_REF] Fuchs | First-Order Reasoning for Attempto Controlled English[END_REF]. Stylistic constraints also abound, whether they are of a general nature or imposed by a given company, domain or genre. Given the complexity and range of these recommendations, it is not surprising on the one hand that native speakers have difficulties expressing requirements that comply with the recommendations, and on the other hand that non-native users struggle even more given the added difficulty of writing in a second language. Requiring the use of controlled and simplified languages does not guarantee that the documents will be less ambiguous and better written where non-native users are concerned (see [START_REF] Grady | System Requirements Analysis[END_REF]. To overcome some of these difficulties, predefined patterns (also called boilerplates) have been defined, but from authors' experiences, it seems that they are not flexible enough to adequately capture the level of complexity necessary in some requirements.

The need for assisting technical writers in their task (in particular those producing specifications) has motivated the development of the LELIE project [START_REF] Barcellini | Risk Analysis and Prevention: LELIE, a Tool dedicated to Procedure and Requirement Authoring[END_REF], [START_REF] Saint-Dizier | Challenges of Discourse Processing: the case of technical documents[END_REF]. LELIE is a system that detects several types of error in technical documents (see These results show that there is an average of about one alert issued every 3 lines of text (about every 50 words), which is a very high rate. Alerts are issued even on documents that have been proofread by several trained writers. These results show that the use of controlled languages is not a catch-all solution to linguistic problems in requirement engineering. There appears to be space for tools that address linguistic problems from another angle, and in the case of our project by targeting errors produced by non-native users writing requirements in English, and more specifically native speakers of French.

Research Methodology and Background

Error Analysis

Our research relies on the manual analysis of corpora of requirements written in English by native speakers of French. Research in Second Language Acquisition has shown the influence of a speaker's first language on their use of a second language (e.g. [START_REF] Jarvis | Crosslinguistic Influence in Language and Cognition[END_REF], notably through the phenomenon of transfer, or cross-linguistic influence. Language transfer plays a major role in error production, and gives precious indications as to the author's intended meaning and possible remediation. Choosing to focus on requirements written by speakers of French means we can better tailor corrections and recommendations to this type of user. However, our method can be reproduced for other languages (e.g. Spanish or Thai speakers writing requirements in English).

We use the methodology of error analysis, a research method initially developed and used in the domain of Second Language Acquisition (Corder, 1981), and which is recognized as a valid way of gathering data about language errors (Cook, 1993). The analysis is conducted by a single trained linguist specialized in English grammar with a background in research on linguistics-based automatic grammar checking for English. For this project, the different steps are performed manually, due to the complexity of the task. Requirements are read and screened for errors, which are then categorized.

It is well-known that, when performing complex linguistic tasks with natural language, it is always preferable to have several researchers working in parallel in order to be able to compare findings through the kappa test. However, from a practical point of view, it is very difficult to find the resources to do so. We are aware of the fact that there is already a large pool of research on L2 learners/users' errors, especially for English, as well as research on L2 corpora (e.g. [START_REF] Granger | The International Corpus of Learner English. Version 2. Handbook and CD-Rom[END_REF]. One of the objectives of our research will therefore be to find out whether the specificity of the task of writing requirements in English for industrial purposes warrants a specific treatment, or can be dealt with in the broader realm of second language acquisition research.

To find out whether errors found in requirements are only representative of errors produced by French speakers of a similar level in English in different types of writing, or highly specific to requirements writing, we compared our error corpus from requirements to a corpus of errors found in L1 French student essays in English, and to a corpus of errors found in scientific papers written in English by French native speakers.

Defining and Classifying Errors

Before presenting how errors are analyzed in our research, we need to clarify our definition of the concept, from which the criteria used for the detection of errors will stem.

At its most basic, an error is defined as "an unsuccessful bit of language" (James, 1998). The most common criterion for declaring a segment of language "unsuccessful" is grammaticality, that is to say whether or not the segment follows the rules of grammar. Acceptability is another criterion, which focuses on whether or not the segment might be produced by a native speaker in an appropriate context (Lyons, 1968). The concepts of competence and performance have also been explored in the definition of errors. Competence errors are attributable to a lack of knowledge in the language, while performance errors are due to external factors, such as lack of attention, stress or fatigue (Corder, 1967). However, researchers have highlighted the fact that even though this distinction is theoretically relevant, it is practically impossible to distinguish competence errors from performance errors (Thouësny, 2011). We adapt these criteria to the objectives of our project as well as to the nature of the documents in our corpus to define the types of segment identified as errors:

-Grammaticality: segments that don't follow morpho-syntactic rules (e.g. plural agreement), lexico-syntactic rules (e.g. choice of preposition after a verb), or other basic grammar rules are identified as errors.

-Acceptability: requirements need to be written as clearly and as intelligibly as possible so as to eliminate ambiguities and confusions, which leads us to stretch the notion of acceptability to include clarity and intelligibility. Segments that introduce ambiguities, lack clarity, or require an effort from the user to understand the intended meaning (e.g. stacking of modifiers in composite nominals) are identified as errors. In this case there is an overlap between errors as we define them and the structures that are prohibited in some controlled languages. -Performance errors/competence errors: spelling errors, since they are most likely due to a lack of attention, (especially since in other instances the correct spelling is used) and can easily be eliminated with the use of a spellchecker, are identified as performance errors. However, their presence in our corpus warrants their inclusion in our categories. Error categories usually rely on a set of criteria, used in combination or alone:

the linguistic domain of the error (e.g. morphology, syntax, spelling, etc.); the part of speech bearing the error or that needs to be modified to correct the error; the linguistic system linked with the error (e.g. agreement, verb complementation, etc.); the description of surface phenomena (e.g. word omission, extra word, word order, etc.). One of the objectives of classifying errors in our research is to obtain precise and comparable data about errors found in requirements. The type of classification that we estimate as the most adequate in order to yield comparable data should include the rank of the linguistic unit that needs to be taken into account for the error to become apparent (i.e. the error domain, [START_REF] Lennon | Error: some problems of definition, identification and distinction[END_REF], usually Noun Phrase, Verb Phrase, Clause, etc., as well as a number of sublevel categories giving more detailed information, such as Missing article, Preposition selection, etc. As a supplement to these domains and categories, we also include categories such as Lexicon, Continuity and coherence of discourse, and Punctuation, since errors in these categories are present in the corpus and have an effect on the overall quality and intelligibility of the requirements. Additionally, we have chosen to flag all errors, regardless of whether they could be successfully detected and corrected by a standard grammar checker, in order to give a neutral snapshot of the distribution of errors in requirement. Moreover, as stated below, the requirements used in our corpus have already gone through the initial stages of editing, and the presence of these errors seems to indicate that the use of grammar checkers is not a systematic practice in all industries.

Corpus Compilation and Validation

Ideally, a representative corpus of requirements should include the following specificities:

-requirements in the corpus must represent various professional activities where requirements are relevant: product definition, functions, properties and design, maintenance, production launch, safety; -requirements must come from various industrial areas; -requirements must follow various kinds of authoring recommendations, business style and format imposed by companies; -requirements must vary in complexity, and must have been produced by different types of staff (technicians, stakeholders, engineers), representing steps in the proofreading and certification process. We have strived to compile a corresponding corpus, but the sensitive nature of the documents concerned means that some aspects are difficult to control for. For example, for some portions of the corpus, it is difficult to know how many engineers have made corrections to them. We compiled a corpus of requirements written in English by French speakers from documents coming from four areas of industry (space exploration, aircraft maintenance, bio-pharmacy, and collaborative systems for computer networks). The corpus comes from four companies kept anonymous at their request. At the time of writing, our corpus is composed of the following resources, for a total of 1140 requirements:

space exploration: 470 requirements; some of these requirements are just 2 lines long whereas others can be half a page long, with equations, charts, figures, etc., -aircraft maintenance: 213 requirements; requirements are between 2 to 6 lines long in general, -bio-pharmacy: 234 requirements; requirements generally include one or two sentences, which are relatively complex in terms of syntax and style, -collaborative systems: 223 requirements; requirements are quite short, stored in a chart with cross references, level of validation, etc. These corpora have very different characteristics that offer an adequate representation of how French native speakers write in technical English:

-all texts were written by several authors, from three to six; these texts have therefore been produced by at least 15 different authors, -all authors are native speakers of French, with intermediate to advanced use of English, with large variations, -some of these texts have been proofread and validated whereas others are drafts, possibly hastily written, -authors follow different types of authoring guidelines; the layout is quite diverse and includes in particular charts and diagrams with text, besides simple textual forms; -these texts represent situations ranging from average to complex, in which requirements are not straightforward to write.

Results

The following results were obtained from the analysis of a subset of our corpus, including 550 requirements taken from the "space exploration" section of the corpus (Corpus A) and from the "aircraft maintenance" section (Corpus B).

Table 2 presents a synthetic list of categories of errors found in the corpus, along with authentic examples. Errors are distributed in 5 error domains and 1 Other category, with 21 sublevel categories in total. In the case of sublevel categories that have an extra level of detail (ex. NP Modification), these are indicated in parentheses next to the sublevel category, but only the total number is shown with the example representing the most numerous type. Errors segments are presented in context and their specific location is underlined for the sake of clarity. The source of the requirement is indicated by a letter preceding the example (A or B for the corresponding corpus). The percentages in the middle column show the proportion of errors from each category in the corpus of errors, with the 6 sublevel categories that each account for at least than 5 % of errors shown in bold. We found a total of 190 errors in 131 requirements, meaning that roughly 1 in 4 requirements contained at least one error, and 1 in 10 contained more than one error, usually in different categories.

Errors are most often found in the domain of Noun Phrase, which totals 58 % of errors, and the category Other, with 25.7 % of errors. In each of them, one sublevel category holds the majority of errors: most errors in the Noun Phrase category are linked to modification (and mostly to modifier stacking, as shown in the example), with 32.8 % of errors in total, and most errors in the Other category are linked to punctuation (mostly to missing commas, as shown in the example), with 10.5 % of errors in total.

In the case of errors linked to the use of prepositions, we have chosen to include them in the error domain of the word that is responsible for selecting the preposition, instead of creating an umbrella "preposition" category, as is often done. The error domain for prepositions only includes errors where the selection of the preposition is linked to semantics rather than to the lexical selection constraints of another word. If all errors linked to prepositions are gathered, they account for 8.5 % of all errors.

Discussion of Results

Discussion of Categories

As is the case in most classification systems for language errors, there is an amount of overlap and ambiguity between some categories. For example, in the case of errors attributed to a missing plural on the head noun, it is sometimes difficult to judge whether a given error results from a missing plural morpheme or is actually linked to other factors. This ambiguity arises particularly when we attempt to offer a correction. For example, in The Supplier shall demonstrate compliance with X for all system with electrical components or wiring, the proposed corrections could either be:

-to add a plural ending to the head noun: for all systems, -to add a definite determiner: for all the system, -to reformulate the noun phrase: for the whole system, -to replace the determiner, as the use of all could stem from a calque-type transfer of the French pour tout système, which actually translates to for any system. As is visible from the high number of categories with very low rates, there is a lot of variety in the errors found in requirements. We find low error rates in the domain of the Verb Phrase, which accounts for a large portion of errors in other studies [START_REF] Garnier | Utilisation de méthodes linguistiques pour la détection et la correction automatisées d'erreurs produites par des francophones écrivant en anglais[END_REF], since verbs are an integral part of sentences. In our corpus, the lack of errors in this domain can be attributed first to the high level of proficiency of the writers and the fact that the requirements are proofread, which may eliminate most agreement errors, and secondly to the use of a controlled natural language prohibiting the use of most complex verb groups. This could also explain the low rate of errors in the domain Sentence and Clause, as the constrained use of complex syntax helps limit errors. The sublevel categories with the highest proportion of errors concern errors which are difficult to detect and correct automatically, such as stacked modifiers (24.1 %, not shown in Table 2), missing articles (12.1 %) and missing commas (8.4 %, not shown in table 2). These categories also correspond to "simplification" strategies, with the omission of function words or punctuation that may be perceived by the author as superfluous or expendable. The sublevel category we have identified as "modifier stacking" gathers segments in which a noun phrase is composed of a head noun and a string of modifiers to its left, sometimes with their own embedded modifiers (the term of "stacking" is used loosely here, and not in its specific syntactic meaning). Here are a few examples of this error type:

-the following probable average operational duty cycle of the X -The system shall include a locking in full closed position device. -the reference computed ventilation flow (flight leg computed minimum reference flow i.e. X). Even though the use of several noun modifiers in an NP is becoming increasingly common in English, especially in technical and journalistic English (Pastor-Gomez, 2011), the fact that they rely on implicit information that needs to be reconstructed by the reader results in cognitive overload for the reader and possible loss of meaning [START_REF] Biber | Longman Grammar of Spoken and Written English[END_REF]. Finally, there is a category of segments that was initially included in the error corpus; these segments all contain a form of ellipsis that renders them ungrammatical. A few examples are shown below:

-In case of PDPS detected failed -On ground, when PACK is selected OFF, corresponding RARV shall stay in position -Both FCVs are selected open -During PACK starting sequence the FCV is not commanded fully open -In the event the RARV is failed stuck in position From the point of view of surface syntax, these segments have different forms (e.g. a past participle followed by a participial adjective, a past participle followed by a preposition, a past participle followed by one adjective or by an adjective phrase composed of a head adjective and a modifying adverb or a PP complement), but from a semantic point of view they are built on the same model, which is close that of verbal expressions such as to turn on, to switch off: the second term or phrase indicates the position or situation of a "mobile" element, such as a switch (e.g. OFF, ON, open, closed, failed), while the first one either specifies the action leading to that situation, or the observation of that situation (e.g. selected, detected). This type of segment was found only in Corpus B, with 31 instances (which made it one of the most numerous categories), but coming from only one company. It is imaginable that this type of phrasing is accepted and even expected in the company for which the requirements were written. This type of segment would therefore be an example of an ungrammatical but acceptable phrasing. These segments were removed from the list of errors, even though we did find an alternative phrasing in one instance (selected to OFF), suggesting that the practice is either not accepted or not stable. Furthermore, the ellipsis of prepositions and other function words, which may conversely be seen as increasing concision and simplicity of expression, also creates a gap that must be filled by the reader, and may therefore lead to ambiguities if the use of such structures is not homogeneous and well-mastered.

Comparison with Other Error Corpora

We compared the proportion of the two most common error types in our requirements corpus with those found in two corpora of English written by native speakers of French with an intermediate to advanced level. One of them is a corpus of argumentative student essays on broad topics while the second one is a corpus of scientific papers. In the case of missing articles, we notice that results in the two comparison corpora are similar, and are much lower than in the requirement corpus. However, determination errors in general account for 24.1 % (student essays) and 15.8 % (scientific papers) of all errors in the two comparison corpora, indicating that authors produce a more varied range of determination errors in these types of writing than in requirements, where determination errors other than missing articles are marginal to non-existent. Again, this can be interpreted as the result of "simplification" strategies having to do with the genre or requirements. In the case of modifier stacking, there is a progression in the number of such errors found in the three corpora, with them being marginal in the corpora of student essays. This is consistent with other studies on such structures (e.g. Pastor-Gomez, 2011), which identify them as a feature of technical, scientific or journalistic English. They are also more varied and complex in the corpus of requirements, with up to five modifiers on the left of a head noun (see examples above).

Error

We can conclude from this comparison that the distribution of errors seems to reflect the specificities of the genre of technical documents, warranting the collection and use of data in this genre, and specific treatment as an L2 production.

Applied Uses

Error Detection Using LELIE

One possible application of the results of this study is the inclusion of errors patterns for these error types in LELIE, as a grammar checking module complementing the system. Linguistics-based strategies are often overlooked in favor of statistical methods in standard grammar checking, but recent research has shown that using finely-tuned patterns based on linguistic research yields satisfactory results when confronted with very specific errors [START_REF] Garnier | Automatic Correction of Adverb Placement Errors for CALL[END_REF][START_REF] Garnier | Utilisation de méthodes linguistiques pour la détection et la correction automatisées d'erreurs produites par des francophones écrivant en anglais[END_REF]. Since they include information about the error pattern, and can incorporate annotations, pattern-matching methods also facilitate the inclusion of remediation strategies (i.e. corrected segments or guidance in the correction) and the creation of feedback messages for corrections or warnings. This aspect is a known weak spot of statistical methods. Early strategies for the detection or errors linked to modifier stacking are presented in [START_REF] Garnier | Utilisation de méthodes linguistiques pour la détection et la correction automatisées d'erreurs produites par des francophones écrivant en anglais[END_REF]. The implementation of this tool entails reusing or developing the following resources:

-the linguistic resources required to develop error patterns, and the features they must be associated with, especially action verbs, deverbal nouns, modal auxiliaries, adverbs, and determiners as well as their subclasses (these resources have already been developed within the framework of TextCoop and LELIE, and they are freely available for French and English), -the grammatical or discourse structures that must be processed (conditionals, causes, illustrations, reformulations, purposes, circumstances, etc.). The recognitions of these structures enables the detection of errors within specific constructs. Since technical documents form a constrained linguistic genre, the recognition of these structures is high (Saint-Dizier, 2014), -the type of error (or alert) message that is produced and the type of correction or suggestion. Explanations may also be included to help authors. Error messages and explanations are tailored to the user and to the domain, similarly to what is done in LELIE. This is implemented in the user profile and may be updated. Evaluating such a system is a complex process, and includes multiple elements. The first criterion for evaluation is the adequacy of the patterns to detect errors, checking for precision and recall. Research in computer-assisted language learning has shown that checkers targeting non-native speakers should aim at maximizing precision, even at the detriment of recall, in order to avoid false positives that might undermine the user's confidence in their knowledge and use of the language, and risk fossilizing the use of erroneous structures [START_REF] Tschichold | Grammar checking for CALL: Strategies for improving foreign language grammar checkers[END_REF], due to a belief in the authority of the automatic system. This guideline applies to requirements authoring as well, especially since the checker is not intended as a replacement for the several levels of proofreading done by human editors, but as assistance in the editing and correcting process. The adequacy of the resources is another crucial criterion in the evaluation of such a system, since gaps in the lexical resources used can lead to low recall rates.

Adapting the system to one given domain yields better results in the form of higher recall rates, but also translates into a lack of effectiveness when applied to other technical domains. Additionally, working with controlled English also means adjusting to the guidelines of the chosen CNL in terms of grammar. The adequacy of the error alerts for requirements authors is also a necessary part of the evaluation, and can be done through the analysis of users' behavior during the use of the system. This is generally achieved using questionnaires, interviews and direct observation of the use of the writing aid, with the help of researchers in ergonomics.

In-Person Training

As mentioned previously, recent years have seen the creation of organizations focused on improving requirements by providing training and certifications to requirement engineers. The results of this research can therefore serve as the basis of coursework or exercises to be used by training providers or trainees, with the necessary adjustments.

Error categories should be adapted to be more user-oriented than research-oriented:

the number of categories should be limited, in order to focus on central error types only; -the phrasing of categories should be changed in order to be usable by trainees with a less extensive knowledge of syntactic concepts and jargon; -remediation possibilities should be systematically included and diversified in order to provide a clearer illustration of error types. Our corpus and error categories can also be used to create tailored grammar and reformulation exercises, a type of training activity that has been requested by requirement training professionals. The evaluation of this applied use of the research would most likely be done as part of the evaluation of the training program performed by the organization offering it. Evaluation could include the results of a post-treatment test, compared with a pre-treatment test, as well as satisfaction questionnaires for trainees and instructors.

Conclusion

In this paper, we proposed a method based on corpus analysis aimed at identifying and documenting errors produced by native speakers of French when writing requirements in English. Errors are collected and classified using comparable categories based primarily on the domain of the error. The distribution of errors differs from what has been observed in similar corpora from different genres, indicating the need to take into account the specificities of requirements in order to provide adequate remediation. The results of this research can serve as the basis for the creation of a linguistics-based writing aid implemented in the LELIE system, and for in-person training activities. Because of the high proportion of errors linked to the Noun Phrase, and the specificity of the phenomenon of modifier stacking, we intend to focus our research on this error domain in order to obtain more precise data on these errors. In addition, the research will be extended to at least two additional industrial domains in order to yield more extensive comparable data in relation to the grammar of requirements.

Acknowledgements

Table 1

 1

	below), at any point in the authoring and revision
	stages. LELIE produces alerts that flag terms, expressions
	or constructions that need various forms of improvements.
	LELIE can also be tailored to the specific constraints of an
	industry or business, for example in the form of controls
	on style and the use of business terms.
	LELIE and the experiments reported below were
	developed on the logic-based <TextCoop> platform
	(Saint-Dizier, 2012). LELIE is fully implemented in
	Prolog; its kernel is freely available for French and
	English. The output of LELIE is the original text with
	annotations.
	Table 1 below shows a selection of the major errors found
	by LELIE. The corpus for this study includes about 35000
	words of proofread technical documents from three
	companies (kept anonymous). Alert numbers relate the
	number of errors observed all over the corpus, doubles
	included, since the same error can be quite frequent and
	generates the same problems with every instance.
	It is important to note that while following authoring
	norms is a necessity, this does not mean that texts become
	understandable. Norms are only one means to reach an
	authoring level that is simple and accessible to users.

Error type Nb. of alerts for 35000 words

	Fuzzy lexical items	182
	Deverbal nouns	80
	Pronouns with unclear reference	54
	Negation	126
	Complex discourse structures	127
	Complex coordinations	57
	Heavy N+N or noun complements	135
	Passives	92
	Sentences too complex	261
	Incorrect references to sections or figures	57

Table 1 .

 1 Selection of LELIE alerts in technical texts

This research was conducted with the support of a grant from the International Requirements Engineering Board, as part of the IREB Academy Program 2015-2016.