Andrew Blair-Stanek
email: ablair-stanek@law.umaryland.edu

Nils Holzenberger
email: nils.holzenberger@telecom-paris.fr

Benjamin Van Durme
email: vandurme@jhu.edu

Can GPT-3 Perform Statutory Reasoning?

Keywords: • Applied computing → Law, • Computing methodologies → Natural language generation, Reasoning about belief and knowledge natural language processing, reasoning, law, statutes, GPT-3

Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature. It is a basic legal skill. In this paper we explore the capabilities of the most capable GPT-3 model, text-davinci-003, on an established statutory-reasoning dataset called SARA. We consider a variety of approaches, including dynamic few-shot prompting, chain-ofthought prompting, and zero-shot prompting. While we achieve results with GPT-3 that are better than the previous best published results, we also identify several types of clear errors it makes. We investigate why these errors happen. We discover that GPT-3 has imperfect prior knowledge of the actual U.S. statutes on which SARA is based. More importantly, we create simple synthetic statutes, which GPT-3 is guaranteed not to have seen during training. We find GPT-3 performs poorly at answering straightforward questions about these simple synthetic statutes.

INTRODUCTION

There has been great excitement about the emergent capabilities of large language models such as GPT-3 [START_REF] Tom | Language Models are Few-Shot Learners[END_REF] and ChatGPT on legal tasks. For example, Bommarito and Katz [START_REF] Bommarito | GPT Takes the Bar Exam[END_REF] show that GPT-3 performs at a passing rate on evidence and torts questions on the U.S. multistate bar exam. In Choi et al. [START_REF] Choi | ChatGPT Goes to Law School[END_REF], law professors had ChatGPT take their final exams and blindly graded its output along with their human students' answers; it achieved low but passing grades. Alexis is a portle. Is section 1001(b) applicable to Alexis? Let's think step by step.

Section 1001(a) defines rolang as either soultratessly or parkinse.

Section 1001(b) defines soultratessly as either oxideney or chastiles.

Section 1001(c) defines parkinse as either portle or frestes.

Since Alexis is a portle, section 1001(b) is applicable to Alexis. We set out to test how the most powerful GPT-3 model currently exposed through public APIs would perform on one of the most basic tasks required of lawyers, statutory reasoning. Statutes are legal rules written by legislative bodies in natural language. Statutory reasoning is the application of those rules to facts that are also in natural language [START_REF] Sarah | A logic for statutes[END_REF]. We first consider the StAtutory Reasoning Assessment (SARA) dataset, a handcrafted computational benchmark covering nine sections of the U.S. tax code [START_REF] Holzenberger | A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering[END_REF]. We find that straightforward application of GPT-3 leads to significant gains compared to prior work on this task, but that GPT-3 makes a number of clear mistakes ( §3).

We then ask why this could be, starting with the question of how much GPT-3 already knows about the current U.S. Code ( §4). We find that it has some imperfect knowledge of the U.S. Code. Then we detail ( §5) an experiment based on novel synthetic statutes to measure how well GPT-3 can answer simple statutory-reasoning questions when prompted with a single fact (see Figure 1 for an example). When presented with statutory language that is easily interpretable by humans, we find that GPT-3 will regularly make mistakes. Our results suggest statutory reasoning as an area of interest for new AI research, as a challenge to motivate future improvements in large language models.

RELATED WORK

Statutory reasoning. The SARA dataset [START_REF] Holzenberger | A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering[END_REF] is a benchmark for statutory reasoning. Efforts to solve SARA through machine reading methods have had limited success [START_REF] Holzenberger | Factoring Statutory Reasoning as Language Understanding Challenges[END_REF]. SARA's curated statutes, facts, and questions were designed to avoid semantic ambiguity. But semantic ambiguity also poses challenges for statutory reasoning, such as the U.S. Supreme Court case Nix v. Hedden about whether tomatoes were "vegetables" in a customs statute. There have been several approaches for using NLP to handle statutory reasoning involving such semantic ambiguity [START_REF] Šavelka | Discovering Explanatory Sentences in Legal Case Decisions Using Pre-trained Language Models[END_REF][START_REF] Šavelka | Legal information retrieval for understanding statutory terms[END_REF]. The task of determining whether one statute entails another one, called statute entailment, is the closest legal NLP task to statutory reasoning. The yearly COLIEE challenge [START_REF] Rabelo | Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021[END_REF] includes statute entailment. BERT-based models [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] have been most successful on COLIEE's statute entailment, until they were recently surpassed by GPT-3-based models [START_REF] Yu | Legal Prompting: Teaching a Language Model to Think Like a Lawyer[END_REF]. Legal judgment prediction is another task related to statutory reasoning, and is also a significant challenge for current machine reading models [START_REF] Bi | Judicial knowledge-enhanced magnitude-aware reasoning for numerical legal judgment prediction[END_REF][START_REF] Lam | The Gap between Deep Learning and Law: Predicting Employment Notice[END_REF][START_REF] Zhong | Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction[END_REF].

Prompting GPT-3. GPT-3 is a large-scale language model, which has shown a surprising ability to solve NLP tasks without the need to fine-tune its parameters [START_REF] Tom | Language Models are Few-Shot Learners[END_REF]. This is achieved by providing GPT-3 with a set of natural language instructions, specifying what task is to be solved, along with a few input-output examples for the task. This prompt-based approach has been further refined, in particular by augmenting the expected output with human-written natural language explanations of how that output was derived from the input [START_REF] Wei | Chain of Thought Prompting Elicits Reasoning in Large Language Models[END_REF]. Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF] show that GPT-3 can be prompted to produce these explanations itself, providing a substitute for human annotation. In particular, the phrase "Let's think step by step" was found, out of a number of prompts, to maximize performance. We prompt GPT-3 in multiple ways, taking inspiration from Wei et al. [START_REF] Wei | Chain of Thought Prompting Elicits Reasoning in Large Language Models[END_REF] and Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF].

Reasoning with GPT-3. Exploring GPT-3's ability to reason has yielded improvements on multiple benchmark datasets. Lu et al. [START_REF] Lu | Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering[END_REF] collect a dataset of common sense and science questions, pairing each question with background information and an explanation of the answer. The additional context, serving as a chain-of-thought, marginally improves GPT-3's performance. Zhou et al. [START_REF] Zhou | Least-to-Most Prompting Enables Complex Reasoning in Large Language Models[END_REF] explicitly decompose reasoning problems, forcing GPT-3 to solve a complex problem in multiple increments. Khot et al. [START_REF] Khot | Decomposed Prompting: A Modular Approach for Solving Complex Tasks[END_REF] show GPT-3 can be prompted to decompose problems into simpler steps, on both artificial tasks and open-domain question-answering. Zelikman et al. [START_REF] Zelikman | STaR: Bootstrapping Reasoning With Reasoning[END_REF] exploit a language model's ability to generate chains of reasoning: GPT-J [START_REF] Wang | GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model[END_REF] is made to generate chains of reasoning for a set of questions and answers, and is further fine-tuned on those chains that lead to the correct answer. Adding structure to the process of natural language generation, Zhang et al. [START_REF] Zhang | Improved logical reasoning of language models via differentiable symbolic programming[END_REF] run a symbolic reasoning engine on the output of a language model, yielding improvements on semi-synthetic benchmarks meant to test reasoning abilities. Thorburn and Kruger [START_REF] Thorburn | Optimizing Language Models for Argumentative Reasoning[END_REF] evaluate GPT Neo's argumentative abilities, by making it suggest claims or reasons to support a claim, with no particular emphasis on logical reasoning. Data comes from Kialo, a collection of arguments curated by online users. Evaluations show that reasons and claims generated by GPT Neo are less coherent than those generated by humans. Saparov and He [START_REF] Saparov | Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought[END_REF] systematically study GPT-3's ability to reason, using synthetic, controlled data, and metrics that measure the internal coherence of chains of reasoning. The authors show that GPT-3 has the ability to perform individual steps of reasoning, but tends to lose track when producing a proof with multiple steps. Results further show that GPT-3 leverages its background knowledge about the world to draw conclusions, so that GPT-3's ability to reason decreases as the overlap between the target domain and its real-world knowledge decreases.

GPT-3 on legal tasks. Hendrycks et al. [START_REF] Hendrycks | Measuring Massive Multitask Language Understanding[END_REF] use GPT-3 to solve legal multiple-choice questions, with data collected from online sources. Results show that the largest GPT-3 model performs significantly better than random, but still far less than expert human performance. Yu et al. [START_REF] Yu | Legal Prompting: Teaching a Language Model to Think Like a Lawyer[END_REF] experiment with various ways of prompting GPT-3 for statute entailment. The authors find that GPT-3 does much better than previous BERT-based models. In particular, prompts designed with inspiration from legal reasoning work best. Bommarito and Katz [START_REF] Bommarito | GPT Takes the Bar Exam[END_REF] test GPT-3's ability to answer multiple choice questions, involving short paragraphs of context, from the U.S. multistate bar exam. Prompting GPT-3 to rank the available answer choices gets close to a passing score on some subject areas. In the medical domain, Gutierrez et al. [START_REF] Bernal | Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again[END_REF] study GPT-3's abilities on two biomedical information extraction tasks. The authors show that fine-tuning a BERT model on a small training set consistently outperforms few-shot learning with GPT-3.

SARA 3.1 Data and Experimental Setup

SARA stands for StAtutory Reasoning Assessment. It consists of nine statutory sections drawn from the U.S. tax code (i.e. title 26 of the U.S. Code) and 376 hand-crafted cases that state simple facts, involving taxpayers named Alice, Bob, Charlie, and Dan, and ask one question that can be answered by applying some of the nine sections to the facts. Obviously, the nine statutes, the facts, and the questions are all in natural language. But SARA also includes a translation of the statutes, facts, and questions to Prolog. Solving the questions via Prolog results in 100% accuracy.

Of the 376 cases, 100 are pure "tax cases" and ask for how much tax an individual had to pay, with answers typically in the many thousands of dollars. Because GPT-3 performs poorly in reasoning with such numbers, we do not consider these 100 cases. All the remaining 276 cases ask for either "Entailment" or "Contradiction". Of these, 193 (121 train, 72 test) involve numbers. The remaining 83 (55 train, 28 test) involve no numbers at all. For all our tests, we break out our results separately on the 72 test cases with numbers and the 28 test cases with no numbers.

In all experiments throughout this paper, for our calls to GPT-3, we set the temperature to 0.0 and the top_P to 1.0. 1 These settings serve two purposes. First, they maximize reproducibility. Second, Figure 2: Prompts used in Section 3 to pose SARA test cases to GPT-3. Top boxes, in orange, contain statutes; depending on the setting, statutes may or may not be included. Example cases are in blue; in zero-shot there are no example cases. At the bottom, in green, are test cases. Text highlighted in yellow is generated by GPT-3. If GPT-3's first response is not clear, we prompt a second time with "Therefore the answer is", following Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF].

they minimize the risk that GPT-3 will wander off topic or hallucinate. In SARA, every question is annotated with a single correct answer.

Three Approaches

We try three approaches to prompting GPT-3 to answer the SARA questions.

First, we use 4-shot dynamic prompting. For each test case, we picked the 4 most similar training cases. For these 4, we gave the full text of the case, labeling the "Premise", "Hypothesis", and "Answer", the latter being either "Entailment" or "Contradiction". Because SARA is balanced, 2 of these 4 training cases had "Answer: Entailment" and the other 2 had "Answer: Contradiction". Then we appended the "Premise" and "Hypothesis" of the test case, followed by the word "Answer". This is illustrated in the left column of Figure 2.

Second, we use zero-shot prompting. There we use no training cases at all. We just put the test case's "Premise" and "Hypothesis" and then "Answer". This is illustrated in the right column of Figure 2.

Third, we do 10-shot chain-of-thought prompting, using a prompt involving 10 of SARA's training examples. For each of the 10, we have the "Premise", the "Hypothesis", and "Answer", followed by a human-written chain of thought explaining the relevant reasoning and concluding with the appropriate word "Entailment" or "Contradiction". The same 10-example prompt is used for all test cases. This is illustrated in the center column of Figure 2. GPT-3 does not always complete its answers with "Entailment" or "Contradiction". In such cases, following Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF], we then pass to GPT-3 our original prompt, plus GPT-3's response, plus an Answer Extraction prompt reading "Therefore, the answer (Entailment or Contradiction) is".

"Let's think step by step."

Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF] find substantial improvement on a range of tasks by simply adding "Let's think step by step. " at the end of the prompt passed to GPT-3. They find this forces GPT-3 to explain its reasoning step by step and thus arrive at better answers. We try each of the zero-shot and 4-shot dynamic prompting experiments both with and without "Let's think step by step." A column in Table 1 indicates whether each row's result used this approach. Note that we do not use this approach for the 10-shot hand-crafted chain-ofthought, since the 10 examples already contain detailed step-by-step reasoning.

With or Without Statute

We hypothesize that GPT-3 may have seen the U.S. tax code in its training data, since U.S. federal statutes cannot be copyrighted and thus appear on multiple websites. So for each experimental setup, we try all the SARA prompts both with and without the text of the statute(s) required to solve all the hypotheses presented, as specified in the "Include Statute" column in Table 1. In 4-shot dynamic prompting and zero-shot prompting, we use the most relevant section from the statutes. For 10-shot chain-of-thought 1: GPT-3 performance on SARA with various setups. Below the double line are non-GPT-3 approaches. We report both the 90% confidence interval and the raw number of accurate answers.

prompting, we use a set of sections most relevant to the entire test set: 2(b), 63(c)( 1) and ( 6), 152, 3306(a), and 7703. When we do not include the statutes, we replace all instances of "section" in the SARA premises and hypotheses with "I.R.C. section". (The U.S. tax code is formally known as the Internal Revenue Code, and "I.R.C. " is the standard citation.)

Discussion

Results for the settings described above are reported in Table 1. As expected, GPT-3 is quite sensitive to the prompt setting, with large variations in performance across settings. The prompt "Let's think step by step." occasionally improves statutory reasoning performance, though not systematically. The BERT-based state of the art we report uses some amount of human-provided structure. Some of our prompt settings surpass that previous state of the art, with far less human input, on the full set of 100 entailment cases.

As measured by Welch's unequal variances 𝑡-test, our best GPT-3based model performs significantly better than the BERT-based state of the art (𝑝 ≈ 0.038), and than the majority baseline (𝑝 ≈ 0.0011). With 𝑝 < 0.05, only our best model has greater accuracy than the previous state of the art, and only our top two have greater accuracy than the majority baseline.

We note that on nine of our ten experimental setups, GPT-3 performs at least as well on the 28 cases involving no numbers than on the 72 cases involving numbers. This is expected, since most of the numbers are quite large (e.g. "In 2016, Alice's income was $310192. ") and GPT-3 has a known limitation in doing calculations with large numbers.

As expected, GPT-3 performs at chance (0.50) or worse in the zero-shot tests where there is no statute included. But we were quite surprised to examine GPT-3's step by step reasoning, which shows some (flawed) knowledge of the U.S. tax code. Here is one example, with our full prompt in regular font, and GPT-3's full response highlighted: Premise: Bob is Alice's brother since April 15th, 2014. Hypothesis: Alice bears a relationship to Bob under I.R.C. section 152(d)(2)(A). Answer: Let's think step by step. First, I.R.C. section 152(d)(2)(A) states that a taxpayer is allowed to claim a dependent if the dependent is a "brother or sister by whole or half blood. " Since Bob is Alice's brother, this would qualify him as a dependent under I.R.C. section 152(d)(2)(A). Therefore, the hypothesis is correct.

Recall that no statutory text was provided in the prompt, so we see that GPT-3 thinks it knows about the U.S. tax code. But its knowledge is incorrect, since section 152(d)(2)(A) actually states that an individual is a qualifying relative if they are "(A) A child or a descendant of a child" of the taxpayer. The closest match is in the following subparagraph, section 152(d)(2)(B), which lists "(B) A brother, sister, stepbrother, or stepsister." In other words, GPT-3 went awry in two ways: first, it mistakenly referred to the wrong subparagraph; and second, it confused half-siblings with step-siblings. As a result of this and similar results, we run experiments, described below in Section 4, to determine how much of the U.S. Code GPT-3 knows.

GPT-3's tendency to mistakenly refer to the wrong part of a statute is not limited to prompts that do not include the text of the statutes. It happens even when we include the statute in the prompt itself. An example is shown in Figure 3. Given the facts in the prompt, GPT-3 correctly reasons that Alice is Bob's stepmother. But then it confuses section 152(b)(2)(D) with (b)(2)(C) and gives the wrong answer. Because of errors like these, we set out in Section 5 to test GPT-3's ability to reason on synthetic statutes that it has never seen before.

Does GPT-3 know about SARA?

SARA was first made available on the internet in 2020, so it is possible that SARA was in GPT-3's training data. This, in turn, might have biased the results of these experiments. To test this possibility, we first asked GPT-3 "What is the StAtutory Reasoning Assessment (SARA)?" and it gave an answer that it was a multiple choice test for "law students, lawyers, and other legal professionals. " (We know of no such test.) We then tried passing 20 randomly selected SARA cases followed by the text "Where does the text above come None of the answers remotely implicated SARA. The modal answer (5 of 20) was that the language was from the IRS website.

GPT-3'S AWARENESS OF THE U.S. CODE

As discussed earlier, GPT-3's responses to zero-shot SARA prompts without any statutory text demonstrate that it has some, possibly imperfect, knowledge of the U.S. tax code. In this section we probe what GPT-3 knows about the tax code and U.S. Code in general. The U.S. Code or U.S.C. is the official compilation of permanent U.S. federal statutes. It is organized into 53 numbered titles, each corresponding to a particular topic. Title 26, also called the Internal Revenue Code, is the U.S. tax code.

We analyze GPT-3's knowledge of the U.S. Code in two directions. One is to give GPT-3 the full text of a section of the U.S. Code and ask it to identify its origin, including title and section. The other is to ask GPT-3 to recite the text of a particular section and compare its answer to the actual text.

Given statutory text from the U.S. Code, GPT-3 can often identify the title number

For each of the 53 titles of the U.S. Code, we randomly selected 10 sections whose bodies are between 100 and 1000 words. The lower limit of 100 words was set to exclude statutes with little semantic content, such as 26 U.S.C. §8003, which in its entirety reads "The Joint Committee shall elect a chairman and vice chairman from among its members." The upper limit of 1000 words was set to avoid running into GPT-3's hard limit of 4000 total tokens, with each word potentially consisting of multiple tokens.

For each section, we passed a prompt consisting of the text of the statute (not including the section's name, number, or title), followed by a newline and the text "Where is the text above from?". Once we had a response, following Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF], we passed the original prompt and response, plus the text "So is it from the U.S.

Code? The answer (Yes or No) is". If the answer was Yes, we passed all the previous prompts and responses concatenated, plus the text "What title of the U.S. Code is it from? The answer (arabic numerals) is". If we got the correct answer, we passed all the previous prompts and responses concatenated, plus the text "What section of title N of the U.S. Code is it from? The answer (arabic numerals) is" where N is the proper title. The results from our 530 tests are in Table 2.

Of the 265 sections for which GPT-3 correctly identified the title but could not correctly identify the section, it identified a numerical section for 250 of the 265. Of these 250, for 36 the section was off by just one (e.g. GPT-3 identified text as from §104 when it was actually from §103). For exactly 100 sections, GPT-3 was off by nine or less (e.g. GPT-3 identified text as from §161 when it was actually from §170).

Even many of the 64 prompts to which GPT-3 responded "No" when asked "So is it from the U.S. Code? The answer (Yes or No) is" are understandable. For example, GPT-3 said that the text of 34 U.S.C. §10707 is not from the U.S. Code. Rather, GPT-3 responded, "The text above is from the Comprehensive Opioid Abuse Grant Program Evaluation Act of 2016. " The section actually was added by the Comprehensive Addiction and Recovery Act of 2016. GPT-3 got the title of the Act slightly wrong and entirely missed its subsequent codification into the U.S. Code.

Of particular relevance to understanding GPT-3's performance on SARA, we separately asked GPT-3 to identify the text of the nine curated statutes in SARA. GPT-3 correctly identified the title and exact section for eight of the nine. (It incorrectly identified 26 U.S.C. §3301 as §3111.) This high performance makes sense, given that the sections used in SARA are central to individuals' tax calculations and thus more likely to be talked about extensively in the training corpus for GPT-3.

Not from U.S. Code 12% (64)

Wrong title 25% (133) Right title, wrong section 50% (265) Title and section correct 13% (68)

Table 2: GPT-3's answers when given the text of the body (but not the name, section number, or title number) of a section from the U.S. Code.

Given a U.S. Code citation, GPT-3 can recite plausible but incorrect statutory language

We just saw that GPT-3, when given statutory text from the U.S. Code, is decent at identifying it as such (89% correct) and even identifying its title number (63% correct). We now turn to the flip side: given a citation from the U.S. Code, can GPT-3 recite the text? For 10 randomly selected statutes in each of the 53 titles of the U.S. Code, we prompted GPT-3 with "The text of __ U.S. Code section __ is:", with the title number in place of the first blank and the section number in place of the second. 2We never saw GPT-3 exactly recite statutory text. It always provides an answer in the same style as the U.S. Code, using Congress' formal, dry style and often organizing into numbered subsections, paragraphs, subparagraphs, etc. Sometimes GPT-3's recitation of a section gets the gist of the section correct, but omits some statutory language and inserts some nonexistent statutory language. But many times GPT-3 recites text that gives the gist of an entirely incorrect section.

We aimed to quantify how closely GPT-3 gets at least the gist of sections. One natural possibility is BLEU, a score used to evaluate machine translations against reference human-generated highquality transactions [START_REF] Post | A Call for Clarity in Reporting BLEU Scores[END_REF]. But standard BLEU strongly penalizes translations that are shorter than the human-generated reference task, and GPT-3 often recites only the key portions of a statute. So, we use the metric of unpenalized-BLEU, which is the standard BLEU score without the brevity penalty. Thus, like BLEU, unpenalized-BLEU ranges from 0 (worst) to 100 (best).

Across the 530 sections we asked GPT-3 to recite, we got a mean unpenalized-BLEU score of 7.11 and median of 4.52. Only 28 of the 530 have an unpenalized-BLEU above 20. For machine translation standards, these are fairly poor results.

Given that GPT-3 often recites roughly correct statutory language for a section different than the one requested, we calculated the unpenalized-BLEU score of the recited text against all the sections in the correct title. For example, for 17 U.S.C. §101 we computed the unpenalized-BLEU score of GPT-3's recitation of the section against all 147 sections contained in title 17. We can then rank the actual U.S.C. sections against GPT-3's prediction. With this, we can measure the rank of the expected section, and thus recall@k. Recall@1 is only 1.5% (8 of the 530 sections sampled) and recall@5 is only 4.7% (25/530).

These results led us to evaluate whether GPT-3's recited text reflected the section number at all. For each of the 530 samples, we normalized the rank of the correct answer within the title by subtracting 1 from the rank and dividing by the number of sections in the title, minus 1. If the correct answer was ranked first for a given section, its normalized recall would thus be 0.0; if GPT-3 is paying no attention whatsoever to the requested section number, we would expect the normalized rank to average 0.5. We observed a mean of 0.48 and median of 0.46, which suggests that GPT-3 pays little attention to the requested section number.

GPT-3 STRUGGLES WITH SIMPLE REASONING ON SYNTHETIC STATUTES

We know of no technique to determine how much of GPT-3's flawed knowledge of the U.S. Code impacts its performance on SARA, which is based on nine sections drawn from the U.S. Code. To test GPT-3's ability to reason on statutes it has definitely never seen before, we wrote code to create entirely synthetic statutes, using the same numbering style (i.e. subsection, paragraph, etc.) as the U.S. Code. We then systematically prompt GPT-3 with the basic statutory reasoning task of giving it one fact and asking it to determine whether a particular subsection applies. 3 These statutory-reasoning prompts are similar to SARA, involving self-contained statutes, facts, then simple questions. Also, we take inspiration from the setting of Saparov and He [START_REF] Saparov | Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought[END_REF].

Creation of Synthetic Statutes

An example of one of our synthetic statutes appeared in Figure 1.

To avoid any ambiguity, the statutes involve repeated application of the most basic logical form 𝐴 =⇒ 𝐵, such as parkinse =⇒ rolang. In other words, the statutes define terms (rather than, say, imposing duties). Many real-world statutes consist of precisely this sort of term defining, such as 26 U.S.C. §152(a)(1) which provides qualifying child =⇒ dependent.

The statutes are defined in terms of depth and width. The width is the number of times, for each term 𝐵, a relation of 𝐴 =⇒ 𝐵 is stated. The depth is the maximum number of times this logical form can be applied. For example, the synthetic statutes in Figures 1 and4 are both 2-wide and 2-deep. Meanwhile, the synthetic statute in Figure 5 is 2-wide and 3-deep. Internally, our code represents these statutes using trees.

Our synthetic statutes are much simpler than many sections of the U.S. Code. They involve no negation or ambiguity, and they are entirely balanced, with all leaf nodes being the same depth from the root, and all non-leaf nodes having the same number of children. In contrast, many sections of the U.S. Code have depth 5, with subsections, paragraphs, subparagraphs, clauses, and subclauses, such as 26 U.S.C. §2.

To ensure we are testing GPT-3 on synthetic statutes it has definitely never seen before, we take two approaches to generating the terms being defined. The first is to use English nonces generated by a nonce generator; examples include roland, parkinse, and oxideney. The second is to use random ids consisting of one letter plus the same numeral repeated twice; examples include s88, f77, m55, and a22.

Zero-Shot Experimental Setup

For zero-shot prompting involving our synthetic statutes, as shown in Figure 1, the prompt we give to GPT-3 consists of four concatenated strings:

• the synthetic statute • a single fact in the form "__ is a __.", as with "Alexis is a portle. " The name is randomly chosen from a set of 30 names The person is N, and S is the subsection. T is the top-level term being defined in the synthetic statute. Note that in all the synthetic prompts we generate, person N is a T ; the only question is whether that happens through S. Note that questions including T uniformly perform worse than the two without. We ran an extra 2000 experiments to choose between the top two phrasings.

(15 female, 15 male). 4 The second part of the fact (e.g. portle) is always a term present somewhere in the synthetic statute. • a question about whether a subsection applies (e.g. "Is section 1001(c) applicable to Alexis?"). This subsection is randomly chosen, although it is never a leaf, meaning it is never a subsection taking up only a single line. We chose the phrasing "Is __ applicable to __?" after trying a variety of different ways of phrasing the question, with the results shown in Table 3.

• the text "Let's think step by step.", where Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF] found that adding this at the end of the prompt maximizes performance on a variety of reasoning tasks.

We prompt GPT-3 with this string to get an initial response. To derive a definitive answer we then call GPT-3 a second time with the prompt being the original prompt, plus the initial response, and the text "Therefore, the answer (Yes or No) is". This follows Kojima et al. [START_REF] Kojima | Large Language Models are Zero-Shot Reasoners[END_REF] and is often necessary to force GPT-3 to give a clear answer. Examples of this prompting can be seen in Figure 4.

To test whether GPT-3 has a problem specifically with reasoning over statutes, our code also generates semantically-identical sentence versions of each synthetic statute. For each defined term in the synthetic statute, our code creates one sentence with the exact same definitional language as the corresponding part of the statute. These sentences are numbered as "Sentence 1", "Sentence 2", etc. for reference in determining whether a particular sentence is applicable. For each zero-shot problem posed to GPT-3, we test the semantically-identical sentence version of the same statute and question. An example appears at the bottom of Figure 4. For these sentence versions, the prompt again consists of four parts:

• the numbered sentences expressing the same definitions as the synthetic statute. • the same fact as in the statutory version (e.g. "Alexis is a portle. ") • a question about whether a sentence applies (e.g. "Does sentence 3 apply to Alexis?"). 4 https://www.ssa.gov/oact/babynames/decades/names2000s.html

• the text "Let's think step by step. "

We run all tests using both nonce and ids terms. We also run all tests using both the statute version of the definitions and the semantically-identical sentence-based versions. We balance all tests with equal numbers having positive groundtruth (e.g. section 1001(b) is not applicable) and negative groundtruth (e.g. section 1001(b) is applicable).

The results are in Table 4. Our primary finding is that GPT-3 performs poorly, with performance around 78% for even the simplest statutes, like that in Figure 1 4: Zero-shot accuracy of GPT-3 on answering whether a statutory section applies to Alice, given a single fact about Alice. The column "term" indicates whether the statutory terms were nonces like "ansgivath" and "propial" or ids like "f55" and "q11". The rightmost column is GPT-3's accuracy answering questions based on sentences that are semanticallyidentical to the synthetic statute, as seen in Figure 4. Emma is an artion. Is section 1001(b) applicable to Emma? Let's think step by step.

Section 1001 defines the term "infarber" as either a "purentiable" or a "digirderasters". Emma is an artion, which is a type of digirderasters. Therefore, section 1001(b) is applicable to Emma.

Therefore, the answer (Yes or No) is Yes.

Sentence 1: The term "infarber" means any purentiable or any digirderasters.

Sentence 2: The term "purentiable" means any packle or any amperseced.

Sentence 3: The term "digirderasters" means any artion or any irtityrating.

Emma is an artion. Is sentence 2 applicable to Emma? Let's think step by step.

Sentence 2 states that the term "purentiable" means any packle or any amperseced. Emma is an artion, which is a type of digirderaster. Therefore, sentence 2 is not applicable to Emma.

Therefore, the answer (Yes or No) is No. We notice comparable performance between statutes with terms that are nonces like portle and statutes with terms that are ids like m77. This makes sense, since both types of synthetic statutes were designed to be entirely novel to GPT-3.

We also observe comparable performance whether the statute is presented in statute version or semantically-identical sentencebased versions. GPT-3 correctly answers 6537 of the 9000 statute questions but 6562 of the 9000 sentence-based versions.

GPT-3's errors are overwhelmingly false positives, meaning it concludes that a section or sentence applies when it actually does not. In the 9000 nonce runs, there were 2272 errors. Of these, 2204 were false positives and 61 false negatives. (There were also 7 where GPT-3 did not give a definitive answer.) Table 5: Two-shot accuracy of GPT-3, when the prompt has two example questions with reasoning and correct answers, as in Figure 5. The leftmost column reproduces the comparable zero-shot accuracy from Table 4.

Two-Shot Experimental Setup

Given GPT-3's poor performance on the zero-shot statutory reasoning discussed above, we turned to whether GPT-3 performs better with two-shot reasoning. In two-shot reasoning we give two correct examples (one answered Yes, the other answered No), before posing the actual question. All two-shot experiments were run with synthetic statutes where the terms were nonces. (We used neither semantically-identical sentences, nor ids like g11.) A two-shot example involving a 2-wide, 3-deep statute appears in Figure 5.

Both the correctly answered questions are of the form "__ is a __. Does section __ apply to __?", followed by a correct explanation and answer. The Yes-answered question comes first with 50% probability. Both questions relate to the same section, which never overlaps with the section asked about in the third question. All three questions involve different randomly-selected names (e.g. Hannah, Alyssa, Nicholas) and randomly-selected terms (e.g. chastiles, goghts, pushotyptopses).

We see uniformly better performance with the two-shot model than with the zero-shot model (Table 5). With two examples, GPT-3 seems capable of handling 100% of 2-wide, 2-deep statutes, such as those shown on our first page in Figure 1. As in the zero-shot setting, GPT-3's accuracy tends to decrease as the statutes' depth and width increase. GPT-3 still performs quite poorly (81%) on 3wide, 3-deep statutes, which themselves pale in comparison to the complexity of many sections in the U.S. Code.

CONCLUSION

Being able to identify whether a specific subsection of an unfamiliar statute is applicable to a given set of facts is one of the most basic skills required of a lawyer. We find that, given a very simple 2wide, 2-deep synthetic statute paired with a single fact and single question (zero-shot), GPT-3 had 78% accuracy, raising doubts about GPT-3's ability to handle basic legal work. Providing two examples of correct statutory reasoning (two-shot reasoning) improves GPT-3's performance, but it still achieves only (81%) on 3-wide, 3-deep synthetic statutes, which are far less complex than what is found in the U.S. Code. This poor performance on synthetic statutory reasoning allows us to understand our results applying GPT-3 to the SARA dataset. These results, at 71% accuracy, are better than the previous state of the art, but leave significant room for improvement.

We hope this work motivates further research into improving the performance of large language models on statutory reasoning. We also look forward to testing new large language models, like GPT-4, on statutory reasoning.

Section 1001 .

 1001 Definition of Rolang. (a) General rule The term "rolang" means-(1) any soultratessly, or (2) any parkinse. (b) Soultratessly The term "soultratessly" means-(1) any oxideney, or (2) any chastiles. (c) Parkinse The term "parkinse" means-(1) any portle, or (2) any frestes.

Figure 1 :

 1 Figure 1: Example of GPT-3 incorrectly answering a simple question about a short synthetic statute. The text generated by GPT-3 is highlighted.

( 2 )

 2 Relationship For purposes of paragraph (1)(A), an individual bears a relationship to the taxpayer described in this paragraph if the individual is any of the following with respect to the taxpayer: (A) A child or a descendant of a child. (B) A brother, sister, stepbrother, or stepsister. (C) The father or mother, or an ancestor of either. (D) A stepfather or stepmother. ... Premise: Charlie is Bob's father since April 15th, 1995. Dorothy is Bob's mother. Alice married Charlie on August 8th, 2018. Hypothesis: Alice bears a relationship to Bob under section 152(d)(2)(C). Answer: In 2018, Alice is the wife of Charlie. Charlie is Bob's father. Therefore, Alice is Bob's stepmother. Therefore, the answer is Entailment.

Figure 3 :

 3 Figure 3: Example of SARA getting the wrong answer, because it confused section 152(d)(2)(C) with (d)(2)(D). This example came from the 10-shot chain-of-thought prompting with the statutes included. We elided much of the prompt for brevity.

Section 1001 .

 1001 Definition of Infarber. (a) General rule The term "infarber" means-(1) any purentiable, or (2) any digirderasters. (b) Purentiable The term "purentiable" means-(1) any packle, or (2) any amperseced. (c) Digirderasters The term "digirderasters" means-(1) any artion, or (2) any irtityrating.

Figure 4 :

 4 Figure 4: Top: an example of GPT-3 incorrectly answering a zero-shot question about a 2-wide, 2-deep statute. Bottom: GPT-3 correctly answering the semantically-identical question where the synthetic statute was written in sentence form. Text generated by GPT-3 is highlighted. Note that sentence 1 corresponds exactly to section 1001(a), sentence 2 to 1001(b), and sentence 3 to 1001(c).

Figure 5 :

 5 Figure 5: Two-shot prompt with GPT-3 giving an incorrect answer. This is a 2-wide, 3-deep statute. Note that the two examples given to GPT-3, involving Hannah and Alyssa, are correct. GPT-3's answer shows is has incorrectly looked at section 1001(b)(2), not (c)(2).

Table 3 :

 3 Results from different phrasing styles of the same question, all measured on nonce 2-wide, 2-deep synthetic statutes.

	phrasing	example	accuracy (%)
	Is S applicable to N ?	Is section 1001(b) applicable to Alexis?	77 (2303/3000)
	Does S apply to N ?	Does section 1001(b) apply to Alexis?	74 (2220/3000)
	Does S apply to N, making her/him a T ? Does section 1001(b) apply to Alexis, making her a rolang? 54 (540/1000)
	Does S apply to make N a T ?	Does section 1001(b) apply to make Alexis a rolang?	58 (576/1000)
	Is N a T because of S?	Is Alexis a rolang because of section 1001(b)?	57 (565/1000)
	Is N a T owing to S?	Is Alexis a rolang owing to section 1001(b)?	54 (535/1000)
	Is N a T as per S?	Is Alexis a rolang as per section 1001(b)?	52 (519/1000)

  , that are 2-wide, 2-deep. Performance declines further as the statutes get wider or deeper.

				accuracy on accuracy on
	term width depth statutes (%) sentences (%)
	nonce	2	2	78 (779/1000) 79 (793/1000)
	ids	2	2	78 (778/1000) 83 (834/1000)
	nonce	3	2	70 (698/1000) 72 (718/1000)
	ids	3	2	68 (682/1000) 69 (692/1000)
	nonce	4	2	69 (688/1000) 72 (723/1000)
	ids	4	2	66 (663/1000) 69 (691/1000)
	nonce	2	3	75 (754/1000) 75 (747/1000)
	ids	2	3	77 (771/1000) 66 (662/1000)
	nonce	3	3	75 (374/500)	74 (371/500)
	ids	3	3	70 (350/500)	66 (331/500)
	Table

All our code is at github.com/BlairStanek/gpt-statutes.

Recall that throughout our experiments we use temperature=0 and top_P=1.0, which ensures minimal creativity.

In this section, as throughout this paper, all our calls to GPT-3 have temperature at 0.0 and the top_P at 1.0 to maximize reproducibility and minimize hallucination and wandering off topic.

ACKNOWLEDGMENTS

This work has been supported by the U.S. National Science Foundation under grant No. 2204926. Any opinions, findings, and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the National Science Foundation.


Agathe Pirog
email: agathe.pirog@ofb.gouv.fr

Adrien Tableau
email: adrien.tableau@ofb.gouv.fr

Jean-Marc M Gillier

Jean-François F Maillard
email: francois.maillard@ofb.gouv.fr

Sébastien Reeber
email: reeber@snpn.fr

Justin Potier
email: justin.potier@ofb.gouv.fr

LIFE Oxyura Technique Assessment Report 2020 Techniques implemented and results

Preface

This national assessment is carried out in the framework of:

-Recommendation n°185 of the Berne Convention Permanent Committee, adopted 18 November 2016, on the eradication of the ruddy duck (Oxyura jamaicensis) in the Western Palearctic by 2020 (Council of Europe 2016).

-EU regulation n°1143/2014 concerning the prevention and management of alien invasive species (AIS) introduction and propagation (Council of Europe, 2014).

-The 2015-2025 national plan to eradicate the ruddy duck, validated by the Water and Biodiversity Directorate on 24 June 2016 (Ministère de l'Ecologie du Développement Durable et de l'Energie, 2016).

-The implementation of the EU AIS regulation listing the ruddy duck as a species of concern for the European Union, transcribed into national law by Decree n°2017-595 of 21 Avril 2017 regarding the control and management of the introduction and propagation of certain animal and plant species, and the Decree of 14 February 2018 regarding the prevention of the introduction and propagation of alien invasive animal species in Metropolitan France.

-The national strategy on AIS, Axis II "Operations for the management of gestion species and restoration of ecosystems", Objective 5 "Control widespread alien invasive species", Action 5.3 "Implement national combat plans".

-The EU project LIFE Oxyura, led by the OFB in partnership with the SNPN, which lasts from October 2018 to December 2023. This financial instrument strengthens past actions in order to achieve two objectives: eradication of the population in the wild by 2025 and control and eradication of the captive population by 2030 (European Commission, 2018). This assessment report covers the period from the 1th of January to 31 December 2020. The information it contains is derived from the results of censuses, field surveys and destruction operations carried out by the OFB and the SNPN -Grand-Lieu RNN. Concerning observations outside of Grand-Lieu Lake, this assessment report is also based, to a large extent, on information gathered from the naturalist community, in particular through regular consultation of the ornithological data of the Visionature network.

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier 6 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier 1 Background

A conservation issue for the white-headed duck

The LIFE Oxyura project responds to a conservation issue of a diving duck species, the white-headed duck -Oxyura leucocephala (European Commission, 2018). This species range is subdivided in two geographically isolated populations. One of them, located in the western Mediterranean, is threatened [START_REF] Green | Action plan for the white-headed duck Oxyura leucocephala[END_REF][START_REF] Hughes | International single species action plan for the conservation of the white-headed duck Oxyura leucocephala[END_REF]. Currently only present in Spain and to a lesser extent in Morocco and Tunisia, it used to nest in Italy and France, the last sedentary pair being observed in Corsica in 1966. In 1977, the population was close to extinction, with 22 individuals recorded in southern Spain. Conservation measures have enabled it to recover, with a fairly stable population of around 2,500 individuals. This increase is associated with occasional sightings of isolated birds in France since the 1980s. (Figure 1). The Iberian white-headed duck population faces a new threat since the introduction of a closely-related species, the ruddy duck -Oxyura jamaicensis [START_REF] Gutiérrez-Expósito | Ruddy duck (Oxyura jamaicensis Gmelin, 1789)[END_REF]. This species, native to North America, is a potential competitor for nesting sites and food resources. Although they probably diverged genetically 1-2 million years ago (Muñoz-Fuentes et al., 2007), these two species nevertheless produce fertile hybrids, of which 69 specimens have been observed in Spain since 1991 [START_REF] Gutiérrez-Expósito | Ruddy duck (Oxyura jamaicensis Gmelin, 1789)[END_REF]. This genetic introgression would likely lead to the extinction of the Iberian white-headed duck population in favour of a hybrid population or a new nucleus of ruddy duck. This transition could be rapid due to the aggressive behaviour of male ruddy ducks during the breeding season, favouring their access to female white-headed ducks (Figure 2) [START_REF] Gutiérrez-Expósito | Ruddy duck (Oxyura jamaicensis Gmelin, 1789)[END_REF].

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier In view of conserving the white-headed duck, the Spanish authorities carried out a systematic campaign to destroy ruddy ducks and hybrids in their territory until 2007. Since then, a few occasional sightings have been made1 . Initially introduced in the UK, the threat of the ruddy duck returning to Spain cannot be ruled out as long as the species is still present on the European continent. Under pressure from the white-headed duck conservation issue, the British authorities in association with local ornithological organisations implemented an eradication plan in 2005 2 , which reduced the population from 6000 individuals to 12 individuals in 2019 and 2020. Breeding in the wild was observed every year between 1953 and 2018, with 2019 marking the end of the population renewal (Henderson, pers. comm.). However, two small nuclei have been established on the continent, one in France remaining at around 200 individuals between 2004 and 2018 thanks to the efforts of the French Biodiversity Agency (OFB) and the French National Nature Protection Society (SNPN), and one in the Netherlands of around 80 individuals 3 . In order to prevent natural breeding in France, the LIFE Oxyura project, led by the OFB, was set up in 2018 to support the 2015-2025 national management plan. In particular, it enabled the hiring of four officers specifically to control the ruddy duck throughout France for five years (European Commission, 2018).

The ruddy duck in brief

The ruddy duck is a small duck with a long tail cocked at 45°, which is characteristic of the Oxyura genus (Figure 3). The male's plumage is predominantly red with a black head and white cheeks [START_REF] Johnsgard | Ruddy ducks & other stifftails: Their behavior and biology (animal natural history series[END_REF]. The bill is blue in the breeding season. The female has brown plumage with dark brown on the upper parts of the head and darker-striped light brown on the lower parts. The undertail feathers are white. Regardless of sex, the C-shaped bill of the ruddy duck is the main feature distinguishing it from the white-headed duck, whose bill is S-shaped. Juveniles have a similar profile to the female and can be mistaken for young common pochard. In its native range, the ruddy duck winters on large, shallow bodies of water (coastal bays or lowland lakes). Its preferred nesting sites are smaller bodies of water with extensive riparian vegetation consisting of sedges, reeds and willow saplings. The ruddy duck feeds on aquatic plants, molluscs, worms and insects [START_REF] Sanchez | The diets of the white-headed duck Oxyura leucocephala, ruddy duck O. Jamaicensis and their hybrids from spain[END_REF][START_REF] Woodin | Foods and dietary strategies of prairie-nesting ruddy ducks and redheads[END_REF]. Midge larvae and nymphs (Tendipedidae), specifically Chironomus, are the primary food sources during the breeding season [START_REF] Siegfried | Summer food and feeding of the ruddy duck in Manitoba[END_REF].

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier A ruddy duck population has been established in France since 1974. At the onset of the first cold weather, most of them gather as from late November on Grand-Lieu Lake, in the Loire-Atlantique department, to spend the winter. They form part of a group of about 20,000 wintering ducks of various species in the central area of the lake (Figure 4). This period is favourable for estimating the size of the French population as almost all the birds are visible at this site. The ducks move from the east to the west of the central area of the lake, depending on the wind and the disturbance caused by any predators present or the few professional fishermen (Figure 5). Until mid-February, the birds stay away from the edges of the lake and show fleeing behaviour as soon as they are approached within 500m.

10 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier From mid-February onwards, in good weather, the ruddy ducks show the first signs of breeding behaviour. They then move closer to the banks, particularly in the coves beside the riverine woodland to the north of the wintering area and around the islands to the west (Figure 5). The males acquire their breeding plumage in late winter without forming pairs. Only part of the population remains on Grand-Lieu to breed, the rest gradually moving to smaller ponds, mainly in the north-western quarter of France. The maximum dispersal of the birds is unknown5 . It is therefore not ruled out that there are also connections with the other European populations (notably the Netherlands one).

The first nuptial displays are observed in late winter, when the weather is mild. The male calls throughout the breeding season, which lasts until the end of summer, both day and at night (S. Reeber & A. Laroche, pers. comm.). This call is very characteristic and thus enables the species to be identified. Ruddy ducks make their nests in the vegetation, for example in sedge clumps. They often lay eggs in the nests of other duck species. The first chicks are seen in May at the earliest, but the breeding peak is usually around early July.

The location of the ducks during moulting after breeding has not been clearly identified. On the one hand, the sudden gathering of the population at the wintering site suggests the existence of a potential site where the ducks gather to moult (see analysis in Appendix A). On the other hand, a few individuals located during the same period at very distant sites suggest that the birds moult, camouflaged, close to their nesting site in the vegetation. The moulting behaviour of the native North American population is also poorly identified [START_REF] Baldassarre | Oxyura against Oxyura -eradicate the ruddy duck to save the endangered white-headed duck from extinction -LIFE17-NAT_FR_000542[END_REF]. Males appear to moult as early as August, after breeding. Some have been seen in large, open areas. The main hypothesis for females is that individuals moult on the nesting sites camouflaged in the vegetation.

The winter strategy

The main difficulty in winter on the Grand-Lieu lake is getting close enough to the birds to have them within shooting range. For this purpose, several complementary tools are used. Investigations have shown that chasing them in boats combined with the use of guns is not very effective and above all counterproductive in terms of disturbing other species that find refuge on the lake in winter. The LIFE programme has enabled the construction of 3 hides positioned on the banks as close as possible to the area occupied in winter (Figure 5 and Figure 6). In windless conditions, these shooting stands are ideal for firing high-precision rifles capable of hitting targets at long distances (300m maximum). The area occupied by the birds during winter shooting campaigns is often far from the stands, so the birds must be moved. An unobtrusive boat can be used to gently bring the birds closer to the shooting stands (Figure 7). From mid-February onwards, playbacks 6 combined with plastic decoys positioned near the shooting stands are used to attract the ducks. An alternative to shooting is capturing ducks with a cage trap. A bird can be lured either with food or by using fellow ruddy ducks. The first option is not selective and may disturb other species. It is also not very effective on ruddy ducks due to their diet, so the use of live decoys is favoured. In partnership with Branféré Zoo7 , the Grand-Lieu SNPN and OFB are developing this approach (Figure 8). The English experience shows that cage traps are time-consuming to use. In order to minimize handling time, the cage developed within the LIFE framework is fitted with an automatic trap door that opens at regular intervals. Photographic traps transmit images by telephone before the trap doors are opened. If a ruddy duck is spotted, the team arrives before it opens to capture the bird.

The summer strategy

Some of the birds nest on Grand-Lieu Lake from spring to the end of summer. The suitable areas vary according to water levels and are not very accessible. The most suitable approach is by boat from the centre of the lake. The males are often found near the nests on the edge of the vegetation and are an indicator of the presence of a female and her nest. Another method is to detect birds calling at night using a sound amplifier. Once the birds have been located, they are shot using a high-precision rifle from a boat stabilised by water lilies. Nests are also occasionally located during surveys in the riverine woodland. In this case, trap cages positioned on the nests are effective for capturing females (Figure 9). As in winter, an alternative at Grand-Lieu Lake is to attract the birds. The sexual activity of the ducks can be exploited to attract them to the hides which are still situated in the water in spring. The combination of playbacks and ruddy duck-shaped decoys is the most attractive solution. Attempts to shoot with rifles fitted with silencers and night vision equipment are possible under favourable weather conditions (little wind and clear nights). In addition, the use of a calling cage is a solution considered throughout the summer season given the continuous sexual activity during this period.

At the end of winter, some birds leave Grand-Lieu Lake to nest in smaller ponds with suitable riparian vegetation. For this part of the population, the main challenge is locating them. The main detection method is regular surveying of historical sites of ruddy duck presence and potential nesting sites (Figure 10). Bird watchers and professional ornithologists observations are also used and are particularly useful for detecting birds outside the main range of ruddy duck in France. Once located, officers secure the area and adapt their approach to the birds and the shooting solution according to the local configuration.

14 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

Management of the captive population

The last study focusing on the captive ruddy ducks in France dates from 2014 with a total of 203 birds counted in 26 departments. Among these individuals, 103 were held by recreational breeders in 21 departments, 80 were detained by 10 zoos in nine departments and 20 were detained by six professional breeders in five departments.

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier The current legislation, which is a translation of the European regulation of the 14 February 2018 on the prevention and management of the introduction and spread of invasive alien species on the metropolitan territory, forbid the detention, transportation, sale or acquisition of the ruddy duck. Except for some zoos authorized to display captive ruddy ducks, breeders who previously detained individuals at this date were authorized to keep their birds until their death, if they declare their birds to the prefecture of their residency place before the 31 December 2019 and if they don't do any breeding (at least eggs destroyed).

In order to monitor the captive population evolution, a survey on the breeders identified in 2014 will be conducted in 2022. It will allow to evaluate at which extent is the legislation respected, and thus conclude on the renewal of the captive population and its threat for the wild population.

On the other hand, the new legislation may lead recreational breeders of diving ducks to turn to another species of the Oxyura genus. The lake duck, also called Argentine ruddy duck (Oxyura vittata), native from South America, may also hybridize with other Oxyura species, and notably the white-headed duck. To date, it is difficult to estimate the number of captive lake ducks in France, but if some of them escape into the wild, they are likely to provoke, as the ruddy duck before, the same issues of genetic introgression in the white-headed duck. It is thus paramount to prevent this species introduction into the wild, notably by promoting new legislation or preventive actions.

16

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier At the beginning of 2020, the number of ruddy ducks reached 57 individuals on Lake Grand Lieu8 . The trend is a decrease of 37% compared to the previous winter. We have to go back to 1997 to find similar numbers (Figure 13).

Figure 13: Numbers of the French wintering population with respect to the numbers culled. Numbers before culling in mid-January 2020 in purple, and numbers of the resulting population in mid-January 2021 in pink.

After the 2019-2020 wintering period, ruddy ducks were observed at 41 sites, 4 more than the previous year (Figure 14). This result does not follow the trend in numbers, but can be explained by the irregular presence of birds in eastern France. These birds probably come from the Dutch population or from birds present in Germany. These sites are located in the same areas as in previous years (See details per department in Appendix B).

Detection and culling in 2020

Continuous work is carried out by the LIFE project field officers who survey potential bird habitats throughout the year. Online ornithology platforms9 are also routinely consulted to detect part of the birds. This partnership is essential because the birdwatching community covers the whole of France and allows for the rapid recognition of new areas. Finally, the officers from the OFB's departmental units assist this detection work by transmitting information during their field missions.

Surveys by LIFE officers were divided as follows: 2/3 monitoring of historical sites of ruddy duck presence, 1/3 exploration of sites with high presence potential. This year, LIFE officers made 449 visits to 142 sites (Figure 15). From the observation data derived from the various sources available, the number of birds actually detected without the breeding period is looked for 10 . To make this estimation, we assume that two successive observations of a ruddy duck 11 on the same site correspond to two different individuals if over one month has elapsed between observations. Naturally, major sources of error 12 result in an uncertain estimation. However, this approach remains the best indicator of bird detectability outside the wintering period.

Using the method described above, we estimate that 87 different individuals were detected. Among these, 51 adults of which 27 males and 24 females. The remaining 36 individuals were juveniles or individuals of undetermined age. The wintering population consisted of 57 individuals. The detection rate for adults that wintered in France is thus 89% 13 .

In parallel with detection, the culls performed by SNPN and OFB officers eliminated 59 individuals 14 , of which 49 were adults: 26 males and 23 females. The remaining 10 individuals were juveniles 15 &16 . The culling rate for adults that wintered in France is 86%. The culling rate for detected adults is 96% 17 .

10 A ruddy duck may be observed repeatedly at a site by one or more bird watchers or professional ornithologists 11 Observations are considered to be potentially of one single individual only if the age and sex characteristics are identical 12 There are two sources of error in this estimation of the number of birds detected: i) the time interval between two observations for which individuals are considered different, and ii) individuals that have travelled between several sites and are therefore counted as many times as they are detected 13 A detection rate over 100% is not necessarily an error as it is likely that birds from a population not wintering in France are detected.

14 During the autumn-winter period, juvenile birds are of adult size but with female plumage. Birds culled during this period are systematically sexed and aged by dissection 15 Very few individuals are not retrieved and could be adults 16 A higher number of juveniles culled than detected is possible due to the uncertainty in the estimation of the number detected 17 We cannot estimate the effectiveness for juveniles because we do not have the number of young produced during the year 20 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier An interesting indicator is the proportion of adults culled before they have reproduced. Indeed, by definition, they have not had time to produce young and therefore contribute to population renewal, which has a significant impact on the decline in numbers the following winter. This rate of culling before reproduction amounts to 44%.

In order to make comparisons from year to year, the detailed culling time series is presented in Figure 13. Historical detection and culling efficiencies are presented in Figure 16. The presence of ruddy ducks on a greater number of sites than last year, mainly located in eastern France, suggests movements of wandering ruddy ducks from the Netherlands, Germany, Belgium or from an unidentified French population. This spread was already suspected last year and appears to be confirmed. To date, winter has not been sufficiently harsh to lead to the migration of the population wintering in the Netherlands, but this threat highlights the importance of coordinating the regulation at an European level.

While the numbers have reached a critical size, the presence of juveniles in the population demonstrates that reproduction occurred in the wild 19 . Conditions favourable for reproduction and for the survival of youngs in spring and summer produced a sharp upswing in the French population in the past. Controlling the species will therefore remains necessary as long as even a minima population replacement is observed. A drop in the number of various officers in the field collaborating with LIFE officers could be observed with the decrease in the frequency of operations inherent to low numbers. A decreased involvement of the network of stakeholders due to the objectives having almost been reached would also be detrimental. These aspects demonstrate the importance of communicating effectively on the input of each party and on the long-term objectives of such a project.

An upswing in the impact of the LIFE project

The overall detection rate seems to have achieved 100% this year. This percentage could be overestimated due to choices of calculation methodology 20 , but it nevertheless reflects good detection efficacy.

The period of inactivity due to the lockdown in spring 2020 was not detrimental since rate of adults culled before reproduction was a historical record, as was the total number of adults culled. The culling rate of wintering birds exceeded 80% for the first time in France. Reaching this culling rate when the population size has decreased substantially is very positive for many reasons. It is a priori more difficult to detect birds present in lower densities because the groups are less detectable. This point must however be put into perspective because the decrease in numbers appears to reduce the distance travelled between wintering and nesting grounds, which facilitates surveying and therefore detection. However, officers have to deal with adult birds that are more cautious after culling campaigns in previous years. Over the long term, strong selection pressure caused by this culling level promotes the survival of the wariest discreet bird's or those which best avoid the shootings, therefore enabling the propagation of their associated genes.

Between 2016 and 2019, juveniles represented 40% of the culled individuals while in 2020 and 2021, they represented only 17% and 25% of the culled individuals, respectively. As the population decrease was the sharpest this same year 21 , this change in the population structure of culled individuals reflects a substantial

18 Taking into account pre-LIFE data, we hope to reach a population of less than 10 individuals by the end of 2023 19 No reproduction in the wild has been observed in the United Kingdom since 2019 20 Birds detected in eastern France have often stayed several months on certain sites and may have changed sites several times. Consequently, certain individuals have probably been falsely counted several times. Culling failure in the sector is often due to the size of the sites in question or insufficient safety conditions. 21 75% decrease in population between winter 2020 and winter 2021

22

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier decrease in reproduction in the wild. Several potentially complementary factors may explain this drop in productivity. The good culling rate of adults before reproduction undoubtedly caused a first decrease in the production of young. On Grand-Lieu Lake, the main observers also noted low reproduction of common pochard (A. Caizergues & S. Reeber, pers. comm.)22 . This decrease may thus be linked to the combination of several environmental factors, notably a sharp drop of the water levels in April and May and a small late flood mid-June (J.-M. Gilliers, pers. comm.). Indeed, excessively rapid shifts of water levels during the breeding season often induce clutches failures in diving ducks. On the other hand, the wetland vegetation23 of the Grand-Lieu lake underwent delays in spring as well as an overall decline, which is another adverse factor for nesting success. Finally, the population may also have reached a critical size reducing its ability to efficiently produce numerous offspring. This process, known as the Allee effect [START_REF] Kuparinen | Atlantic cod recovery from the Allee effect zone: Contrasting ecological and evolutionary rescue[END_REF], may be due to the low encounter probability between two parteners when bird density is low for example.

Alternative methods still being used

The reactivity and effectiveness of the field officers ensure that the culling rate of detected is birds good. The margin for improving bird detection and culling effectiveness is therefore low.

Even a small-scale improvement in culling rates before reproduction could have a significant impact on the capacity of the population to recover. It is for this reason that considerable effort has been implemented within the LIFE framework to test different capture methods, notably in winter (Section 2.1.2).The use of shooting stands was part of the first 2020 winter tests and enabled the culling of 2 females. The cage trap design has been completed, but the team is faced with administrative difficulties for obtaining calling females. The avian flu virus episode in France due to the H5N8 virus since autumn 2020 has also prevented the first tests from being carried out. An exemption request is still awaiting a response (05/02/2021). It is important to test this method because without undue delay it could also be applied to the Netherlands where culling by shooting is difficult due to the urbanisation of sites inhabited by the ruddy duck.

A Evidence of moulting in coastal areas?

The moulting season of ducks is a vulnerable period that is propitious for capturing them. If, as with other species, ruddy ducks gather together in this period of the year, it would therefore be efficient to identify the moulting sites. This aspect of ruddy duck ecology is poorly known within its natural range. The low number of sightings during this period shows that the birds are more wary at this time.

Throughout the year, ruddy ducks in France are mainly observed in areas less than 100 km from the coast (Figure 17). This could reflect the potential species' dependence on coastal habitats, notably during the moulting season. Analysis of the positions of the sites where the ruddy ducks are observed shows that the birds sighted during the moulting season24 are more numerous in coastal areas than during the rest of the year. This could be evidence that some of the population moults in coastal areas. Unfortunately, this result is not significant enough to conclude that the moult occurs in coastal areas, and therefore to consider carrying out specific surveys to identify a potential moulting site. A plausible hypothesis concerning the presence of ruddy ducks in this area in autumn is the utilization of the wastewater treatment lagoons of seaside resorts as a food source and refuge area during the hunting season.

Figure 1 :

 1 Figure 1: Evolution of the white-headed duck presence in France

Figure 2 :

 2 Figure 2: Two males in breeding plumage fighting in Mayenne -France; a ruddy duck to the left, a white-headed duck to the right. © Eric Médard

•

  Figure 3: A pair of ruddy ducks in summer. © OFB

  into the biological life cycle of the ruddy duck in France 4

Figure 4 :

 4 Figure 4: Geography of Grand-Lieu, a lowland lake in western France (47°05' 45" N, 1°40' 3" W).The lake has two tributaries, the Boulogne and the Ognon, and flows into the Acheneau which in turn flows into the Loire. The light green areas are water lily beds which disappear in winter. © Geoportail

Figure 5 :

 5 Figure 5: Area where ruddy ducks spend the winter (blue) -Location of hides (orange) -Location of the calling cage (red). © Geoportail

Figure 6 :

 6 Figure 6: One of the three hides built on Grand-Lieu Lake with funds from the LIFE Oxyura project. © OFB 6 An audio system to transmit the sound of the males' nuptial calls

Figure 7 :

 7 Figure 7: Unobtrusive boat with electric motor for gently bringing birds towards the shooting location. © SNPN

Figure 8 :

 8 Figure 8: Calling cage under testing at Branféré Zoo. © Diagram: Jean-Marc Gillier, © Photo: Branféré Zoo

Figure 9 :

 9 Figure 9: Cage trap system on nests, here an example with a common pochard. © Alain Caizergues

Figure 10 :

 10 Figure 10: Boat survey during the breeding season. © Valentin Boniface

Figure 11 :

 11 Figure 11: Ruddy duck numbers detained in captivity per sector

Figure 12 :

 12 Figure 12: Argentine ruddy duck, Oxyura vittata, male

Figure 14 :

 14 Figure 14: Trends in spatial range of ruddy ducks -See the interactive map for the historical details of the sites.

Figure 15 :

 15 Figure 15: Cumulative number of surveys in 2020.

Figure 16 :

 16 Figure 16: Proportion of wintering numbers detected and culled. Please note that these figures are only estimates and therefore uncertain.

Figure 17 :

 17 Figure 17: Distribution of ruddy ducks with regard to the coast during and outside the moulting season.

The last three sightings date from

2011, 2014 and 2020, respectively 2 The plan was set up after a 5-years research phase 3 Some birds have been observed in Belgium, but are not corresponding to an entire population 8 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

The information in Section 2.1.1 is based on observations made by staff from the Grand Lieu SNPN and French Biodiversity Agency, as well as by bird watchers and professional ornithologists.

In the British island context, maximum dispersal appears to be about 150km, but continental European birds are known to have originated from the founder population in England Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

Zoo located in Morbihan and member of the French Association of Zoological Parks (AFDPZ)Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

A few isolated individuals can be observed elsewhere in France, but they do not necessarily belong to the French population and are therefore not taken into account here

http://www.faune-france.org -http://www.observation.org -BDBiodiv Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

190 common pochard females with chicks were identified in spring 2020 compared to 385 on average over 5 previous springs and 480 over 10 years

Mainly aquatic or herbaceous marsh vegetationTechnical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

This period is defined in the analysis as between the 1th of August and the 1th of December, from the evidence provided by[START_REF] Baldassarre | Oxyura against Oxyura -eradicate the ruddy duck to save the endangered white-headed duck from extinction -LIFE17-NAT_FR_000542[END_REF] Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

‡ National Society for the Protection of Nature (SNPN) -Director of the Grand-Lieu national natural Reserve -jean-marc.gillier@ snpn.fr § French Biodiversity Agency (OFB) -Project manager on Alien Invasive ¶ National Society for the Protection of Nature (SNPN) -Grand-Lieu national natural Reserve project manager-sebastien.

B Data by department

For the detail of the historical observation sites, see the interactive map online.