Shayne Longpre

Robert Mahari

Anthony Chen

Naana Obeng-Marnu

Damien Sileo

William Brannon

Niklas Muennighoff

Nathan Khazam

Jad Kabbara

Kartik Perisetla

Alexis Wu

Enrico Shippole

Kurt Bollacker

Tongshuang Wu

Luis Villa

Sandy Pentland

Sara Hooker

The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI

The race to train language models on vast, diverse, and inconsistently documented datasets has raised pressing concerns about the legal and ethical risks for practitioners. To remedy these practices threatening data transparency and understanding, we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use. Our landscape analysis highlights the sharp divides in composition and focus of commercially open vs closed datasets, with closed datasets monopolizing important categories: lower resource languages, more creative tasks, richer topic variety, newer and more synthetic training data. This points to a deepening divide in the types of data that are made available under different license conditions, and heightened implications for jurisdictional legal interpretations of copyright and fair use. We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+. This points to a crisis in misattribution and informed use of the most popular datasets driving many recent breakthroughs. As a contribution to ongoing improvements in dataset transparency and responsible use, we release our entire audit, with an interactive UI, the Data Provenance Explorer, which allows practitioners to trace and filter on data provenance for the most popular open source finetuning data collections: www.dataprovenance.org.

Introduction

The latest wave of language models, both public [START_REF] Hyung Won | Scaling instruction-finetuned language models[END_REF][START_REF] Taori | Stanford alpaca: An instruction-following llama model[END_REF][START_REF] Geng | Koala: A dialogue model for academic research[END_REF] and proprietary [START_REF] Anil | Palm 2 technical report[END_REF][START_REF] Openai | [END_REF][START_REF] Anthropic | Model card and evaluations for claude models[END_REF][START_REF] Yoo | Scalable training of language models using jax pjit and tpuv4[END_REF] attribute their powerful abilities in large part to the diversity and richness of ever larger training datasets, including pre-training corpora, and finetuning datasets compiled by academics [START_REF] Wei | Finetuned language models are zero-shot learners[END_REF][START_REF] Sanh | Multitask prompted training enables zero-shot task generalization[END_REF][START_REF] Muennighoff | Crosslingual generalization through multitask finetuning[END_REF], synthetically generated by models [START_REF] Taori | Stanford alpaca: An instruction-following llama model[END_REF]Wang et al., 2022a), or aggregated by platforms like Hugging Face [START_REF] Lhoest | Datasets: A community library for natural language processing[END_REF]. Recent trends see practitioners combining and re-packaging thousands of datasets and web sources [START_REF] Gao | The pile: An 800gb dataset of diverse text for language modeling[END_REF][START_REF] Penedo | The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only[END_REF]Wang et al., 2022b;Longpre et al., 2023a), but despite some notable documentation efforts [START_REF] Spacerini | Gaia search tool[END_REF][START_REF] Biderman | Datasheet for the pile[END_REF], there are diminishing efforts to attribute, document or understand the raw ingredients into new models [START_REF] Dodge | Documenting large webtext corpora: A case study on the colossal clean crawled corpus[END_REF][START_REF] Bandy | Addressing "documentation debt[END_REF]Bommasani et al., 2023a).

A Crisis in Data Transparency & its Consequences. Increasingly, widely used dataset collections are treated as monolithic, instead of a lineage of data sources, scraped (or model generated), curated, and annotated, often with multiple rounds of re-packaging (and re-licensing) by successive practitioners. The disincentives to acknowledge this lineage stem both from the scale of modern data collection (the effort to properly attribute it), and the increased copyright scrutiny [START_REF] Saveri | Paul tremblay, mona awad vs[END_REF]. Together, these factors have seen fewer Datasheets [START_REF] Gebru | Datasheets for datasets[END_REF], non-disclosure of training sources [START_REF] Openai | [END_REF][START_REF] Anil | Palm 2 technical report[END_REF][START_REF] Touvron | Llama 2: Open foundation and fine-tuned chat models[END_REF], and ultimately a decline in understanding training data (Sambasivan et al., 2021b;Longpre et al., 2023b). This lack of understanding can lead to data leakages between training and test data [START_REF] Elangovan | Memorization vs. generalization : Quantifying data leakage in NLP performance evaluation[END_REF][START_REF] Carlini | Quantifying memorization across neural language models[END_REF], expose personally identifiable information (PII) [START_REF] Bubeck | Sparks of artificial general intelligence: Early experiments with gpt-4[END_REF], present unintended biases or behaviours [START_REF] Welbl | Challenges in detoxifying language models[END_REF][START_REF] Xu | Detoxifying language models risks marginalizing minority voices[END_REF][START_REF] Pozzobon | On the challenges of using black-box apis for toxicity evaluation in research[END_REF], and generally result in lower quality models than anticipated. Beyond these practical challenges, information gaps and documentation debt incur substantial ethical and legal risks. For instance, model releases appear to contradict data terms of use (e.g., WizardCoder [START_REF] Luo | Wizardcoder: Empowering code large language models with evol-instruct[END_REF] licensed for commercial use, while training on commerciallyprohibited OpenAI data), license revisions post-public release (with MPT-StoryTeller [START_REF] Frankle | Tweet by mosaic ml[END_REF]), and even copyright lawsuits (e.g. Stability AI [START_REF] Arstechnica | Stable diffusion copyright lawsuits could be a legal earthquake for ai[END_REF] and OpenAI [START_REF] Saveri | Paul tremblay, mona awad vs[END_REF]). As training models on data is both expensive and largely irreversible, these risks and challenges are not easily remedied. In this work, we term the combination of these indicators, including datasets' sourcing, creation and licensing heritage, as well as its characteristics, Data Provenance.

Unreliable Data Provenance & Licensing. Our work motivates the urgency of tooling that facilitates informed and responsible use of data in both pretraining and finetuning. To empower practitioners to attribute data provenance, we develop a set of tools and standards to trace the data lineage of 44 of the most widely used and adopted text data collections, spanning 1800+ finetuning datasets. We compile and expand relevant metadata with a much richer taxonomy than Hugging Face, Papers with Code, or other aggregators (see Section 2.1). With legal experts, we design a pipeline for tracing dataset provenance, including the original source of the dataset, the associated licenses, creators, and subsequent use.

As a byproduct of our work establishing the Data Provenance of widely used datasets, we are able to characterize the AI data ecosystem/supply chain [START_REF] Cen | Ai supply chains (and why they matter[END_REF]Bommasani et al., 2023c), as well as state of the field for policymakers, researchers and legal experts. Our work points to a crisis in license laundering and informed usage of popular datasets, with systemic problems in sparse, ambiguous, or incorrect license documentation. Notably, we find that 70%+ of licenses for popular datasets on GitHub and Hugging Face are "Unspecified", leaving a substantial information gap that is difficult to navigate in terms of legal responsibility. Second, the licenses that are attached to datasets uploaded to dataset sharing platforms are often inconsistent with the license ascribed by the original author of the dataset-our rigorous re-annotation of licenses finds that 66% of analyzed Hugging Face licenses were in a different use category, often labeled as more permissive than the author's intended license. As a result, much of this data is risky to use (or harmfully misleading) for practitioners who want to respect the data provenance of a work. Our initiative reduces "Unspecified" licenses from 72%+ to 30% and attaches license URLs for under-resourced model developers to more confidently select appropriate data for their needs. To this end, the Data Provenance Initiative supports attribution and responsible AI with the following contributions:

The Initiative to Audit Data Provenance

The Data Provenance Initiative's goal is to audit popular and widely used datasets with large-scale Legal and AI expert-guided annotation. We propose a base set of indicators necessary for tracing dataset lineage and understanding dataset risks (described in Section 2.1). As a first contribution of the initiative, we audit 44 instruction or "alignment" finetuning data collections composed of 1858 individual datasets, selected by experts for their widespread adoption and use in the community. The selected collections and their variants see 100s to 10M+ monthly downloads on Hugging Face, with the datasets within these collections tallying to many more Table 1. The initiative's initial focus on alignment finetuning datasets was decided based on their growing emphasis in the community for improving helpfulness, reducing harmfulness, and orienting models to human values [START_REF] Ouyang | Training language models to follow instructions with human feedback[END_REF]. Some collections have overlapping datasets and examples, but we choose not to deduplicate to preserve the original design choices, that may include different templates, formatting, and filtering. We remove datasets related to common benchmarks like MMLU [START_REF] Hendrycks | Measuring massive multitask language understanding[END_REF] and BigBench [START_REF] Srivastava | Beyond the imitation game: Quantifying and extrapolating the capabilities of language models[END_REF].

Links to Aggregators

Original Sources

Categorize Licenses

Resolve Conflicts

Machine Analysis

Datasets Collections

Name / IDs Source URLs -GitHub -Hugging Face -Semantic Sch.

Data Provenance Explorer (DPExplorer)

Our information audit spans (I) identifier information, bridging metadata from several aggregators, including Hugging Face, GitHub, Papers with Code, Semantic Scholar, and ArXiv, (II) detailed dataset characteristics for a richer understanding of training set composition, and (III) dataset provenance for licensing and attribution. We expand our provenance metadata beyond just licenses, because conversations with practitioners revealed they rely not only on data licenses, but on a specific legal & ethical risk tolerance, parameterized by (a) the lineage of licenses, (b) the data source, (c) the creator's identity, and (d) the precedence of adoption by other developers.

We release our extensive audit, as two tools: (1) a data explorer interface, the Data Provenance Explorer (DPExplorer) for widespread use, and (2) an accompanying repository for practitioners to download the data 1: Alignment tuning collections and their characteristics. Properties of the collections include the numbers of datasets, dialogs, unique tasks, languages, topics, text domains, Huggingface monthly downloads ("Downs"), and the average length of input and target text, by characters. The Source column indicates whether a collection includes human web text ( ), or model generated text ( ). The dialog formats of each collection can be: zero-shot (Z), few-shot (F), chain-of-thought (C), response ranking (R), and multi-turn dialog (M). The Use column indicates whether a collection includes data licensed for commercial use ( ), data with no license ("unspecified": ), data only licensed for non-commercial or academic use ( ). Note that these licenses are self-reported and their applicability is complicated, requiring legal consultation. The "O" column indicates if the collection includes OpenAI model generations, which may or may not affect commercial viability (see Section 4) filtered for license conditions. Practitioners are also able to generate a human-readable, markdown summary, or Data Provenance Card, of the used datasets, and compositional properties for languages, tasks, and licenses (Section 2.3). Modern researchers training on hundreds of datasets often find it onerous to manually curate extensive data cards for these compilations [START_REF] Mitchell | Model cards for model reporting[END_REF][START_REF] Gebru | Datasheets for datasets[END_REF]. We hope this tool will aid in writing the data attribution and composition sections of these documentation efforts, by providing auto-generated, copy-and-pastable dataframe summaries.

Collecting comprehensive metadata for each dataset required leveraging several sources including collection by linking to resources already on the web ( ), human annotation by legal experts ( ), or using GPT-4 to assist in human annotation ( ).

Identifier Information discloses links and connects aggregator identifiers.

1. Dataset Identifiers : The dataset's name, associated paper title, and description of the dataset.

2. Dataset Aggregator Links : A link to each major aggregator, including GitHub, Hugging Face, Papers with Code, Semantic Scholar, and ArXiv allows us to incorporate and compare their crowdsourced metadata.

3. Collection : The name and URL to the data collection of which this dataset is a part.

Dataset Characteristics detail information relevant to understanding data representation/composition, and curating a training set.

1. Languages : Each of the languages represented in the dataset, so developers can easily follow the "Bender Rule" [START_REF] Bender | On achieving and evaluating language-independence in nlp[END_REF]. 4. Text Length Metrics : The minimum, maximum, and mean number of dialog turns per conversation, of characters (agnostic to tokenization/non-whitespace languages, as this introduces biases [START_REF] Petrov | Language model tokenizers introduce unfairness between languages[END_REF]) per user instruction and assistant responses.

Format

: The format and intended use of the data. The options are zero-shot prompts, few-shot prompts, chain-of-thought prompts, multi-turn dialog, and response ranking.

Time of Collection

: The time as which the work was published, which acts as an upper bound estimate of the age of the text.

Dataset Provenance

Licenses

: The license name and URLs associated with the data, using the process described in Section 2.2. We also enable filtering by license use classes, categorized by legal professionals.

Text Source

: The original sources of the text, often Wikipedia, Reddit, or other scraped online/offline sources.

3. Creators : The institutions of the dataset authors, including universities, corporations, and other organizations.

4. Attribution : The attribution information for the authors of the paper associated with the dataset.

Citation & Download Counts :

The citation and Hugging Face download count for the paper and dataset, dated September 2023. This acts as an estimate of community use, and is commonly used as precedence to decide on the risk level for using these datasets .

License Annotation Process

One of our central contributions is to validate the licenses associated with widely used and adopted datasets. This followed a time-intensive human annotation protocol, to collect dataset authors' self-reported licenses, and categorize them according to stated conditions. Note that this protocol reflects best efforts to verify self-reported licenses, and does not constitute legal advice (see Section 4). Additionally, it is important to note that the enforceability of these licenses depends on several factors discussed in Section 4. One especially important assumption in cases where datasets are based on data obtained from other sources is that dataset creators actually have a copyright interest in their dataset. This depends on the data source and how creators modify or augment this data, and requires a case-by-case analysis. However, it appears that most developers operate under the general assumption that they alone own their datasets. Our license annotation workflow follows these steps:

1. Compile all Self-Reported License Information We aggregate all licensing information reported on GitHub, ArXiv, Hugging Face, Papers with Code, and the collection itself (e.g. Super-Natural Instructions, Wang et al. (2022c)).

Search for explicit Data Licenses

The annotator searches for a license specifically given to the dataset (not the accompanying code) by the authors. A license is found if (a) the GitHub repository mentions or links a license in reference to the data, (b) the Hugging Face license label was uploaded by the dataset creator themselves, (c) the paper, Hugging Face, or Papers with Code provide a dataset-specific license link, attributable to the data authors.

3. Identify a License Type A license may fall into a set of common types (e.g. MIT, Apache 2, CC BY SA, etc.), be a "Custom" license, a permission Request Form, or if none was found for the data, Unspecified.

If a dataset has multiple licenses, the annotator will list each of them, according to their types.

Categorize Licenses

From the perspective of a machine learning practitioner, licensing typically is viewed through the lens of how it impacts the model lifecycle-does it impede or allow for training on the data, downstream use conditions, attributing, modifying or re-distributing it. Based on discussions with industry experts, we categorize licenses based on three important features that impact the model lifecycle: is data usage limited to academic or non-commercial purposes (Permitted Use), does the data source need to be attributed (Attribution), and do derivatives of the data need to licensed under the same terms as the original (Share-Alike). If there are multiple licenses for a dataset, its categorization for each feature is the chosen as the strictest across licenses.

Additional Provenance

In practice, legal teams may wish to balance their risk tolerance with more nuanced criteria. For instance, they may be satisfied with using (more permissive) GitHub licenses, even when it is ambiguous whether these apply to the code or the data. They may also wish to include or exclude datasets based on whether these are already widely used in practice, where the original data was sourced from, and if the creator is a competitor. To supplement the above license categories, we also collect all this metadata for fine-grained selection and filtering.

Data Provenance Card-A Data Bibliography

Prior work has stressed the importance of data documentation and attribution [START_REF] Bender | Data statements for natural language processing: Toward mitigating system bias and enabling better science[END_REF]Bommasani et al., 2023a). In particular, Gebru et al. ( 2021)'s Datasheets breaks down documentation into motivation, composition, collection process, processing, uses, maintanence, and distribution. Similarly, Bender and Friedman (2018) ask for curation rationale, language variety, speaker demographic, annotator demographic, speech situation, and text characteristics, among others. However, when models train on many sources of data, even if they are each rigorously documented for each of these fields (rarely the case), it is challenging to cleanly synthesize comprehensive and navigable documentation for the resulting bundle.

To make this process tractable with scale, we propose leveraging Symbolic Attribution, where our tools autogenerate a structured store of the provenance and attribution metadata, similar to a bibliography for data. † We find a long tail of custom licenses, adopted from software for data. 73% of all licenses require attribution, and 33% share-alike, but the most popular are usually commercially permissive.

Our collected schema allows this store to succinctly capture the attribution (links to repositories, aggregator copies, papers, creators), provenance (text/machine sources, licenses), and compositional properties of the data (languages, tasks, text metrics, format, and time). This file of references and metadata, known as a Data Provenance Card enables comprehensive documentation, proposed by prior work, while providing some advantages from its structure. First, the Data Provenance Card can be easily searched, sorted, filtered and analyzed, whereas Datasheets or Statements, designed for individual datasets, are meant to be manually read. Second, developers can efficiently assemble relevant information without losing any detail, by symbolically linking to the original datasets and their documentation. Third, as datasets are continually re-packaged and absorbed into newer and bigger collections, Data Provenance Cards are easily adaptable by simply appending or concatenating them together. Altogether, we hope this tooling enables and promotes the thorough documentation proposed in prior work [START_REF] Bender | Data statements for natural language processing: Toward mitigating system bias and enabling better science[END_REF][START_REF] Gebru | Datasheets for datasets[END_REF][START_REF] Mitchell | Model cards for model reporting[END_REF][START_REF] Pushkarna | Data cards: Purposeful and transparent dataset documentation for responsible ai[END_REF] 3 Empirical Analysis of Data Provenance

Licenses in the Wild

This work constitutes the first extensive study of empirical license use for Natural Language Processing datasets. In this section, we share the insights we have gathered from our large-scale annotation and categorization. There is an important assumption in this section: the OpenAI Terms of Use is a contract, not a license, which prohibits the development of competing models using its outputs. For simplicity, we treat this as a Non-Commercial license in our analysis, though this is disputed for third parties who did not generate the OpenAI data themselves and therefore may not be bound by their terms (see Section 4 for discussion).

Given the intention of OpenAI not to facilitate competitive commercial uses, we follow their categorization for this analysis.

Frequency of license types

Figure 2 shows the distribution of licenses. The most common licenses are CC-BY-SA 4.0 (15.7%), the OpenAI Terms of Use (12.3%), and CC-BY 4.0 (11.6%). While most licenses are common and recognizable, there is a long tail of variants with unique settings, as well as a large set of Custom licenses accounting for 9.6% of all recorded licenses on their own. This wide license diversity illustrates the challenge to startups and less resourced organizations attempting to navigate responsible training data collection, its legality and ethics.

Distribution of Restrictive Licenses

In total, 85% of dataset licenses request attribution, and 30% include a share alike clause. ‡ Datasets which request attribution pose challenges for practitioners who commonly train on hundreds of datasets and either don't cite them at all [START_REF] Openai | [END_REF][START_REF] Anil | Palm 2 technical report[END_REF][START_REF] Touvron | Llama 2: Open foundation and fine-tuned chat models[END_REF] or simply cite an aggregation of data, which often falls short of the license's conditions of attributing the specific repository or paper. Futhermore, "Share alike" clauses poses challenges for practitioners re-packaging data collections usually with multiple conflicting share-alike licenses without a clear way to resolve them (like Longpre et al. (2023a); Wang et al. (2022c) and others in the DPCollection). Frequently, practitioners will over-write share-alike licenses with more restrictive or even less restrictive conditions.

Missing or Unspecified Licenses. Next, we compare our manually reviewed licensing terms, to the licenses for the same datasets, as documented in the aggregators GitHub, HuggingFace, and Papers with Code. Table 2 shows that these crowdsourced aggregators have an extremely high proportion of missing ("Unspecified") licenses, ranging from 69-72%, as compared to our protocol which yields only 30% "Unspecified". The problem with "Unspecified" licenses is that it is unclear whether it is due to a shortcoming of the aggregator or because creators intentionally released them without a license. Consequently, risk-averse developers are forced to avoid many valuable datasets, which they would use otherwise if they were given assurance that there is indeed no license. As part of DPCollection, we manually reassign 46-65% of dataset licenses (depending on the platform), resulting in much higher coverage, thus giving risk-averse developers more confidence and breadth in their dataset utilization.

Incorrectly Specified Licenses.

How does Data Availability Differ by License Use Category?

While non-commercial and academic-only licenses play important roles in protecting data use, their presence can also exclude communities from participating (or competing) in the development of these technologies. In this section, we break down datasets according to their license restrictions and see how they differ. Specifically, we ask: Does complying with licenses dictate systematic differences in resources for commercially-permissive ("open") and non-commercial ("closed") development? And what particular features of data are particularly constrained by non-commercial prohibitions?

We compare datasets by categories of permitted use, according to their licenses: (1) Commercially viable, (2) Non-Commercial/Academic-Only (NC/A-O), or (3) Unspecified license. We group together Non-Commercial and Academic-Only conditions as the distinction will rarely matter for developers. We argue in Section 4 that datasets without any license (Unspecified) have not imposed any conditions, so can often be treated as commercially viable, but this may depend on a developer's risk tolerance and jurisdiction.

Non-Commercial & Academic-Only Licensed Datasets have statistically greater diversity in their representation of tasks, topics, sources, and target text lengths. For each of these features, Table 3 illustrates the mean number per dataset, broken down by license category and entropy to measure the randomness, and thus diversity, of each feature. NC/A-O datasets see greater diversity of tasks, topics, and sources represented in the text than commercial datasets. Figure 4 shows where this diversity comes from. Target Text Lengths are significantly higher for NC/A-O datasets than commercial datasets. Not only do NC/A-O datasets appear more textually and functionally diverse, their length characteristics differ substantially. While Table 3 shows the input text lengths across license categories are similar on average, the target text lengths are significantly higher for NC/A-O datasets (103 vs 677). This breakdown is further illustrated in Figure 5, where we see greater representation of both NC/A-O and synthetic datasets above the 100 target token threshold (y-axis).

The rise of synthetic datasets generated using APIs with non-commercial terms of use may explain the differences in text diversity and length. et al. (2023a) and their variants, all generated in part using commercial APIs, exhibit stronger task and topic diversity than traditional academic datasets, as they cater to longer form generations, by design. This is 2023 has a large spike in license usage, and in NC/A-O licensed data, representing 61%, as compared to 20% on average in prior years. Among the large collection of datasets we trace, we record the date at which they are released, by cross-referencing their associated GitHub, ArXiv, and Hugging Face dates. We find a striking change in the pattern of licensing restrictions. As shown in Figure 3, prior to 2023, no year saw greater than 1/3 of the datasets released as NC/A-O. However, in 2023, which includes many of the most popular and diverse datasets, the NC/A-O rate is 61%. Furthermore, most datasets were unaccompanied by a license prior to 2022 ( 50-80%), as compared to only 12% in 2023. The shift to more license use, and more restrictively conditioned data releases may foretell future challenges to open data, if the trend continues.

< 2 0 1 3 2 0 1 3 2 0 1 4 2 0 1 5 2 0 1 6 2 0 1 7 2 0 1 8 2 0 1 9 2 0 2 0 2 0 2 1 2 0
Commercial datasets have greater language variety, but low-resource language datasets see the least commercial coverage. Table 3 shows that commercial datasets actually have greater diversity of languages than NC/A-O. However, when broken down by language family, as in Figure 3, we see stark differences in permitted use by group. Code language datasets are nearly all commercially viable (78%), because dataset creators can easily filter GitHub for permissively licensed repositories. Interestingly, English, Atlantic-Congo, and Afroasiatic languages also see large permissive representation. However, Turkic, Sino-Tibetan, Japonic, and Indo-European languages see in excess of 35% as non-commercial. Note that while the Indo-European language family contains many high-resource European language families, there is a long tail of lowerresource ones. These NC/A-O language families provide directions for open data practitioners to focus their future efforts.

Broader Characteristics of the Data

In addition to understanding systematic differences in the data by license, there are research questions regarding the overall composition and characteristics of these widely used and adopted datasets. Our compilation of metadata through the DPCollection allows us to map the landscape of data characteristics, and inspect particular features. Note that all these details are also available with interactive visualizations at www.comingsoon.com, for further research and examination. 2022)'s similar analysis for pretraining corpora, we find a stark Western-centric skew in representation. Figure 6 illustrates the coverage per country according to the spoken languages and their representation in DPCollection. We compute a Language Representation score S k for each country k, parametrized by p kl , the percentage of people in country k that speak language l, and w li which is a binary indicator that is 1 if dataset i ∈ D contains language l and 0 otherwise.

S k = l∈L p kl × i∈D w li
The distribution visualized in Figure 6 shows that Asian, African, and South American nations are sparsely covered if at all. Even when nations from the Global South appear to have linguistic representation, according to Section 3.3, the text source and dialect of the language contained in these datasets almost always originates from North American or European creators and web sources (though this is difficult to measure precisely). These observations corroborate similar findings in the geo-diversity of image data in the vision domain [START_REF] Shankar | No classification without representation: Assessing geodiversity issues in open data sets for the developing world[END_REF][START_REF] Vries | Does object recognition work for everyone[END_REF][START_REF] Mahadev | Understanding gender and racial disparities in image recognition models[END_REF]. The resulting models trained on these datasets are likely to have inherent bias, underperforming in critical ways for users of models outside of the west [START_REF] Ahia | The low-resource double bind: An empirical study of pruning for low-resource machine translation[END_REF].

The primary drivers of dataset curation are Academic organizations, supplying 69%, followed by 21% industry labs, and 17% research institutions. These metrics describe the scale of dataset curation contributions, but not the influence each dataset has had on the community. [START_REF] Sanh | Multitask prompted training enables zero-shot task generalization[END_REF]Wang et al., 2022a;Longpre et al., 2023a), but rarely by their actual topics (except [START_REF] Gao | The pile: An 800gb dataset of diverse text for language modeling[END_REF] in their Appendix). Table 4b shows the most popular topics, clustered by category, with their representation across datasets. Like most NLP tasks, much of this text data focuses on communication and language understanding topics, followed closely by general knowledge, routine, sports, and education.

Text datasets are sourced primarily from Online Encyclopedias (22%), Social Media (16%), scraped from the General Web (11%), News (11%), Entertainment web resources (9%). While practitioners document their individual dataset sources in their published papers, this information is unstructured and can be hard to find. As a result, massive collections of widely used datasets rarely compile the distribution of their original sources, instead just citing the papers. After a series of dataset compilations and re-packaging, the original sources are often lost or not well known. By manually scanning approximately 500 academic papers our volunteers annotated the original text sources and compiled them into domain clusters, to permit attribution and analysis, as summarized in Table 4c. Among the individual most adopted sources by the used sources are wikipedia.org (14.9%), undisclosed webpage scrapes (7.0%), reddit (6.2%), and Twitter (4.0%). The least represented domains are Commerce, Reviews, Legal, Academic Papers, and Search Queries, among others.

Legal Discussion

Our empirical analysis highlights that we are in the midst of a crisis in dataset provenance and practitioners are forced to make decisions based on limited information and opaque legal frameworks. While we believe our tooling will enable better transparency about where licenses are in tension, major legal ambiguities remain in data licensing.

Background

Copyright laws aim to encourage written and artistic expression by giving authors exclusive rights to copy, distribute, and adapt their work [START_REF] Patterson | Copyright in 1791: An essay concerning the founers' view of the copyright power granted to congress in article i, section 8, clause 8 of the us constitution[END_REF][START_REF] Burger | The berne convention: Its history and its key role in the future[END_REF]. Open-source licenses first emerged as legal tools to encourage collaboration around software development [START_REF] Krogh | Special issue on open source software development[END_REF]. A range of licenses with different terms and purposes exists including the MIT License, Creative Commons Licenses, and the Apache License, as well as the newer Responsible AI License (RAIL) and AI2 ImpACT Licenses. ¶ The interplay between copyright and licenses can be understood in the following way: copyright automatically gives creators exclusive rights in their works and creators assign these rights to others through license agreements. As we will explore, the open-source licenses that emerged in the last three decades are not always well-equipped to handle the unique characteristics of data, and especially supervised AI training data. Meanwhile, it remains unclear how relevant laws, including those related to copyright and fair use, should be applied to the unique challenges raised by Generative AI and supervised datasets [START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF]. In this section, we highlight some of the key legal challenges and ambiguities related to supervised datasets.

Lifecycle of a dataset

We focus on supervised datasets, which we define as datasets that are created for machine learning (mainly for finetuning and alignment) and where dataset creators made copyrightable contributions in the form of annotations or compilations. A typical supervised dataset is the result of a process that involves several stages of scraping (or machine generation) and annotation by different entities. Generally, raw data is created by people interacting with internet platforms, such as individuals writing articles, sharing artworks, or engaging in online discussion forums. The copyrights to this raw data are normally held by individual users (e.g. Reddit) or by the platform (e.g. Amazon Reviews). Much of this data has been scraped to construct unsupervised datasets for machine learning and this use is commonly justified on the basis of fair use or data mining exceptions to copyright [START_REF] Henderson | Foundation models and fair use[END_REF][START_REF] Benjamin | Artificial intelligence's fair use crisis[END_REF][START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF][START_REF] Samuelson | Generative ai meets copyright[END_REF][START_REF] Mark | Fair learning[END_REF]. However, we find that many common supervised datasets are generated by annotating small samples of scraped raw data using human annotators or large language models. The annotated data is then published with a license agreement. In stark contrast to the copyrighted content that is scraped from the web, supervised datasets were created for the sole purpose of furthering machine learning. The focus of the legal discussion in this section is on how supervised dataset creators can constrain the usage of the copyrightable content they create through licenses and other legal mechanisms. Though we do not address them here, there are several important related questions on the use of copyrighted works to create supervised datasets and on the copyrightability of training datasets.

Surpervised Dataset Example: SQuAD

Rajpurkar et al. ( 2016) present a prototypical supervised dataset on reading comprehension. To create the dataset, the authors take paragraph-long excerpts from 539 popular Wikipedia articles and hire crowd-source workers to generate over 100,000 questions whose answers are contained in the excerpt. For example:

Wikipedia Excerpt In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity. Worker-generated question: What causes precipitation to fall? Answer: Gravity

Here the authors use Wikipedia text as a basis for their data and their dataset contains 100,000 new question-answer pairs based on these texts.

Copyright laws vary by jurisdiction and are subjective, so it is challenging to develop technical safeguards that guarantee compliance. The legal analysis surrounding supervised datasets is complicated by the lack of a uniform global legal framework to address copyright concerns. Different jurisdictions have different and evolving laws. Therefore, the location of model developers and training data creators as well as where and when data was collected may influence the legal analysis. For example, the United States has a fair-use exception to copyright that allows the limited use of copyrighted material under certain circumstances without requiring permission from the rights holders (17 U.S.C. §107) . The EU has no fair-use provision but does have an explicit copyright exception to allow data mining under certain conditions, like obtaining lawful access to the data [START_REF] Margoni | A deeper look into the eu text and data mining exceptions: harmonisation, data ownership, and the future of technology[END_REF]. Meanwhile, datasets themselves generally enjoy copyright protection in the U.S. [START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF] while the E.U. recently created a unique set of rights for dataset creators with the purpose of incentivizing research and development related to databases [START_REF] Derclaye | Sui generis database protection 2.0: judicial and legislative reforms[END_REF]. In addition to differences across jurisdictions, there are also several international agreements related to copyright [START_REF] Ricketson | International Copyright and Neighboring Rights: The Berne Convention and Beyond[END_REF]. Ultimately, it can be challenging to determine which laws should apply to a given machine learning project when the relevant rules vary between the locations where the data was scraped and annotated, where it was downloaded, where the model was trained, and where the model was deployed.

While geographical disparities in regulatory frameworks present one set of challenges, the subjectivity inherent in determining whether copyright infringement has occurred makes it even more challenging to design technical safeguards. For example, in the U.S. part of the copyright infringement analysis depends on whether two works are subjectively similar from the perspective of an ordinary person [START_REF] Mohler | Toward a better understanding of substantial similarity in copyright infringement cases[END_REF][START_REF] Cohen | Masking copyright decisionmaking: The meaninglessness of substantial similarity[END_REF][START_REF] Balganesh | Judging similarity[END_REF]. This is a subjective standard and existing case law may be challenging to extend to generative AI outputs. As a result, while there are technical strategies that can reduce the risk of infringement [START_REF] Henderson | Foundation models and fair use[END_REF][START_REF] Sag | Copyright safety for generative ai[END_REF][START_REF] Vyas | Provable copyright protection for generative models[END_REF], it will be difficult for developers to create technical safeguards that eliminate this risk entirely.

Open legal question regarding copyright and model training. Apart from these jurisdictional and interpretive ambiguities, the process of training a model raises specific copyright questions [START_REF] Epstein | Art and the science of generative ai[END_REF].

Training a model poses several interesting legal questions with respect to copyright and infringement may occur in several ways even before any outputs are generated.

First, the act of creating a training dataset by scraping existing works involves making a digital copy of the underlying data. As the name implies, copyright gives the author of a protected work the exclusive right to make copies of that work. If the scraped data is protected by copyright, then creating training data corpora may raise copyright issues [START_REF] Quang | Does training ai violate copyright law? Berkeley Tech[END_REF]. Second, copyright holders generally have an exclusive right to create derivative works (e.g., translations of a work) but it is not clear whether a trained machine learning model should be considered a derivative of the training data [START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF]. If models are considered to be derivative works, then training a model would be more likely to violate the rights of the training data's copyright holders (Gervais, 2021).

In the U.S., the fair use exception may allow models to be trained on protected works [START_REF] Henderson | Foundation models and fair use[END_REF][START_REF] Mark | Fair learning[END_REF][START_REF] Benjamin | Artificial intelligence's fair use crisis[END_REF][START_REF] Samuelson | Generative ai meets copyright[END_REF]. As these authors explain, the training of machine learning models on copyrighted content may be permissible if the underlying works are significantly "transformed" into model weights, only a small amount of each work in the training data is included in the trained model, model training is designed to only glean generalizable insights from the training data, and the trained model does not have a strong effect on the economic success of the works in the training data. It is important to underscore that, while training a machine learning model itself may be protected by fair use this does not mean that model outputs will not infringe on the copyright of prior works. As the authors above highlight, the application of fair use in the context is still evolving and several of these issues are currently being litigated (see e.g., Andersen v. Stability, Doe v. GitHub, and Tremblay v. OpenAI).

Fair use is less likely to apply when works are created for the sole purpose of training machine learning models as in the case of supervised datasets with copyrightable compositions or annotations. The prior literature on fair use and machine learning tends to focus on copyrighted art or text that was scraped to train a model. These scraped works were not created for the purpose of training machine learning models. By contrast, in this paper, we focus on supervised datasets that were created for the sole purpose of training machine learning models. As underscored by [START_REF] Henderson | Foundation models and fair use[END_REF] and Sobel ( 2017), the fair use analysis depends in part on whether a trained model copies the "expressive purpose" of the original work. While the expressive purpose of a piece of text or art is not to train machine learning models, the purpose of a training dataset is to do just that. As a result, we expect that it is less likely that fair use would apply to the use of curated data. Instead, the creators of these datasets hold a copyright in the dataset ‖ and the terms of the dataset license agreement govern the subsequent use of this data However, it is rare in practice for an LLM to use a single supervised dataset and often multiple datasets are compiled into collections. This further complicates the legal analysis because we find that the license terms of many popular dataset collections are conflicting.

Licenses used for datasets are often ill-suited for this purpose. Beyond the intricate interplay between training data and fair use, the frequently misapplied licensing frameworks for datasets present another set of complications. Most open-source licenses were designed for software, but we find them being attached to datasets. These licenses were intended to be applied to software, not data, which creates challenges [START_REF] Meeker | Beyond open data: The only good license is no license[END_REF]. One of the challenges is that licenses like the Apache and the Creative Commons outline restrictions related to "derivative" or "adapted works" but it remains unclear if a trained model should be classified as a derivative work. This issue is further exacerbated when multiple datasets, each potentially governed by a different open-source license, are amalgamated into collections. If the requirements of the underlying license agreements are irreconcilable, such as different copyleft requirements, this makes it extremely hard for developers to use certain collections while respecting all license terms. To remedy these issues, new licenses are being proposed to address the needs of machine learning datasets such as the BigScience Responsible AI License or an adaptation of the MIT License that requires additional permissions for model training proposed by [START_REF] Ioannidis | Are chatgpt and other similar systems the modern lernaean hydras of ai?[END_REF]. Despite these new proposals, we find that the majority of datasets are licensed under conventional open-source licenses.

‖ Data ownership and data copyright are complex topics [START_REF] Ginsburg | No sweat copyright and other protection of works of information after feist v. rural telephone[END_REF]. We assume that the creators of supervised datasets have some form of copyright in their dataset, though there is often content in these datasets that is owned by third parties. If they satisfy the requirements for copyrightability, dataset creators would have a copyright interest in any new content they create (e.g. annotations). In the U.S., datasets themselves may also be copyrightable as compilations [START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF] while the E.U. provides so-called sui generis rights for databases [START_REF] Derclaye | Sui generis database protection 2.0: judicial and legislative reforms[END_REF].

LLM-generated annotations raise additional legal considerations

We find that approximately 12% of the datasets we audit were annotated using OpenAI. The OpenAI Terms of Use state that outputs from the OpenAI service may not be used to "to develop models that compete with OpenAI" * * . These terms seem to preclude a developer from using OpenAI to generate training data to train a competing LLM. However, it is not clear whether they would also limit the ability of a developer to use OpenAI to create and publish an annotated dataset. On the one hand, publishing such a dataset does not directly compete with OpenAI. On the other hand, it seems foreseeable that such a dataset could enable third parties (who did not themselves use OpenAI) to create competing LLMs. In the U.S., there are several doctrines of secondary or indirect copyright liability aimed to enforce copyright in cases where there is no direct infringement [START_REF] Craig A Grossman | From sony to grokster, the failure of the copyright doctrines of contributory infringement and vicarious liability to resolve the war between content and destructive technologies[END_REF][START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF]. The application of these doctrines depends on many factors, most importantly on whether OpenAI has a copyright interest in its outputs. If these copyright doctrines do not apply, then it is still possible that publishing the dataset constitutes a breach of contract by the dataset developers. While it would be more challenging for OpenAI to pursue a case against third parties, there are myriad other business torts, from unfair competition to misappropriation, that may be relevant to this situation, and which go beyond the scope of this paper [START_REF] Marks | The Law of Business Torts and Unfair Competition: Cases, Materials, and Problems[END_REF]. Time will tell the extent to which OpenAI and other LLM service providers can enforce their terms of use against third parties. However, a prominent researcher at Google has already resigned citing concerns that OpenAI outputs were used to train BARD [START_REF] Victor | Alphabet's google and deepmind pause grudges, join forces to chase openai[END_REF] In light of these legal ambiguities, our tool gives developers the ability to exclude OpenAI-generated datasets.

While legal issues remain ambiguous, practitioners are making decisions on data use and model training.

In the face of these pervasive legal uncertainties, practitioners' decisions regarding data usage are ultimately guided by a blend of factors including the specific licensing terms, the origin of datasets, and the degree of usage of a given dataset by others. Navigating this landscape requires striking a delicate balance between risk mitigation and the need for sufficient resources. This equation, however, varies across regions, applications, and corporate environments, influenced by factors such as competition, risk, and regional legislation. A strategy for partially mitigating these uncertainties is for model providers to indemnify users, as done by Google Cloud [START_REF] Suggs | Protecting customers with generative AI indemnification[END_REF]. However, this may not be feasible for resource-constrained developers and, while it protects end-users, it does not solve the issues faced by model developers or dataset curators.

Our Approach. The fundamental purpose of copyright is to encourage creativity and innovation. As we highlighted in the sections above, the current legal landscape remains ambiguous and this lack of clarity can stifle innovation as developers fear legal repercussions. Through our audit and tooling, we seek to provide important information for practitioners to make informed decisions in an otherwise ambiguous landscape, guided by their own own legal interpretation and risk tolerance. This information includes data license lineages, a categorization of license terms, details on data creators, and the underlying data sources (e.g. web or LLM). In light of ongoing litigation and a lack of legal certainty, we attempted to give developers In creating a repository of data licensing information, we are also taking a step towards encouraging dataset creators to be more thoughtful about the licenses that they select. Dataset creators are well-positioned to understand the appropriate uses of the datasets they publish and licenses can be a tool to communicate these restrictions and to encourage responsible AI development. We further aim to highlight that machine learning practitioners should take dataset license terms seriously, as they may have real impacts on how their models may be used in practice. Ultimately, thoughtful data licensing could be leveraged to promote more responsible, inclusive, and transparent machine learning practices. NOTICE: Collected License Information is NOT Legal Advice. It is important to note we collect selfreported licenses, and categorize them according to our best efforts, as a volunteer research and transparency initiative. The information provided by any of our works and any outputs of the Data Provenance Initiative do not, and are not intended to, constitute legal advice; instead, all information, content, and materials are for general informational purposes only. Readers and users should seek their own legal advice from counsel in their relevant jurisdiction.

Related Work

Data Documentation A long line of work has highlighted the importance of data and its documentation in natural language processing [START_REF] Paullada | Data and its (dis) contents: A survey of dataset development and use in machine learning research[END_REF][START_REF] Rogers | Changing the world by changing the data[END_REF][START_REF] Meyer | The dataset multiplicity problem: How unreliable data impacts predictions[END_REF][START_REF] Gururangan | Annotation artifacts in natural language inference data[END_REF]Muennighoff et al., 2023b). In particular, these works stress the challenges posed by poor documentation to reproducibility, good science, and generally well-understood model behavior (Sambasivan et al., 2021a;[START_REF] Bandy | Addressing "documentation debt[END_REF]Longpre et al., 2023b). Recent work has also explored the importance of documenting AI ecosystems (Bommasani et al., 2023b) and the supply chain from data to models [START_REF] Cen | Ai supply chains (and why they matter[END_REF].

Data Analysis and Exploration

Several notable works have conducted large-scale analyses into data, particularly pretraining text corpora [START_REF] Gao | The pile: An 800gb dataset of diverse text for language modeling[END_REF][START_REF] Dodge | Documenting large webtext corpora: A case study on the colossal clean crawled corpus[END_REF][START_REF] Kreutzer | Quality at a glance: An audit of web-crawled multilingual datasets[END_REF]Laurençon et al., 2022;Scao et al., 2022a,b;[START_REF] Mcmillan-Major | Documenting geographically and contextually diverse data sources: The bigscience catalogue of language data and resources[END_REF]. Other works have investigated the geo-diversity of vision-based datasets [START_REF] Shankar | No classification without representation: Assessing geodiversity issues in open data sets for the developing world[END_REF][START_REF] Vries | Does object recognition work for everyone[END_REF][START_REF] Mahadev | Understanding gender and racial disparities in image recognition models[END_REF]. Different forms of data governance have been proposed to centralize responsibility and documentation over datasets, including for the BigScience project [START_REF] Jernite | Data governance in the age of large-scale data-driven language technology[END_REF] and a Public Data Trust [START_REF] Chan | Reclaiming the digital commons: A public data trust for training data[END_REF]. In terms of finding and visualizing datasets, a few recent tools have been proposed [START_REF] Färber | Datahunter: A system for finding datasets based on scientific problem descriptions[END_REF][START_REF] Viswanathan | Datafinder: Scientific dataset recommendation from natural language descriptions[END_REF].

Transparency and accountability Adjacent to the realm of legality, prior works have strongly advocated and provided frameworks for documentation and audits to increase transparency and accountability in AI systems [START_REF] Miceli | Documenting data production processes: A participatory approach for data work[END_REF][START_REF] Kapoor | Reforms: Reporting standards for machine learning based science[END_REF][START_REF] Inioluwa | Actionable auditing revisited: Investigating the impact of publicly naming biased performance results of commercial ai products[END_REF]. In a manner akin to DPI, which draws upon the collective knowledge of legal and machine learning experts, earlier research has also underscored the significance of interdisciplinary collaborations [START_REF] Hutchinson | Towards accountability for machine learning datasets: Practices from software engineering and infrastructure[END_REF]. Datasheets for datasets [START_REF] Gebru | Datasheets for datasets[END_REF] and Data Statements [START_REF] Bender | Data statements for natural language processing: Toward mitigating system bias and enabling better science[END_REF] both provide structured frameworks for revealing essential metadata such as the motivation behind intended use. [START_REF] Pushkarna | Data cards: Purposeful and transparent dataset documentation for responsible ai[END_REF] expanded on datasheets with "Data Cards" for sources, collection, ethics, and adoption.

Similarly, [START_REF] Mitchell | Model cards for model reporting[END_REF] introduced model cards to benchmark model performance across demographic groups and disclose evaluation procedures. [START_REF] Crisan | Interactive model cards: A humancentered approach to model documentation[END_REF] proposed interactive model card as an alternative mode of documentation and metadata sharing. Complementary to transparency regarding the dataset's creation process, [START_REF] Corry | The problem of zombie datasets: A framework for deprecating datasets[END_REF] provide a framework that guides users on how to navigate datasets as they approach the end of their life-cycle. DPI builds upon the foundational frameworks laid out in these earlier studies, with a specific focus on addressing the licensing aspects of dataset curation. Our goal is to equip users with a comprehensive understanding of the legal risks associated with dataset usage.

Dataset legality

The legality of the datasets used to train large base models has recently received significant attention [START_REF] Sag | The new legal landscape for text mining and machine learning[END_REF][START_REF] Henderson | Foundation models and fair use[END_REF]. The challenge of determining the legality of employing different datasets becomes particularly complex due to the intricate nature of dataset creation processes. [START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF] break up the stages of dataset creation and model generation and assess the relevant copyright questions in the US legal system. These processes often involve multiple licenses and restrictions that can interact in ways that obscure the final legal risk. [START_REF] Soh | Building legal datasets[END_REF] propose a high-level framework for pinpointing the areas within dataset creation and usage where legal analysis is necessary, but do not apply this framework to any existing datasets. [START_REF] Min | Silo language models: Isolating legal risk in a nonparametric datastore[END_REF] demonstrate that refraining from training on copyrighted or highly restricted datasets has a detrimental impact on downstream performance. Their proposed solution involves using a language model trained on "low-risk" text and augmenting it with a data-store containing "high-risk" text which can be modified appropriately as the legal landscape clarifies over time. [START_REF] Lee | Talkin"bout ai generation: Copyright and the generative-ai supply chain[END_REF] DPI enhances these investigations by involving legal experts in the development of a framework for assessing a dataset's "risk" and annotating the "risk" associated with numerous existing high-profile datasets.

B Exact Licenses and Citations

See Table 5 for a summary of the Data Provenance Collection licenses and citations. More comprehensive details are available at https://github.com/Data-Provenance-Initiative/Data-Provenance-Collection.

C Details on Collecting Data Provenance

This data was collected with a mix of manual and automated techniques, leveraging dataset aggregators like GitHub, Hugging Face and Semantic Scholar. Annotating and verifying license information, in particular, required a carefully guided manual workflow, designed with legal practitioners (see Section 2.2). Once these information aggregators were connected, it was possible to synthesize or scrape additional metadata, such as dataset languages, task categories, and time of collection. And for richer details on each dataset, like text topics and source, we used carefully tuned prompts on language models inspecting each dataset.

Automated Annotation Methods

Based on the manually retrieved pages, we automatically extract Licenses from HuggingFace configurations and GitHub pages. We leverage the Semantic Scholar public API [START_REF] Michael Kinney | [END_REF] to retrieve the released date and current citation counts associated to academic publications. Additionally, we compute a series of other helpful, but often overlooked data properties such as text metrics (the min/mean/max for input and target lengths), and dialog turns. We elected to measure sequence length in characters rather than word tokens, for fairer treatment across language and script given well-known differences in tokenizer performance across different languages [START_REF] Petrov | Language model tokenizers introduce unfairness between languages[END_REF].

API Annotation Methods

While Task Categories have become the established measurement of data diversity in recent instruction tuning work [START_REF] Sanh | Multitask prompted training enables zero-shot task generalization[END_REF]Wang et al., 2022a), there are so many other rich features describing data diversity and representation. To augment this, we use OpenAI's GPT-4 API to help annotate for text topics. We randomly sampled 100 examples per dataset and carefully prompt GPT-4 to suggest up to 10 topics discussed in the text.

To annotate for the original data sources, AI experts (PhD students and postdocs) reviewed the papers and filled out the original text sources, whether machines or template-generation were used for synthetic generation, and whether human annotators were used. GPT-4 was used as an in-context retriever on the dataset's ArXiv paper to extract snippets that the experts may have missed. We split the ArXiv paper into 4000 characters chunks and prompt the API to return a json list of any mentions of the dataset source, e.g. from scraping, synthetic or manual generation.

  sit amet, consectetur adipiscing elit. Nulla ipsum mauris, vulputate quis arcu nec, malesuada rhoncus diam. Vestibulum molestie velit ac turpis ullamcorper, a fermentum augue finibus. Donec gravida euismod nisl ac viverra. Sed vitae mi ut risus condimentum malesuada. Proin at posuere leo, ac fermentum purus.Donec placerat nec ipsum quis malesuada. Donec sit amet diam vestibulum, dapibus nulla placerat, fermentum dolor.Aenean lacinia ut felis ut tristique. Praesent feugiat dolor in leo

Figure 1 :

 1 Figure 1: The DPCollection annotation pipeline uses human and human-assisted procedures to annotate dataset Identifiers , Characteristics , and Provenance . The Data Lifecycle is traced, from the original sources (web scrapes, human or synthetic text), to curated datasets and packaged collections. Information is collected at each stage, not just the last. The License Annotation Procedure is described in Section 2.2.

  2. Task Categories: The 20+ task categories represented in the instructions, such as Question Answering, Translation, Program Synthesis, Toxicity Identification, Creative Writing, and Roleplaying. 3. Text Topics : An automated annotation of the topics discussed in the datasets, with GPT-4 labeling a sample of 100 examples for up to 10 covered topics.

Figure 2 :

 2 Figure2: We plot the distributions of licenses used in the DPCollection, a popular sample of the major supervised NLP datasets. We find a long tail of custom licenses, adopted from software for data. 73% of all licenses require attribution, and 33% share-alike, but the most popular are usually commercially permissive.

Figure 3 :

 3 Figure 3: The distribution of datasets in each time of collection (top) and language family (bottom) category, with total count above the bars, and the portion in each license use category shown via bar color. Red is Non-commerical/Academic-Only, Yellow is Unspecified, and Blue is Commercial. Lower resource languages, and datasets created in 2023 see a spike in non-commercial licensing.

Figure 4 :Figure 5 :

 45 Figure 4: The distribution of datasets in each Domain Source (top) and task (bottom) category, with total count above the bars, and the portion in each license use category shown via bar color. Red is Noncommerical/Academic-Only, Yellow is Unspecified, and Blue is Commercial. Creative, reasoning, and long-form generation tasks, as well as datasets sourced from models, exams, and the general web see the highest rate of non-commercial licensing.

Figure 6 :

 6 Figure 6: A global heatmap measuring how well each country's spoken languages are represented by the composition of natural language datasets in DPCollection, as calculated by Section 3.3. English-speaking and Western European nations are best represented, while the Global South sees limited coverage.

Table

	Collection			Property Counts			Text Lens	Dataset Types
	Airoboros	1	17k	5	2	10	1	1k	347	1k	✔		✔
	Alpaca	1	52k	8	1	10	1	100k 505 270	✔		✔
	Anthropic HH	1	161k	3	1	10	1	82k	69 311	✔
	BaizeChat	4	210k	12	2	37	3	<1k	74 234	✔		✔
	BookSum	1	7k	4	1	10	1	<1k 14k	2k	✔
	CamelAI Sci.	3	60k	2	1	29	1	<1k 190	2k	✔		✔
	CoT Coll.	6	2,183k 12	7	29	1	<1k 728 265	✔		✔
	Code Alpaca	1	20k	3	2	10	1	5k	97 196	✔		✔
	CommitPackFT	277	702k	1 278 751	1	4k	645 784	✔
	Dolly 15k	7	15k	5	1	38	1	10,116k 423 357	✔
	Evol-Instr.	2	213k	11	2	17	1	2k	570	2k	✔		✔
	Flan Collection	450	9,813k 19 39	1k	23	19k	2k 128	✔ ✔ ✔		✔
	GPT-4-Alpaca	1	55k	7	1	10	1	1k	130 543	✔		✔
	GPT4AllJ	7	809k	10	1	56	1	<1k 883	1k	✔		✔
	GPTeacher	4	103k	8	2	33	1	<1k 227 360	✔		✔
	Gorilla	1	15k	4	2	10	2	<1k 119	76	✔		✔
	HC3	12	37k	6	2	102	6	2k	119 652	✔		✔
	Joke Expl.	1	<1k	2	1	10	1	<1k	96 547	✔
	LAION OIG	26	9,211k 12	1	171	11	<1k 343 595		✔	✔
	LIMA	5	1k	10	2	43	6	3k	228	3k	✔ ✔	✔
	Longform	7	23k	11	1	63	4	3k	810	2k	✔		✔
	OpAsst OctoPack	1	10k	3	20	10	1	<1k 118 884		✔
	OpenAI Summ.	1	93k	5	1	10	1	14k	1k 134	✔		✔
	OpenAssistant	19	10k	4	20	99	1	14k 118 711		✔
	OpenOrca	4	4,234k 11	1	30	23	28k	1k 492	✔		✔
	SHP	18	349k	6	2	151	1	4k	824 496	✔
	Self-Instruct	1	83k	6	2	10	1	3k	134 104	✔		✔
	ShareGPT	1	77k	9	1	10	2	<1k 303	1k		✔	✔
	StackExchange	1	10,607k 1	2	10	1	<1k	1k 901	✔
	StarCoder	1	<1k	1	2	10	1	<1k 195 504	✔
	Tasksource Ins.	288	3,397k 13	1	582	20	<1k 518	18	✔		✔
	Tasksource ST	229	338k	15	1	477	18	<1k	3k	6	✔		✔
	TinyStories	1	14k	4	1	10	1	12k 517 194k	✔		✔
	Tool-Llama	1	37k	2	2	10	1	-	7k	1k		✔	✔
	UltraChat	1	1,468k	7	1	11	2	2k	282	1k	✔	✔	✔
	Unnatural Instr.	1	66k	4	1	10	1	<1k 331	68	✔		✔
	WebGPT	5	20k	4	1	35	3	1k	737 743	✔		✔
	xP3x	467 886,240k 5 245 151	14	<1k 589 441	✔

Datasets Dialogs Tasks Langs Topics Domains Downs Inpt Tgt Source Z F C R M Use O

Correct license License according to aggregators (Agg.)

	License	Count Agg. Comm.	Unspec. Non-Comm. Acad.-Only
	Commercial	856 (46.1%)	349 176	507 677	0 1	0 2
			313	520	1	22
	Unspecified	570 (30.7%)	112 164	458 395	0 6	0 5
			31	523	1	15
	Non-Commercial	352 (19.0%)	49 113	303 152	0 80	0 7
			2	191	157	2
	Academic-Only	80 (4.3%)	9 9	71 65	0 2	0 4
			5	65	2	8
	Total	1858 (100%)	519 (28%) 1339 (72%) 462 (25%) 1289 (69%)	0 (0%) 89 (5%)	0 (0%) 18 (1%)
			351 (19%) 1299 (70%) 161 (9%)	47 (3%)

Table 2 : The distribution of license use categories shows our licenses have far fewer "Unspecified" omis- sions than

 2

GitHub ( , 72%), Hugging Face ( , 69%), and Papers with Code ( , 70%), categorizing license more confidently into commercial or non-commercial categories. GitHub, Hugging Face, and Papers with Code match our licenses (green regions) 43%, 35%, and 54% of the time, respectively, and suggest incorrect licenses that are too permissive 29%, 27%, and 16% of the time.

Table 3 :

 3 Table 2 also finds real licenses as assigned by us are frequently stricter than the ones by aggregators. GitHub, Hugging Face and Papers with Code each label license use cases too permissively in 29%, 27%, and 16% of cases respectively. Our inspection suggests this is due to contributors on these platforms often mistaking licenses attached to code in GitHub repositories for licenses attached to data. The mean number of features (e.g. tasks or languages) per dataset, and the mean entropy of the distribution, representing the diversity of categories. Non

	Metrics	Commercial	Unspecified	NC / A-O
		Mean	Entropy Mean Entropy	Mean	Entropy
	Tasks	1.7±0.1	0.61	1.6±0.1	0.53	3.4±0.2	0.69
	Languages	1.3±0.0	0.52	1.2±0.0	0.16	1.1±0.0	0.45
	Topics	8.2±0.2	0.70	9.2±0.1	0.75	9.1±0.2	0.77
	Sources	1.6±0.1	0.67	1.8±0.1	0.72	4.2±1.3	0.78
	Input Text Lengths 1043.4±151.9 6.37 860.2±67.7 6.66	950.3±112.9	6.46
	Target Text Lengths 102.7±14.6	4.39	90.5±14.3	4.09 1580.7±965.6 5.37
	Synthetic	12.8%±2.1	-	13.6%±1.7	-	45.5%±3.4	-

-Commercial / Academic-Only datasets have consistently and statistically higher task, topic, and source variety than Commercial datasets.

  We use Normalized Shannon Entropy for discrete features, and Differential Entropy for continuous features, which are both measures of randomness.

  The most NC/A-O task categories include Brainstorming, Explanation, Logic & Math, as well as Creativity and Creative Writing. In comparison, the most commercially viable task categories are Short Text Generation, Translation, and Classification. Similarly, among Source Domains, Governments and Search Queries are largely viable for commercial (and unspecified) purposes, whereas General Web, Exams, and Model-generated sources are among the most restrictive.

Table 4 :

 4

A summary of the distribution of Creators, Topics, and Source Domains across all 1800+ datasets. Datasts can have multiple creators, text topics, and sources.

Text datasets focus on topics of Language & Linguistics, General Knowledge, Logic, & Lifestyle.

  Prior data collection work focuses predominantly on describing datasets by their task compositions

† Auto-generated at https://github.com/Data-Provenance-Initiative/Data-Provenance-Collection

‡ "Share alike" is a copyright term meaning adaptations or copies of a work are required to be released under the same license as the original.

§ https://openai.com/policies/terms-of-use

¶ See https://www.licenses.ai/blog/2023/3/3/ai-pubs-rail-licenses and https://allenai.org/impact-license# licenses. These license templates propose terms aimed at encouraging more responsible or risk-based machine learning practices, see also[START_REF] Contractor | Behavioral use licensing for responsible ai[END_REF]

* * https://openai.com/policies/terms-of-use

Acknowledgements

We would like to thank Katherine Lee, A. Feder Cooper, Peter Henderson, Aviya Skowron and Stella Biderman for valuable comments and feedback.

Appendix

A Contributors

Here we enumerate the author contributions. We would like to emphasize that all authors contributed crucial elements to this project, and Core Contributors in particular are recognized with hands on service to the design and construction of Data Provenance's first implementation.

• Shayne Longpre Core Contributor. Primary designer and coder of the repository and explorer interface. Led audit implementation, and analysis, as well as the manual annotation process.

• Robert Mahari Core Contributor. Led the legal analysis, and licensing annotation design.

• Anthony Chen Core Contributor. Led automatic inferencing of dataset text metrics, topics, and task category annotations. Supported writing, analysis, and code testing.

• Naana Obeng-Marnu Core contributor. Led visualization design, particularly interactive visualizations in the Data Provenance Explorer.

• Damien Sileo Core contributor. Led data aggregator linking, and metadata scraping. Supported writing, analysis, source annotation and adding datasets.

• William Brannon Core contributor. Added 8 data collections, supported writing and data analysis.

• Niklas Muennighoff Core contributor. Added several large data collections, supported writing, analysis, visualization, and source annotations.

• Nathan Khazam Core contributor. Led licensing annotation effort and supported adding datasets along with testing.

• Jad Kabbara Core contributor and advisor. Led text source annotation effort and supported with framing, writing and analysis.

• Kartik Perisetla Core contributor. Added several datasets, supported writing, analysis, and dataset preparation for Hugging Face.

• Xinyi (Alexis) Wu Core contributor. Added several datasets, testing, and supported automatic metadata collection.

• Enrico Shippole Core contributor. Led final dataset preparation for Hugging Face upload and testing.

• Kurt Bollacker Advisor on project design and framing.

• Tongshuang Wu Advisor, particularly on data analysis and visualizations. Supported writing and Data Provenance Explorer design.

• Luis Villa Advisor on data copyright and licensing, and supporting writing in the legal discussion section.

• Sandy Pentland Advisor on general project design and framing.

• Sara Hooker Advisor on general project design and framing, as well as supporting writing, analysis, and directing experiments.

Cite


Damien Sileo
email: damien.sileo@inria.fr

Marie-Francine Moens

Probing neural language models for understanding of words of estimative probability

. In this study, our focus is to gauge the competency of neural language processing models in accurately capturing the consensual probability level associated with each WEP. Our first approach is utilizing the UNLI dataset (Chen et al., 2020), which links premises and hypotheses with their perceived joint probability p. From this, we craft prompts in the form: "[PREMISE]. [WEP], [HYPOTHESIS]." This allows us to evaluate whether language models can predict if the consensual probability level of a WEP aligns closely with p. In our second approach, we develop a dataset based on WEP-focused probabilistic reasoning to assess if language models can logically process WEP compositions. For example, given the prompt "[EVENTA] is likely. [EVENTB] is impossible.", a wellfunctioning language model should not conclude that [EVENTA&B] is likely. Through our study, we observe that both tasks present challenges to out-of-the-box English language models. However, we also demonstrate that fine-tuning these models can lead to significant and transferable improvements.

Introduction

Expression of uncertainty is an important part of communication. Formal statistics are the rigorous way to quantify uncertainty but do not fit all communication styles. Words of estimative probability (WEP) such as maybe and believe are adverbs or verbs that are informal alternatives. [START_REF] Kent | Words of estimative probability[END_REF] noted the importance of clarifying WEP meaning for intelligence analysis in the Central Intelligence Agency, and provided guidelines for mapping WEP to numerical probabilities. Several studies then measured the human perceptions of probability words and discovered some agreement with [START_REF] Kent | Words of estimative probability[END_REF]'s guidelines. In this work, we use the scale derived from a survey [START_REF] Fagen-Ulmschneider | Perception of probability words[END_REF], which is the largest and most recent WEP perception survey available. 123 participants were asked to label WEP with numerical probabilities. We use the median of the participant answers to assign a consensual value to each WEP. Associated probabilities for the 19 WEP we use are available in Appendix A, table 2.

Here, we assess whether neural language models learn the consensual probability judgment of WEP from language modeling pretraining. We develop datasets and a methodology to probe neural language model understanding of WEP. The first dataset leverages previously annotated probability scores between a premise and a hypothesis, in order to measure a language model's ability to capture the agreement between numerical probabilities and WEP-expressed probabilities. The second dataset is based on compositions of facts with WEP-expressed probabilities, and measures verbal probabilistic reasoning in language models. Our contributions are as follows: (i) two datasets and methods to measure understanding of WEP; and (ii) evaluation of the ability of neural language models (GPT2, RoBERTa-trained on MNLI) to tackle WEP-related problems, showing that offthe-shelf models are very little influenced by them, even though fine-tuning on our constructed datasets quickly leads to high accuracies. The code and generated datasets are publicly available1

Related work

Our work probes a particular aspect of language understanding. We do not analyze the inside of the models [START_REF] Rogers | A primer in BERTology: What we know about how BERT works[END_REF]. We focus on the models' ability to perform controlled tasks [START_REF] Naik | Stress test evaluation for natural language inference[END_REF]Richardson et al., 2020) involving WEP. WEP were studied in the context of intelligence analysis and linguistics, our work is the first to look at them through natural language processing (NLP) models. Our study also pertains to NLP analyses of logical reasoning and probability problems, and to uncertainty in natural language inference tasks.

Linguistics study of WEP Kent (1964)'s seminal work was the first to link WEP and numerical probability estimates, with intelligence analysis motivations (Dhami and Mandel, 2021) and a prescriptivist approach. This inspired further quantifications of human perceptions of WEP, in the context of medical reports (O 'Brien, 1989;[START_REF] Douglas | Words representing numeric probabilities in medical writing are ambiguous and misinterpreted[END_REF] and weather reports [START_REF] Emily D Lenhardt | How likely is that chance of thunderstorms? a study of how national weather service forecast offices use words of estimative probability and what they mean to the public[END_REF]. [START_REF] Fagen-Ulmschneider | Perception of probability words[END_REF] proposed the largest survey up to date with 123 participants about generaldomain WEP perception.

Logical and probabilistic reasoning Another strand of work probes NLP text encoders capabilities, notably reasoning abilities. [START_REF] Weston | Towards ai-complete question answering: A set of prerequisite toy tasks[END_REF] probed understanding of specific problems like negation, spatial and temporal reasoning with the bAbI dataset. Richardson et al. (2020) probe understanding of first-order logic reasoning, [START_REF] Sileo | Mindgames: Targeting theory of mind in large language models with dynamic epistemic modal logic[END_REF] probe epistemic logic reasoning. Our work is the first to address probabilistic logic, alongside [START_REF] Dries | Solving probability problems in natural language[END_REF]; [START_REF] Suster | Mapping probability word problems to executable representations[END_REF] who construct a dataset of natural language probability problems, e.g., "A bag has 4 white and 8 blue marbles. You pull out one marble and it is blue. You pull out another marble, what is the probability of it being white?". They also rely on the ProbLog solver [START_REF] De Raedt | Problog: A probabilistic prolog and its application in link discovery[END_REF], but focus on numeric probability problems. By contrast, our work targets WEP, and textual probabilistic logical reasoning.

Natural language inference, uncertainty, modality, evidentiality Uncertainty was also studied in the context of natural language inference tasks. [START_REF] Zhou | Distributed nli: Learning to predict human opinion distributions for language reasoning[END_REF] study the disagreement across annotators when labeling entailment relationships. [START_REF] Zhang | Ordinal common-sense inference[END_REF] annotate graded entailment with 5 probability levels, and the UNLI dataset [START_REF] Chen | Uncertain natural language inference[END_REF]) go further by annotating numerical probabilities. Our work also pertains to the study of modality [START_REF] Palmer | Words and worlds; on the linguistic analysis of modality[END_REF][START_REF] Saurí | Annotating and recognizing event modality in text[END_REF] and more particularly evidentiality [START_REF] Su | Evidentiality for text trustworthiness detection[END_REF], but where previous work focused on WEP.

3 Probing WEP understanding

Verbalization and distractor generation

Our goal is to measure the understanding of WEP. One requirement of WEP understanding is capturing the consensual probability level. To test that, we use contexts (PREMISE) paired with a conclusions (HYPOTHESIS). The likelihood of a conclusion, p, depends on the associated context. One example from UNLI [START_REF] Chen | Uncertain natural language inference[END_REF], which annotates that, is (A man in a white shirt taking a picture , A man takes a picture , 1.0).

We convert a triplet (PREMISE, HYPOTHESIS, p) to the following verbalization:

PREMISE. T p (HYPOTHESIS).

(

) 1
where T p is a text template assigned to the probability p. To select a template, we find the WEP whose associated median probability (see table 2) is the closest to p. We then use handcrafted templates to construct a modal sentence from the selected WEP and the hypothesis, e.g., "It is certain that a man takes a picture". Table 3 in appendix B displays the templates that we associate with each WEP.

We also generate an invalid verbalization by randomly selecting an incorrect WEP (a WEP whose consensual probability differs from p by at least 40%)2 , e.g., It is unlikely that a man takes a picture. We hypothesize that language models and entailment recognition models should give a higher score (respectively likelihood and entailment probability) to the correct valid verbalization than to the invalid verbalization of p.

WEP-UNLI: probability/WEP matching

The UNLI dataset annotates (PREMISE, HYPOTH-ESIS) pairs from the SNLI dataset [START_REF] Samuel | A large annotated corpus for learning natural language inference[END_REF] with joint probability scores p, totaling 55k training examples, 3k/3k validation/test examples. We use these examples to generate WEPunderstanding dataset with verbalization validity prediction as shown in the previous subsection.

% Round 1 template

Sampled round 1 (premise) p1::factA.

There is a very good chance that Bernhard is a swan. p2::factB.

It is almost certain that Greg is gray.

p3::factC.

There is a better than even chance that Sandra left the apple.

% Round 2 template

Sampled round 2 (premise, continued) p4::factX:-op1(fact1, fact2).

Chances are slight that if Bernhard is a swan, or Sandra left the apple, then sheep are afraid of mice.

p5::factY:-op2(fact3, fact4).

It is improbable that if Greg is gray, and Bernhard is a swan, then Lily is a rhino. p6::factZ:-op3(fact5, fact6).

There is a very good chance that if Greg is gray, and Sandra left the apple, then Sumit is thirsty.

% Round 3 template

Sampled hypothesis hypothesis:-op4(fact7, fact8). Figure 1: WEP-reasoning task constructions, with 2 hops. We sample randomly concrete facts f act i and probabilities p i then build modal sentences with verbalization templates. We randomly sample logical operators to compose the modal sentences from the previous rounds to construct a premise, then a hypothesis, and we use a probabilistic soft logic solver to compute the hypothesis probability. We then correctly and incorrectly verbalize this probability. This process generates data for the task of probability verbalization validity. 1 hop reasoning skips the second round: fact7 and fact8 are sampled from {factA,factB,factC}

WEP-Reasoning: WEP compositions

Here, our goal is to assess models' ability to reason over combinations of probabilistic statements. We construct synthetic (PREMISE, HYPOTHESIS, p) examples from random factoids extracted from the bAbI dataset [START_REF] Weston | Towards ai-complete question answering: A set of prerequisite toy tasks[END_REF]. Figure 1 illustrates the construction of WEP-reasoning examples:

We randomly sample initial facts and associated probability levels, and we verbalize them with the previously mentioned templates from Table 3 (Round 1). We further compose them with randomly sampled logical operators (and, or, xor). We then generate a hypothesis with logical combinations of the previous round. Finally, we feed the constructed premise and hypothesis to a probabilistic soft reasoning engine in order to derive the likelihood of the hypothesis given the premise. We rely on the ProbLog [START_REF] De Raedt | Problog: A probabilistic prolog and its application in link discovery[END_REF] reasoner which implements [START_REF] Dantsin | Probabilistic logic programs and their semantics[END_REF] semantics.

To evaluate different complexities of reasoning, we propose two variants: 2-hop reasoning, where facts in Round 2 combine facts from Round 1, and the final hypothesis combines facts from Round 2. and 1-hop reasoning where facts from the hypothesis combine Round 1 facts (Round 2 is skipped).

Since we want to sample more than two facts and we cannot a priori use text from the UNLI dataset, because UNLI only provides entailment likelihood for specific pairs. Combining several sentences could cause unaccounted interference. Therefore, we sample subject/verb/object factoids from the bAbI [START_REF] Weston | Towards ai-complete question answering: A set of prerequisite toy tasks[END_REF] datasets instead, which is built with handwritten arbitrary factoids such as John went to the kitchen. To sample multiple factoids, we prevent any overlap of concepts (verb, subject, object) between any pair of facts to make the facts independent of one another.

We sample probability levels from the list of medians of all WEP to prevent sampling the levels that too distant from a known WEP. When we assign a WEP to a probability level, we assume that the correct semantics is the consensual one, but humans differs slightly from this consensus. Still, when adding random perturbations of 20% to sampled p 1...6 , the hypothesis probability is perturbed by less than 40% for 98% of examples.

We generate 5k examples using the template depicted in Figure 1, and use 10%/10% of the data for the validation/test splits. Appendix C shows the distribution of correct WEP for each dataset.

Experiments

We conduct verbalization validity prediction (binary classification task of WEP correctness detection between two candidates) under two settings. Table 1: Test accuracy percentage of different models over the 3 WEP-understanding tasks. The last three rows display the accuracy when fine-tuning on each task, and transferability of the fine-tuned model outside the diagonal.

Zero-shot models

We use off-the-shelf language models to assign likelihood scores to a context and its conclusion. We evaluate the rate at which valid verbalization is scored higher than invalid verbalization. We refine the scores by also considering the average likelihood per token [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Schick | Exploiting cloze-questions for few-shot text classification and natural language inference[END_REF] and calibrated scores [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Zhao | Calibrate before use: Improving few-shot performance of language models[END_REF] where we divide the score of a PREMISE. T p (HYPOTHESIS). by the score of T p (HYPOTHESIS). We evaluate the normalized, length-normalized, and calibrated likelihood on the validation sets of each dataset and select the most accurate method for each dataset and model. We also consider a pretrained natural language inference model, which is trained to predict entailment scores between a context and a conclusion.

GPT2

We use the pretrained GPT2 base version with 127M parameters [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], which is a causal language model trained to estimate text likelihood. We concatenate the premise and hypothesis and compute their likelihood as a plausibility score.

RoBERTa We also use the pretrained RoBERTa base model with 123M parameters [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] to score the masked language modeling likelihood of the premise/hypothesis pair.

RoBERTa-MNLI

We fine-tune RoBERTa on the MNLI entailment detection dataset [START_REF] Williams | A broad-coverage challenge corpus for sentence understanding through inference[END_REF] with standard hyperparameters (see the following subsection).

Human baseline To establish human baseline performance on the constructed dataset, we had two NLP researchers annotate 100 examples randomly sampled from the test set of each dataset, with a multiple-choice question answering setting.

Overall inter-annotator agreement is relatively high, with a Fleiss's κ of 0.70/0.68/0.71 for WEP Reasoning 1 hop, 2 hops and WEP-UNLI respectively.

Fine-tuning and transfer across probes

We fine-tune RoBERTa-base models on our datasets, using standard [START_REF] Mosbach | On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines[END_REF] hyperparameters3 (3 epochs, sequence length of 256, learning rate of 2.10 -5 batch size of 16. We use length-normalization with GPT2 likelihood and calibration with RoBERTa likelihood as they worked best on the validation sets.). We use a multiplechoice-question answering setup (we predict logit scores for the valid and invalid verbalization, combine their score with a softmax, then optimize the likelihood of the valid verbalization). The same format is applied to all tasks, so we can also study the transfer of capacities acquired during fine-tuning of each probe, for instance, between probability matching and compositional reasoning.

Results and discussion

Table 1 shows the results of our experiments. The very low accuracy of causal and masked language models (first two rows) demonstrates how challenging the WEP-understanding tasks are.

RoBERTa fine-tuned on MNLI dataset performs better than chance for WEP-UNLI. MNLI contains 814 instances of probably in the MNLI dataset, but we found little to no evidence of WEP compositions among them, which can explain the results.

Finally, fine-tuning on the dataset of a particular probe leads to high test accuracy on the associated test set. More surprisingly, fine-tuning on one dataset also causes substantial accuracy gain on other probes. This suggests that our datasets can be incorporated in text encoder training in order to improve WEP handling.

Conclusion

We investigated WEP understanding in neural language models with new datasets and experiments, showing that WEP processing is challenging but helped by supervision which leads to transferable improvement. Future work could extract WEP probability scales from the UNLI dataset as an alternative to human perception surveys, but our work suggests that this requires language modeling progress.

A Associated probabilities

  Either Bernhard is a swan or sheep are afraid of mice. It is likely that either Bernhard is a swan or sheep are afraid of mice. It is unlikely that either Bernhard is a swan or sheep are afraid of mice. 0

	query(hypothesis). ProbLog Reasoner template Sample bAbI facts A,B,C,X,Y,Z p=0.7235 Sample chances p1…p6 T p1 hypothesis Sample facts 1...8 from facts T p' (hyp.): 1 premise hyp. p A,B,C,X,Y,Z in previous rounds Sample op 1…4 from {and, or, xor} distractor Compute hypothesis likelihood, relevant WEP factA Probability verbalization premise T p' (hyp.) y p' ≈ p likelihood T p' (hyp.): Reasoning Generated label y: p verbalization validity Generated input: premise, T p' (hyp.)

Table 2 :

 2 Median probability percentage associated to words of estimative probability according to[START_REF] Fagen-Ulmschneider | Perception of probability words[END_REF]. First and last words ( †) are taken from[START_REF] Kent | Words of estimative probability[END_REF].

	WEP	Median probability judgment
	certain almost certain highly likely very good chance 80.0 ± 10.8 100 † 95.0 ± 10.9 90.0 ± 8.4 we believe 75.0 ± 15.0 likely 70.0 ± 11.3 probably 70.0 ± 12.9 probable 70.0 ± 14.7 better than even 60.0 ± 9.1 about even 50.0 ± 4.9 probably not 25.0 ± 14.4 we doubt 20.0 ± 16.9 unlikely 20.0 ± 15.0 little chance 10.0 ± 12.2 chances are slight 10.0 ± 10.9 improbable 10.0 ± 17.5 highly unlikely 5.0 ± 17.3 almost no chance 2.0 ± 17.0 impossible 0 †
	B WEP verbalization template
	WEP	Verbalization template
	about even	chances are about even that [FACT]
	almost certain	it is almost certain that [FACT]
	almost no chance there is almost no chance that [FACT]
	better than even	there is a better than even chance that [FACT]
	certain	it is certain that [FACT]
	chances are slight chances are slight that [FACT]
	highly likely	it is highly likely that [FACT]
	highly unlikely	it is highly unlikely that [FACT]
	impossible	it is impossible that [FACT]
	improbable	it is improbable that [FACT]
	likely	it is likely that [FACT]
	little chance	there is little chance that [FACT]
	probable	it is probable that [FACT]
	probably	it is probably the case that [FACT]
	probably not	it is probably not the case that [FACT]
	unlikely	it is unlikely that [FACT]
	very good chance there is a very good chance that [FACT]
	we believe	we believe that [FACT]
	we doubt	we doubt that [FACT]

Table 3 :

 3 Templates used to convert a fact and a WEP expressed uncertainty into a modal sentence.

	C WEP frequencies on the generated datasets
	WEP-reasoning	(1 hop) WEP-Reasoning	(2 hops) WEP-USNLI
	WEP	frequency WEP	frequency WEP	frequency
	about even probably not better than even we believe highly likely certain highly unlikely almost no chance impossible almost certain very good chance chances are slight little chance probable unlikely likely probably we doubt improbable	11.1 impossible 9.7 about even 7.7 probably not 7.1 highly unlikely 6.4 almost no chance 6.0 better than even 5.9 we believe 5.8 highly likely 5.3 very good chance 5.1 we doubt 4.7 improbable 3.6 chances are slight 3.5 unlikely 3.2 little chance 3.1 almost certain 3.1 certain 3.0 likely 2.9 probable 2.9 probably	13.2 impossible 10.8 better than even 9.0 certain 8.2 about even 8.0 almost certain 6.6 highly likely 4.3 very good chance 4.0 almost no chance 4.0 we believe 4.0 highly unlikely 3.9 probably not 3.9 likely 3.6 probable 3.5 probably 2.9 unlikely 2.7 little chance 2.5 chances are slight 2.4 improbable 2.2 we doubt	25.6 10.7 7.2 6.9 6.7 6.0 5.9 5.0 4.1 4.1 3.4 2.5 2.4 2.4 1.5 1.5 1.5 1.4 1.4

Table 4 :

 4 Validation set frequency of WEP in the correct answer of each dataset (percentages).

/hf.co/.../probability_words_nli

This threshold ensures sufficient distance, while also ensuring that each WEP has at least one possible distractor.

Deviation from these hyperparameters did not yield significant improvement on the validation sets.

Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8713-8721.

Acknowledgements

This work is part of the CALCULUS project, which is funded by the ERC Advanced Grant H2020-ERC-2017 ADG 788506 4 .