MultiLegalPile: A 689GB Multilingual Legal Corpus
Subject: Legal Language Processing and Multilingual Datasets
Author or source: Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis and Daniel E. Ho
Updated:
Complete ACL 2024 paper about constructing and evaluating MultiLegalPile, with multilingual legal pretraining experiments, source discussion and appendices.
Overview of the complete resource
Source page 1–18
This is the complete 18-page research paper, not the 689 GB dataset named in its title. It describes a multilingual legal collection, its constituent sources and experiments with domain-pretrained language models. The authors compare multilingual and language-specific models using legal-language benchmarks and discuss the composition of the training material. The appendices and bibliography are retained. For study, identify which jurisdictions and languages an experiment actually covers before assuming the results transfer to another court collection. The paper does not establish reliable legal advice or accuracy on Indian case law. The publication itself is available under CC BY 4.0; the underlying corpora, source documents, models and software have their own conditions, which must be checked separately before reuse. LawSphereX hosts this paper only and does not download, redistribute or claim to host the large dataset. Related learning resources are supplementary rather than a page-specific interpretation.
Original research paper: reading preview
Source page 1
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15077–15094 August 11-16, 2024 ©2024 Association for Computational Linguistics MultiLegalPile: A 689GB Multilingual Legal Corpus Joel Niklaus1,2,4 Veton Matoshi2 Matthias Stürmer1,2 Ilias Chalkidis3 Daniel E. Ho4 1University of Bern 2Bern University of Applied Sciences 3University of Copenhagen 4Stanford University Abstract Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, few datasets are available for specialized critical domains such as law and the available ones are often small and only in English. To fill this gap, we curate and re- lease MULTILEGALPILE, a 689GB corpus in 24 languages from 17 jurisdictions. MULTILE- GALPILE includes diverse legal data sources and allows for pretraining NLP models under fair use, with most of the dataset licensed very permissively. We pretrain two RoBERTa mod- els and one Longformer multilingually, and 24 monolingual models on each of the language- specific subsets and evaluate them on LEX- TREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEX- TREME and our English models on LexGLUE. We release the dataset, trained models, and all code under the most open licenses possible. 1 Introduction Recent years have seen LLMs achieving remark- able progress, as demonstrated by their perfor- mance on various benchmarks such as SuperGLUE (Wang et al., 2019), MMLU (Hendrycks et al., 2021), and several human Exams (OpenAI, 2023), including U.S. bar exams for admission to practice law (Katz et al., 2023). These models are typically trained on increasingly large corpora, such as the Pile (Gao et al., 2020a), C4 (Raffel et al., 2020), and mC4 (Xue et al., 2021). However, public cor- pora available for training these models are predom- inantly in English, and often constitute web text with unclear licensing. This even led to lawsuits against LLM producers1, highlighting this critical issue. Furthermore, there is a scarcity of large- scale, domain-specific pretraining corpora, which 1https://www.theverge.com/2022/11/8/23446821/ microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data Figure 1: MULTILEGALPILE Source Distribution constitutes a significant gap in the current body of resources available for the training of LLMs. We find that only one in every thousand docu- ment in mC4 contains legal text. Similarly, LLMs are predominantly English, especially considering domain-specific models, e.g., ones specialized in biomedical, legal, or financial texts. Legal texts, often produced by public instru- ments (e.g., state governments, international or- ganizations), are typically available under public li- censes, offering a rich resource for domain-specific pretraining. Given this context, we curate a mas- sive, openly available, corpus of multilingual law text spanning across numerous jurisdictions (legal systems), predominantly under permissive licenses. Further on, we continue pretraining XLM-R models (Conneau and Lample, 2019) on our cor- pus and evaluated these models on the recently introduced LEXTREME (Niklaus et al., 2023) and LexGLUE (Chalkidis et al., 2022) benchmarks. Given the often extensive nature of legal text, we also pretrained a Longformer model (Beltagy et al., 15077