LEGAL-BERT: The Muppets straight out of Law School
Subject: Legal Language Processing and Artificial Intelligence
Author or source: Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras and Ion Androutsopoulos
Updated:
Complete ACL research paper comparing strategies for adapting BERT to legal tasks, including domain pretraining, fine-tuning and released LEGAL-BERT models.
Overview of the complete resource
Source page 1–7
This complete seven-page conference paper studies how a general language model can be adapted to legal text. The authors compare using BERT without additional domain training, continuing pretraining on legal material and training a domain model from scratch. They examine downstream tasks and fine-tuning choices rather than treating one training recipe as universally best. Read the dataset descriptions, experiment settings and limitations alongside the reported results. These experiments concern their particular datasets and models; they do not establish that automated legal answers are reliable, legally correct or suitable for Indian practice. The supplementary BERT tutorial explains the model family and is not a lecture presenting this exact paper. The PDF and original authorship are preserved unchanged.
Original research paper: reading preview
Source page 1
Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2898 LEGAL-BERT: The Muppets straight out of Law School Ilias Chalkidis † ‡ Manos Fergadiotis † ‡ Prodromos Malakasiotis † ‡ Nikolaos Aletras ∗ Ion Androutsopoulos † ‡ † Department of Informatics, Athens University of Economics and Business ‡ Institute of Informatics & Telecommunications, NCSR “Demokritos” ∗Computer Science Department, University of Sheffield, UK [ihalk,fergadiotis,rulller,ion]@aueb.gr n.aletras@sheffield.ac.uk Abstract BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guide- lines in specialised domains. Here we focus on the legal domain, where we explore sev- eral approaches for applying BERT models to downstream legal tasks, evaluating on multi- ple datasets. Our findings indicate that the previous guidelines for pre-training and fine- tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the avail- able strategies when applying BERT in spe- cialised domains. These are: (a) use the orig- inal BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific cor- pora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we re- lease LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computa- tional law, and legal technology applications. 1 Introduction Pre-trained language models based on Transform- ers (Vaswani et al., 2017), such as BERT (Devlin et al., 2019) and its variants (Liu et al., 2019; Yang et al., 2019; Lan et al., 2019), have achieved state-of-the-art results in several downstream NLP tasks on generic benchmark datasets, such as GLUE (Wang et al., 2018), SQUAD (Rajpurkar et al., 2016), and RACE (Lai et al., 2017). Typically, transfer learning with language mod- els requires a computationally heavy step where the language model is pre-trained on a large corpus and a less expensive step where the model is fine- tuned for downstream tasks. When using BERT, the first step can be omitted as the pre-trained mod- els are publicly available. Being pre-trained on Figure 1: The three alternatives when employing BERT for NLP tasks in specialised domains: (a) use BERT out of the box, (b) further pre-train BERT (FP), and (c) pre-train BERT from scratch (SC). All strategies have a final fine-tuning step. generic corpora (e.g., Wikipedia, Children’s Books, etc.) BERT has been reported to under-perform in specialised domains, such as biomedical or scien- tific text (Lee et al., 2019; Beltagy et al., 2019). To overcome this limitation there are two possible strategies; either further pre-train (FP) BERT on domain specific corpora, or pre-train BERT from scratch (SC) on domain specific corpora. Conse- quently, to employ BERT in specialised domains one may consider three alternative strategies before fine-tuning for the downstream task (Figure 1): (a) use BERT out of the box, (b) further pre-train (FP) BERT on domain-specific corpora, and (c) pre-train BERT from scratch (SC) on domain specific corpora with a new vocabulary of sub-word units. In this paper, we systematically explore strate- gies (a)–(c) in the legal domain, where BERT adap- tation has yet to be explored. As with other spe- cialised domains, legal text (e.g., laws, court plead- ings, contracts) has distinct characteristics com- pared to generic corpora, such as specialised vocab- ulary, particularly formal syntax, semantics based on extensive domain-specific knowledge etc., to the extent that legal language is often classified as a ‘sublanguage’ (Tiersma, 1999; Williams, 2007; Haigh, 2018). Note, however, that our work con- tributes more broadly towards a better understand- ing of domain adaptation for specialised domains. Our key findings are: (i) Further pre-training (FP) or pre-training BERT from scratch (SC) on domain-