Mark Shope (National Yang Ming Chiao Tung University; Indiana University Robert H. McKinney School of Law) has posted “GPT Performance on the Bar Exam in Taiwan” on SSRN. Here is the abstract:

This paper reports the performance of the GPT-4 Model of ChatGPT Plus (“ChatGPT4”) on the multiple-choice section of the 2022 Lawyer’s Bar Exam in Taiwan. ChatGPT4 outperforms approximately half of human test-takers on the multiple-choice section with a score of 342. This score, however, would not advance a test taker to the second and final essay portion of the exam. Therefore, this paper will not include an evaluation of ChatGPT4’s performance on the essay portion of the exam.

Erdem Dogukan Yilmaz (Erasmus Univeristy Rotterdam), Ivana Naumovska (INSEAD), and Vikas A. Aggarwal (INSEAD) have posted “AI-Driven Labor Substitution: Evidence from Google Translate and ChatGPT” on SSRN. Here is the abstract:

Although artificial intelligence (AI) has the potential to significantly disrupt businesses across a range of industries, we have limited empirical evidence for its substitution effect on human labor. We use Google’s introduction of neural network-based translation (GNNT) in 2016-2017 as a natural experiment to examine the substitution of human translators by AI in the context of a large online labor market. Using a difference-in-differences design, we show that the introduction of GNNT reduced the number of (human translation) transactions at both the overall market and individual translator levels. In addition, we show that GNNT had a stronger effect on translation tasks with analytical elements, as compared to those with cultural and emotional elements. In supplemental analyses, we document a similar pattern after the launch of ChatGPT using question and answer patterns in Stack Exchange forums. Our study thus offers robust and causal empirical evidence for a heterogeneous substitution effect of human tasks by skilled knowledge workers. We discuss the relevance of our findings for research on competitive advantage, technology adoption, and strategy microfoundations.

Rupert Macey-Dare (St Cross College – University of Oxford; Middle Temple; Minerva Chambers) has posted “How ChatGPT and Generative AI Systems will Revolutionize Legal Services and the Legal Profession” on SSRN. Here is the abstract:

In this paper, ChatGPT, is asked to provide c.150+ paragraphs of detailed prediction and insight into the following overlapping questions, concerning the potential impact of ChatGPT and successor generative AI systems on the evolving practice of law and the legal professions as we know them:

• Which are the individual legal business areas where ChatGPT could make a significant/ transformative impact and reduce costs and increase efficiencies?
• Where can ChatGPT use its special NLP abilities to assist in legal analysis and advice?
• Which are the specific areas where generative AI systems like ChatGPT can revolutionize and improve the legal profession?
• How can systems like ChatGPT help ordinary people with legal questions and legal problems?
• What is the likely timeframe for ChatGPT and other generative AI systems to transform legal services and the legal profession?
• What are the potential implications for new and intending law students?
• How will ChatGPT and similar systems impact professional lawyers in future?

Some of ChatGPT’s key insights and predictions (see full paper attached for detailed responses and analysis) are as follows:

ChatGPT identifies the following key individual legal business areas where it could make a significant/ transformative impact and reduce costs and increase efficiencies: Alternative dispute resolution, Automated billing, Case analysis, Case management, Compliance monitoring, Contract management, Contract review, Document automation, Document review, Discovery and E-discovery, Drafting legal documents, Due diligence, Expertise matching, Intellectual Property and IP management, Legal advice, Legal analytics, Legal chatbots, Legal drafting, Legal document review, Legal education, Legal marketing, Legal research, Litigation support, Natural language processing (NLP), Patent analysis, Predictive analytics, Regulatory compliance, Research, Risk assessment, Training and education, Translation and Virtual assistants.

ChatGPT flags up its special NLP abilities to assist in legal analysis and advice, particularly in the following key areas: Contract analysis, Document classification, Document summarization, Due diligence, Legal chatbots, Legal document review, Legal document summarization, Legal drafting, Legal language translation, Legal research, Named entity recognition, Predictive analytics, Regulatory compliance, Sentiment analysis and Topic modelling.

On the question of which are the specific areas where generative AI systems like ChatGPT can revolutionize and improve the legal profession, ChatGPT identifies: Accessibility, Accuracy, Collaboration, Cost reduction, Customization, Decision-making, Efficiency, Error-reduction, New business and innovation, Job displacement potential, Legal research, Risk management and Scalability.

On the question of how can systems like ChatGPT help ordinary people with legal questions and legal problems, ChatGPT identifies the following areas: 24/7 availability, Automated legal services, Consistency of advice, Contract review, Cost-effectiveness, Court filings, Customization, Document preparation, Education, Empowerment, Faster response times, Language translation, Legal advice, Legal chatbots, Legal education, Legal research, Mediation and dispute resolution, Privacy, Scalability and Simplified language.

On the question of the likely timeframe for ChatGPT and other generative AI systems to transform legal services and the legal profession, Chat GPT comments that “It is difficult to predict with certainty, as it will depend on a variety of factors such as technological advancements, regulatory changes, and market demand. [However] There are several reasons to believe that the transformation of legal services through generative AI systems like ChatGPT will happen relatively quickly…the transformation of legal services through generative AI systems is likely to happen relatively quickly, potentially within the next 5-10 years.”

On the potential implications for new and intending law students, ChatGPT comments that: “It is essential for students contemplating studying law to be aware of the ability of ChatGPT and generative AI systems to perform legal jobs instead of humans in the future. These technologies are already transforming the legal industry, and their impact is likely to continue to grow in the years to come. Understanding how these systems work and how they can be used in legal practice will be crucial for anyone seeking a career in law, and may also provide a competitive advantage in the job market. By understanding the potential impact of these technologies on the legal industry, students can better prepare themselves for the changing nature of legal work and take advantage of the new opportunities that are emerging as a result of these developments.”

On the all-important question for lawyers of how ChatGPT and similar systems will impact professional lawyers in future, ChatGPT comments that: “Generative AI systems like ChatGPT have the potential to significantly impact the demand and payment for professional lawyers in the future. As these systems become more advanced and capable of performing a wider range of legal tasks, it is likely that they will begin to replace some of the work that is currently performed by human lawyers.”

“One area where this is already happening is in document review. AI systems can review and analyze large volumes of documents much more quickly and accurately than human lawyers, which can save significant time and cost for law firms and their clients. As these systems become more sophisticated, they may also be able to perform other tasks such as legal research and analysis, drafting legal documents, and even providing legal advice.”

“The impact of generative AI systems on the demand for lawyers will depend on the specific tasks and areas of law that they are able to perform. It is likely that some areas of law will be more heavily impacted than others. For example, areas such as contract law and intellectual property may be more susceptible to automation, while litigation and dispute resolution may be less so.”

“In terms of payment for professional lawyers, the impact of generative AI systems is also likely to vary depending on the specific tasks and areas of law. In some cases, these systems may allow lawyers to perform their work more quickly and efficiently, which could potentially lead to higher billable hours and increased income. However, if these systems are able to replace some of the work that is currently performed by human lawyers, it could also lead to a reduction in demand for these services and a decrease in fees.”

“One potential impact of systems like ChatGPT on the legal industry is a reduction in the demand for certain types of legal work that can be automated or performed more efficiently by AI systems. For example, tasks like document review, contract drafting, and legal research may be performed more accurately and quickly by AI systems than by humans, leading to a decrease in the number of lawyers needed to perform these tasks.”

“It is also possible that the development of AI systems like ChatGPT will lead to changes in the way that legal services are priced and delivered. As these technologies become more common, it is likely that clients will begin to expect lower costs and faster turnaround times for certain types of legal work. This could lead to increased competition among legal service providers, which in turn could put pressure on lawyers to lower their rates or find ways to deliver legal services more efficiently….it is clear that these technologies have the potential to significantly change the legal industry, and that lawyers will need to adapt in order to remain competitive and relevant in a rapidly changing market. This may involve developing new skills and knowledge related to working alongside AI systems, or focusing on areas of law that are less susceptible to automation.”

Interestingly, although ChatGPT does discuss practical contract management, IP and evidence, it does not seem to predict inroads being made into academic legal analysis, statutory construction, complex case analysis or the development of new legal thinking and principles, so not into the theoretical domain of law professors and senior lawyers and judges, (although there are additional reasons why there are likely to be knock-on reductions in demand for these specialist lawyers too).

But for the vast majority of procedural (routinely turning-the-handle type) practitioner law and practice, ChatGPT seems to be predicting a seismic sectoral shock, a reduction in human-centric legal work, an increase in legal self-help for clients and the public, and a technological transformation in and fundamental repricing and manpower shock for the legal sector within a timeframe of 5-10 years.

N.B. This is only one set of predictions, which could prove right or wrong, indeed from an unconscious chatbot machine ChatGPT. However it has the credibility of being made based on both a huge body of knowledge data, and on the consistent rules programmed into ChatGPT itself, and by apparently coherently reasoned responses. Time will soon tell of course…

Tammy Pettinato Oltz (University of North Dakota School of Law) has posted “ChatGPT, Professor of Law” on SSRN. Here is the abstract:

Although ChatGPT was just released by OpenAI in November 2022, legal scholars have already been delving into the implications of the new tool for legal education and the legal profession. Several scholars have recently written fascinating pieces examining ChatGPT’s ability to pass the bar, write a law review article, create legal documents, or pass a law school exam. In the spirit of those experiments, I decided to see whether ChatGPT had potential for lightening the service and teaching loads of law school professors.

To conduct my experiment, I created an imaginary law school professor with a tough but typical week of teaching- and service- related tasks ahead of her. I chose seven common tasks: creating a practice exam question, designing a hand-out for a class, writing a letter of recommendation, submitting a biography for a speaking engagement, writing opening remarks for a symposium, developing a document for a law school committee, and designing a syllabus for a new course. I then ran prompts for each task through ChatGPT to see how well the system performed the tasks.

Remarkably, ChatGPT was able to provide useable first drafts for six out of seven of the tasks assigned in only 23 minutes. Overall and unsurprisingly, ChatGPT proved to be best at those tasks that are most routine. Tasks that require more sophistication, particularly those related to teaching, were harder for ChatGPT, but still showed potential for time savings.

In this paper, I describe a typical work scenario for a hypothetical law professor, show how she might use ChatGPT, and analyze the results. I conclude that ChatGPT can drastically reduce the service-related workload of law school faculty and can also shave off time on back-end teaching tasks. This freed-up time could be used to either enhance scholarly productivity or further develop more sophisticated teaching skills.

Rebecca Kunkel (Rutgers Law) has posted “Artificial Intelligence, Automation, and Proletarianization of the Legal Profession” (Creighton Law Review, Vol. 56, 2022) on SSRN. Here is the abstract:

Recent advances in computer programming, broadly categorized as “artificial intelligence,” (“Al”) have renewed debates over machines as viable replacements for human lawyers. Some prominent lawyers and legal scholars now adhere to a vision of the future heavily seasoned with Silicon Valley-style techno-utopianism: the legal profession may endure but only in a form in which it would be almost unrecognizable today, while legal innovators will need to immerse themselves in the possibilities opened up by artificial intelligence in order to survive. For others, the view of artificial intelligence and its potential application to law is more limited, as they argue for the impossibility of automating many essential aspects of legal service. These views share key assumptions about the nature of Al technology: that technological development follows its own course and that the widespread adoption of technologies is primarily determined by objective measures of efficacy. This essay offers an alternate Marxian account of legal Al which places it in the larger history of automation and proletarianization.

Ilias Chalkidis (University of Copenhagen) has posted “ChatGPT May Pass the Bar Exam Soon, but Has a Long Way to Go for the LexGLUE Benchmark” on SSRN. Here is the abstract:

Following the hype around OpenAI’s ChatGPT conversational agent, the last straw in the recent development of Large Language Models (LLMs) that demonstrate emergent unprecedented zero-shot capabilities, we audit the latest OpenAI’s GPT-3.5 model, ‘gpt-3.5-turbo’, the first available ChatGPT model, in the LexGLUE benchmark in a zero-shot fashion providing examples in a templated instruction-following format. The results indicate that ChatGPT achieves an average micro-F1 score of 49.0% across LexGLUE tasks, surpassing the baseline guessing rates. Notably, the model performs exceptionally well in some datasets, achieving micro-F1 scores of 62.8% and 70.1% in the ECtHR B and LEDGAR datasets, respectively. The code base and model predictions are available at https://github.com/coastalcph/zeroshot_lexglue.

Daniel Martin Katz (Illinois Tech – Chicago Kent College of Law; Bucerius Center for Legal Technology & Data Science; Stanford CodeX – The Center for Legal Informatics; 273 Ventures), Dirk Hartung (Bucerius Law School – Center for Legal Technology and Data Science; Stanford University – Stanford Codex Center), Lauritz Gerlach (Bucerius Law School), Abhik Jana
(University of Hamburg; Language Technology Group, Department of Informatics, Universität Hamburg), and Michael James Bommarito (273 Ventures; Licensio, LLC; Stanford Center for Legal Informatics; Michigan State College of Law; Bommarito Consulting, LLC) have posted “Natural Language Processing in the Legal Domain” on SSRN. Here is the abstract:

In this paper, we summarize the current state of the field of NLP and Law with a specific focus on recent technical and substantive developments. To support our analysis, we construct and analyze a corpus of more than six hundred NLP and Law related papers published over the past decade. Our analysis highlights several major trends. Namely, we document an increasing number of papers written, tasks undertaken, and languages covered over the course of the past decade. We observe an increase in the sophistication of the methods which researchers deployed in this applied context. Slowly but surely, Legal NLP is beginning to match the methodological sophistication of general NLP. We believe this to be a positive trend for the future of the field, but many questions in both the academic and commercial sphere still remain open.

Jonathan H. Choi (U Minnesota Law) et al. have posted “ChatGPT Goes to Law School” on SSRN. Here is the abstract:

How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses. After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.

John Nay (Stanford CodeX) has posted “Large Language Models as Corporate Lobbyists” on SSRN. Here is the abstract:

We demonstrate a proof-of-concept of a large language model conducting corporate lobbying related activities. An autoregressive large language model (OpenAI’s text-davinci-003) determines if proposed U.S. Congressional bills are relevant to specific public companies and provides explanations and confidence levels. For the bills the model deems as relevant, the model drafts a letter to the sponsor of the bill in an attempt to persuade the congressperson to make changes to the proposed legislation. We use hundreds of novel ground-truth labels of the relevance of a bill to a company to benchmark the performance of the model. It outperforms the baseline of predicting the most common outcome of irrelevance. We also benchmark the performance of the previous OpenAI GPT-3 model (text-davinci-002), which was the state-of-the-art model on many academic natural language tasks until text-davinci-003 was recently released. The performance of text-davinci-002 is worse than the simple baseline. These results suggest that, as large language models continue to exhibit improved natural language understanding capabilities, performance on lobbying related tasks will continue to improve.​​ Longer-term, if AI begins to influence law in a manner that is not a direct extension of human intentions, this threatens the critical role that law as information could play in aligning AI with humans. Initially, AI is being used to simply augment human lobbyists for a small portion of their daily tasks. However, firms have an incentive to use less and less human oversight over automated assessments of policy ideas and the written communication to regulatory agencies and Congressional staffers. The core question raised is where to draw the line between human-driven and AI-driven policy influence.

Matthew Waddington (Legislative Drafting Office, States of Jersey) has posted “Rules As Code: Drawing Out the Logic of Legislation for Drafters and Computers” (Modern Legislative Drafting – A Research Companion, Constantin Stefanou (ed.) (Routledge) (Forthcoming)) on SSRN. Here is the abstract:

This chapter outlines developments in the digitisation of legislative drafts, looking at “computational law” and the spectre of artificial intelligence (“AI”), but focussing mainly on “Rules as Code” (or “RaC”). The concept of RaC presented here is not one that claims to be able to digitise all of the law, or even all aspects of a piece of legislation. Nor is it one that claims computers should interpret the substantive terms used in legislation, or fill in implied concepts, as opposed to interpreting terms like “if”, “and”, “or”, “not”, “means”, “includes”, “must” and “may” that drafters are already trying to use in a disciplined way.

The chapter examines what can be drawn from the increasingly systematic approach among legislative drafters in Commonwealth countries to handling different key components of legislation. The shift from “shall” to “must” (and “is”) has more clearly exposed differences between constitutive provisions (such as definitions, or rules about whether a notice or application is valid), provisions taking effect by operation of law (such as establishing a statutory body corporate), and normative provisions (such as imposing an obligation, or creating an offence). Basic deontic logic symbols can be used to illustrate the way drafters limit normative terms (other than in offences) to basic building blocks of “must”, “must not” and “may”. Drafters use those building blocks to create what others label as “rights” and “powers” in ways that mean modern legislative drafting in the Commonwealth may be able to avoid many of the problems and complications that have beset those who have tried to formalise or digitise law. In particular it may avoid some of the difficulties that occur when, before attempting the digital capture of the elements of legislation, an attempt is made to apply deontic logic to law in general, or to wrestle with systematising legal expressions such as rights and privileges in the fashion attempted by Hohfeld, or to formalise fundamental legal concepts like Sartor’s, or to pin down a large range of concepts as in LegalRuleML. Those broader issues may need to be tackled in the longer term, but in the short term a simplified approach could produce results and help drafters grasp the insights that can be obtained from this approach.