Events

Math and Democracy Seminar: Large Legal Fictions: Detecting Legal Hallucinations in Large Language Models

Speaker: Matthew Dahl (Yale University)

Location: 60 Fifth Avenue, Room 650

Date: Monday, March 31, 2025

Do large language models (LLMs) know the law? LLMs are increasingly being used to augment legal practice, but their revolutionary potential is threatened by the presence of legal “hallucinations” — textual output that is not consistent with the content of the law. In this talk, I theorize the provenance and nature of these hallucinations and discuss methods for detecting them in LLM outputs. I then share results from three experiments auditing off-the-shelf LLMs and industry retrieval-augmented generation (RAG) models, showing that legal errors remain widespread. I conclude by emphasizing the need for empirical evidence in an age of ever-increasing hype about AI’s ability to replace lawyers and expand access to justice.