Lexis and Westlaw have been best in class in legal research. As of today, that is no longer true.<\/p>\n
Your skepticism is welcome. Lawyers are paid to be right. But they are expected to not be wrong. You are justifiably hesitant to experiment. But you should experiment. Your competitors already are.<\/p>\n
I am not arguing that you should leave Westlaw or Lexis. Yet. But if you review the evidence and access our tool (which we are giving away until the end of October), I am confident you will be convinced, not only that these other platforms might not be necessary, but that they are no longer state of the art.<\/p>\n
Let\u2019s start off with some background.<\/p>\n
For decades, two companies have controlled public case law and legal research. They have succeeded largely in two ways:<\/p>\n
First, both companies have manufactured a quasi-governmental hold over case law.<\/p>\n
Thankfully, these unprincipled bureaucratic protections prove no match for our agentic world. American case law, our national legal heritage, yearns to be free. So we freed it. With LOIS, Filevine has built a formidable corpus of American case law from every state and every federal court from the country\u2019s founding to today. The corpus is updated daily and, beginning later this month, will include reporter pagination and citations for published opinions.<\/p>\n
Filevine assembled this collection from a wide range of public and licensed legal sources, including resources made available by community-minded organizations like CourtListener. Other materials come from established legal-data providers whose research is used by some of the world\u2019s largest companies. To our research partners and others involved in this effort to broaden access to legal information, our gratitude runs deep.<\/p>\n
Second, Westlaw and Lexis have, historically, had the only credible citators in the industry.<\/p>\n
Practicing law requires more than a strong corpus. It requires lawyers to cite cases and holdings that are still \u201cgood law.\u201d<\/p>\n
West\u2019s KeyCite and Lexis\u2019 Shepardization, also called \u201cShepard\u2019s,\u201d have been institutionalized as authoritative.<\/p>\n
Yet these tools carry a hidden cost. They force lawyers to adapt their research to the limitations of the software. A lawyer cannot simply describe a legal issue and ask for the best authority. They must learn specialized search syntax, anticipate the database\u2019s taxonomy, and manually interpret a maze of signals and citing references. Even experienced lawyers struggle to use them reliably. They call vendor help desks, delegate searches to less expensive associates, and spend hours validating authorities. The legal industry has come to treat this complexity as an inherent part of legal reasoning. It is not.<\/p>\n
KeyCite and Shepard\u2019s are beset by serious structural deficits we will detail below. Both citators are remarkably brittle, and the authorities they return often have little overlap. The gaps in citator results aren\u2019t widely known but should be shocking to any practitioner. As John Rizner (J.D., University of Chicago, 2018), who heads our citation project, observed, reiterating Paul Hellyer\u2019s findings below: \u201cA lawyer\u2019s citation accuracy has more to do with which product they subscribe to than the case law being evaluated.\u201d<\/p>\n
Filevine is not the first to point out the problems inherent to legacy citators. Susan Nevelow Mart, Professor Emeritus at University of Colorado Law School and former director of the William A. Wise Law Library, has written extensively on the topic, including The Relevance of Results Generated by Human Indexing and Computer Algorithms: A Study of West\u2019s Headnotes and Key Numbers and LexisNexis\u2019s Headnotes and Topics<\/em>, 102 Law Libr. J. 221 (2010) and her follow-up The Case for Curation: The Relevance of Digest and Citator Results in Westlaw and Lexis<\/em>, 32 Legal Reference Services Q. 13 (2013). In The Case for Curation<\/em>, Professor Mart and her team compiled a set of 90 cases, and then compared the results from KeyCite and Shepard\u2019s relating to headnotes in each of these cases, for relevance and overlap. They found that the percentage of overlap between KeyCite and Shepard\u2019s was only 33%. For the cases considered both relevant and unique by Mart and her researchers, 42% of everything Shepard\u2019s returned was both unique to Shepard\u2019s and relevant; for KeyCite the figure was 31%. Mart noted her surprise at \u201chow few cases each citation system had in common; there was not that much overlap in the cases found using KeyCite with the cases found using Shepard\u2019s.\u201d<\/p>\n To the practitioners possibly worried about their own legal research, given the findings of the study, she described the remedy as looking at other legal research sources: \u201cRedundancy from using multiple other legal resources should cure the problem.\u201d<\/p>\n Paul Hellyer, Senior Reference Librarian at the William & Mary Law School, conducted a related study in Evaluating Shepard\u2019s, KeyCite, and BCite for Case Validation Accuracy<\/em>, 110 Law Libr. J. 449 (2018). Hellyer started with 73 published decisions of the Ninth Circuit Court of Appeals from January 1984, and collected 357 citing references (which at least one citator designated as being negative) for those 73 opinions, from Shepard\u2019s, KeyCite, and BCite. He then read the opinions from the negative citing references, assessed the negative treatment, and compared his assessments of the three legacy citators. He found the following:<\/p>\n [I]n 85% of these citing relationships, the three citators do not agree on whether there was negative treatment. Even when they all agree there was negative treatment, their descriptions often conflict. The three databases substantively agree on the type of negative treatment in only 40 of these citing relationships, which means that in this sample, they all agree with one another only 11% of the time.<\/p>\n<\/blockquote>\n Hellyer recognized the dangerous pitfalls for practitioners that emerge from these findings:<\/p>\n<\/blockquote>\n On the surface, even without any further examination, these numbers are troubling. They mean that at least one citator is making a lot of mistakes, or (viewed more charitably) that the citators are very likely to have differences of opinion. Even the latter explanation is worrisome, because all three citators present their results as fact \u2014 none of the citators labeled any of the citing relationships as ambiguous or open to interpretation. This means that when you citate a case that has negative treatment, the results you get depend mainly on which citator you happen to be using.<\/p>\n<\/blockquote>\n He concluded:<\/p>\n<\/blockquote>\n Some readers may remain unconvinced. But it\u2019s not only my own judgment that conflicts with the citators\u2014for almost 90% of the negative labels reviewed here, at least one citator conflicts with another. If these discrepancies can\u2019t be objectively resolved, then the citators themselves are based on nothing more than idiosyncratic opinion. If true, this would undermine the citators\u2019 usefulness. The citators can be reliable or they can be idiosyncratic, but they can\u2019t be both.<\/p>\n<\/blockquote>\n<\/blockquote>\n Our testing echoes the academic findings: legacy tools routinely mischaracterize negative treatment, struggle to find sub silentio or otherwise indirect negative citations, and demonstrate significant divergence in citational results when these legacy citators are compared with one another.<\/p>\n The opportunity is to open case law to everyone, not only the trained few. But access is only half the problem. No amount of expertise can find relationships a citator was not built to detect.<\/p>\n LOIS Citator was built from first principles in an AI-native world. We worked independently of the Duopoly\u2019s corpora to find case relationships no editorial team could map by hand. American case law is simply too vast and voluminous.<\/p>\n Other legaltech companies say their AI \u201csupplements\u201d Westlaw and Lexis for case law research. We aim to replace them, first for case law, and then for statutory and regulatory research, and then for the rest of the legacy suite.<\/p>\n At Filevine, we have long believed a better way existed. This project has been on our minds for years. When LLMs began to show real promise in legal research, we wondered whether that time had come. Instead, LLMs have brought with them a tidal wave of fake cases, inaccurate quotes, and unsupported (and often erroneous) reasoning. Reputations have been marred. Monetary sanctions and public admonishment are now commonplace. (See our work on anti-hallucination: https:\/\/www.filevine.com\/blog\/solving-hallucinations-with-lois\/<\/a><\/strong>)<\/p>\n Worse, judges have almost certainly relied on hallucinated research, without anyone ever knowing. Those incidents may never appear in the public record.<\/p>\n Hallucination cannot remain a feature of our justice system. It must be rooted out at every turn, with zero tolerance. The minute we accept hallucinations as the cost of doing business with LLMs, we concede to a justice system shaped not only by verified facts and law, but by probabilistic algorithms.<\/p>\n And yet \u2026 AI holds such promise. We can all see it. Broad adoption is already here. There is no going back. Its over. Just as in the software industry, where AI landed first, legal has already hit saturation at a rate that feels magical and bewildering. It turns out lawyers were never Luddites. They just didn\u2019t like technology that was hard to learn, technology that required a whole semester\u2019s course in law school to figure out. They had big jobs to do, and updating document metadata wasn\u2019t one of them. AI can, and should, be a boon to justice. At Filevine, we measure our addressable market not by seats or tokens, but by the gap between how much justice the world needs and how much it has today. AI may be everywhere. But justice is not.<\/p>\n So our dogged team, led by John, Matej, Marek, David, Jonas, Michael, Dan, and others, built (1) anti-hallucination tooling for any usage of opinions by LOIS and (2) a state-of-the-art AI citator, which we call LOIS Citator. And it\u2019s phenomenal.<\/p>\n The development of this tool was expensive and extremely challenging. John will share some details on it next week. But today, we are giving it to the world to try for free. Between now and LEX, Filevine\u2019s flagship legal conference this October, any legal professional can use LOIS Citator and the rest of our anti-hallucination tools free of charge. I\u2019m sure you will find some gaps. For now, we\u2019re only focused on American case law. We welcome your feedback.<\/p>\n LOIS Citator isn\u2019t perfect. We want your feedback. But as of today, we\u2019re confident you will find cases Westlaw and Lexis will miss. One case can make all the difference for a client in need. We are thrilled to share these tools with you.<\/p>\n Welcome to LOIS for Legal Research.<\/a><\/p>\n Ryan Anderson The post Open Letter To Users Of Westlaw And LexisNexis<\/a> appeared first on Above the Law<\/a>.<\/p>\n Lexis and Westlaw have been best in class in legal research. As of today, that is no longer true.<\/p>\n Your skepticism is welcome. Lawyers are paid to be right. But they are expected to not be wrong. You are justifiably hesitant to experiment. But you should experiment. Your competitors already are.<\/p>\n I am not arguing that you should leave Westlaw or Lexis. Yet. But if you review the evidence and access our tool (which we are giving away until the end of October), I am confident you will be convinced, not only that these other platforms might not be necessary, but that they are no longer state of the art.<\/p>\n Let\u2019s start off with some background.<\/p>\n For decades, two companies have controlled public case law and legal research. They have succeeded largely in two ways:<\/p>\n First, both companies have manufactured a quasi-governmental hold over case law.<\/p>\n Thankfully, these unprincipled bureaucratic protections prove no match for our agentic world. American case law, our national legal heritage, yearns to be free. So we freed it. With LOIS, Filevine has built a formidable corpus of American case law from every state and every federal court from the country\u2019s founding to today. The corpus is updated daily and, beginning later this month, will include reporter pagination and citations for published opinions.<\/p>\n Filevine assembled this collection from a wide range of public and licensed legal sources, including resources made available by community-minded organizations like CourtListener. Other materials come from established legal-data providers whose research is used by some of the world\u2019s largest companies. To our research partners and others involved in this effort to broaden access to legal information, our gratitude runs deep.<\/p>\n Second, Westlaw and Lexis have, historically, had the only credible citators in the industry.<\/p>\n Practicing law requires more than a strong corpus. It requires lawyers to cite cases and holdings that are still \u201cgood law.\u201d<\/p>\n West\u2019s KeyCite and Lexis\u2019 Shepardization, also called \u201cShepard\u2019s,\u201d have been institutionalized as authoritative.<\/p>\n Yet these tools carry a hidden cost. They force lawyers to adapt their research to the limitations of the software. A lawyer cannot simply describe a legal issue and ask for the best authority. They must learn specialized search syntax, anticipate the database\u2019s taxonomy, and manually interpret a maze of signals and citing references. Even experienced lawyers struggle to use them reliably. They call vendor help desks, delegate searches to less expensive associates, and spend hours validating authorities. The legal industry has come to treat this complexity as an inherent part of legal reasoning. It is not.<\/p>\n KeyCite and Shepard\u2019s are beset by serious structural deficits we will detail below. Both citators are remarkably brittle, and the authorities they return often have little overlap. The gaps in citator results aren\u2019t widely known but should be shocking to any practitioner. As John Rizner (J.D., University of Chicago, 2018), who heads our citation project, observed, reiterating Paul Hellyer\u2019s findings below: \u201cA lawyer\u2019s citation accuracy has more to do with which product they subscribe to than the case law being evaluated.\u201d<\/p>\n Filevine is not the first to point out the problems inherent to legacy citators. Susan Nevelow Mart, Professor Emeritus at University of Colorado Law School and former director of the William A. Wise Law Library, has written extensively on the topic, including The Relevance of Results Generated by Human Indexing and Computer Algorithms: A Study of West\u2019s Headnotes and Key Numbers and LexisNexis\u2019s Headnotes and Topics<\/em>, 102 Law Libr. J. 221 (2010) and her follow-up The Case for Curation: The Relevance of Digest and Citator Results in Westlaw and Lexis<\/em>, 32 Legal Reference Services Q. 13 (2013). In The Case for Curation<\/em>, Professor Mart and her team compiled a set of 90 cases, and then compared the results from KeyCite and Shepard\u2019s relating to headnotes in each of these cases, for relevance and overlap. They found that the percentage of overlap between KeyCite and Shepard\u2019s was only 33%. For the cases considered both relevant and unique by Mart and her researchers, 42% of everything Shepard\u2019s returned was both unique to Shepard\u2019s and relevant; for KeyCite the figure was 31%. Mart noted her surprise at \u201chow few cases each citation system had in common; there was not that much overlap in the cases found using KeyCite with the cases found using Shepard\u2019s.\u201d<\/p>\n To the practitioners possibly worried about their own legal research, given the findings of the study, she described the remedy as looking at other legal research sources: \u201cRedundancy from using multiple other legal resources should cure the problem.\u201d<\/p>\n Paul Hellyer, Senior Reference Librarian at the William & Mary Law School, conducted a related study in Evaluating Shepard\u2019s, KeyCite, and BCite for Case Validation Accuracy<\/em>, 110 Law Libr. J. 449 (2018). Hellyer started with 73 published decisions of the Ninth Circuit Court of Appeals from January 1984, and collected 357 citing references (which at least one citator designated as being negative) for those 73 opinions, from Shepard\u2019s, KeyCite, and BCite. He then read the opinions from the negative citing references, assessed the negative treatment, and compared his assessments of the three legacy citators. He found the following:<\/p>\n [I]n 85% of these citing relationships, the three citators do not agree on whether there was negative treatment. Even when they all agree there was negative treatment, their descriptions often conflict. The three databases substantively agree on the type of negative treatment in only 40 of these citing relationships, which means that in this sample, they all agree with one another only 11% of the time.<\/p>\n<\/blockquote>\n Hellyer recognized the dangerous pitfalls for practitioners that emerge from these findings:<\/p>\n<\/blockquote>\n On the surface, even without any further examination, these numbers are troubling. They mean that at least one citator is making a lot of mistakes, or (viewed more charitably) that the citators are very likely to have differences of opinion. Even the latter explanation is worrisome, because all three citators present their results as fact \u2014 none of the citators labeled any of the citing relationships as ambiguous or open to interpretation. This means that when you citate a case that has negative treatment, the results you get depend mainly on which citator you happen to be using.<\/p>\n<\/blockquote>\n He concluded:<\/p>\n<\/blockquote>\n Some readers may remain unconvinced. But it\u2019s not only my own judgment that conflicts with the citators\u2014for almost 90% of the negative labels reviewed here, at least one citator conflicts with another. If these discrepancies can\u2019t be objectively resolved, then the citators themselves are based on nothing more than idiosyncratic opinion. If true, this would undermine the citators\u2019 usefulness. The citators can be reliable or they can be idiosyncratic, but they can\u2019t be both.<\/p>\n<\/blockquote>\n<\/blockquote>\n Our testing echoes the academic findings: legacy tools routinely mischaracterize negative treatment, struggle to find sub silentio or otherwise indirect negative citations, and demonstrate significant divergence in citational results when these legacy citators are compared with one another.<\/p>\n The opportunity is to open case law to everyone, not only the trained few. But access is only half the problem. No amount of expertise can find relationships a citator was not built to detect.<\/p>\n LOIS Citator was built from first principles in an AI-native world. We worked independently of the Duopoly\u2019s corpora to find case relationships no editorial team could map by hand. American case law is simply too vast and voluminous.<\/p>\n Other legaltech companies say their AI \u201csupplements\u201d Westlaw and Lexis for case law research. We aim to replace them, first for case law, and then for statutory and regulatory research, and then for the rest of the legacy suite.<\/p>\n At Filevine, we have long believed a better way existed. This project has been on our minds for years. When LLMs began to show real promise in legal research, we wondered whether that time had come. Instead, LLMs have brought with them a tidal wave of fake cases, inaccurate quotes, and unsupported (and often erroneous) reasoning. Reputations have been marred. Monetary sanctions and public admonishment are now commonplace. (See our work on anti-hallucination: https:\/\/www.filevine.com\/blog\/solving-hallucinations-with-lois\/<\/a><\/strong>)<\/p>\n Worse, judges have almost certainly relied on hallucinated research, without anyone ever knowing. Those incidents may never appear in the public record.<\/p>\n Hallucination cannot remain a feature of our justice system. It must be rooted out at every turn, with zero tolerance. The minute we accept hallucinations as the cost of doing business with LLMs, we concede to a justice system shaped not only by verified facts and law, but by probabilistic algorithms.<\/p>\n And yet \u2026 AI holds such promise. We can all see it. Broad adoption is already here. There is no going back. Its over. Just as in the software industry, where AI landed first, legal has already hit saturation at a rate that feels magical and bewildering. It turns out lawyers were never Luddites. They just didn\u2019t like technology that was hard to learn, technology that required a whole semester\u2019s course in law school to figure out. They had big jobs to do, and updating document metadata wasn\u2019t one of them. AI can, and should, be a boon to justice. At Filevine, we measure our addressable market not by seats or tokens, but by the gap between how much justice the world needs and how much it has today. AI may be everywhere. But justice is not.<\/p>\n So our dogged team, led by John, Matej, Marek, David, Jonas, Michael, Dan, and others, built (1) anti-hallucination tooling for any usage of opinions by LOIS and (2) a state-of-the-art AI citator, which we call LOIS Citator. And it\u2019s phenomenal.<\/p>\n The development of this tool was expensive and extremely challenging. John will share some details on it next week. But today, we are giving it to the world to try for free. Between now and LEX, Filevine\u2019s flagship legal conference this October, any legal professional can use LOIS Citator and the rest of our anti-hallucination tools free of charge. I\u2019m sure you will find some gaps. For now, we\u2019re only focused on American case law. We welcome your feedback.<\/p>\n LOIS Citator isn\u2019t perfect. We want your feedback. But as of today, we\u2019re confident you will find cases Westlaw and Lexis will miss. One case can make all the difference for a client in need. We are thrilled to share these tools with you.<\/p>\n Welcome to LOIS for Legal Research.<\/a><\/p>\n Ryan Anderson The post Open Letter To Users Of Westlaw And LexisNexis<\/a> appeared first on Above the Law<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" Lexis and Westlaw have been best in class in legal research. As of today, that is no longer true. Your skepticism is welcome. Lawyers are paid to be right. But they are expected to not be wrong. You are justifiably hesitant to experiment. But you should experiment. Your competitors already are. I am not arguing […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-164348","post","type-post","status-publish","format-standard","hentry","category-above_the_law"],"_links":{"self":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts\/164348","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/comments?post=164348"}],"version-history":[{"count":0,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts\/164348\/revisions"}],"wp:attachment":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/media?parent=164348"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/categories?post=164348"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/tags?post=164348"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n
\n
\n
\n
\n
CEO, Filevine<\/p>\n\n
\n
\n
\n
\n
\n
CEO, Filevine<\/p>\n