We read about it every day. A lawyer uses a large language model (LLM) to do some research. They copy that research into a brief, but the research contains cases that don\u2019t exist. The lawyer is busted, the judge furious, and the client starts looking for a better lawyer.<\/p>\n
It has everyone scratching their heads. I mean, everyone knows the AI systems will do this, so why does it keep happening? A new\u00a0Cornell University\u00a0study and paper<\/a>\u00a0sheds some light on this, the problem of overreliance, and why the volcano of serious AI flaws may be about to erupt. Quite simply, the cost of verifying the results of the AI tools exceeds any savings from their use. It\u2019s a paradox.<\/p>\n In Part I of an examination of the\u00a0why a volcano of AI problems may be about to erupt,\u00a0I looked at the dangers of overreliance on AI given the gaps in the underlying infrastructure. But there\u2019s more to the story.\u00a0The simple fact is that AI\u00a0tools have fundamental reality and transparency flaws is risky and downright foolhardy. Given the profound and breadth of the impact of these flaws and the corresponding\u00a0cost\u00a0to verify outputs, the use\u00a0and\u00a0role of AI in legal may\u00a0end up\u00a0being\u00a0more limited than many think.<\/p>\n The Assumptions<\/strong><\/p>\n As pointed\u00a0out in the study, the assumption fueling the explosion of AI use in legal is that will save gobs of time. This savings will inure to the benefit of lawyers and clients, will lead to fairer methods of billing like alternative fee structures, will get better results, improve access to justice, and lead to world peace. Well,\u00a0maybe even the vendors would not go so far as to guarantee the last one. But vendors do seem to be guaranteeing everything but that.\u00a0And pundits talk as if AI will transform legal from the ground up.\u00a0Law firms are buying into the hype, investing in expensive systems that do things they barely understand.\u00a0<\/p>\n But not so fast. All this hinges on the assumption that the time saved will vastly exceed the additional steps needed to verify the output and that any issues of AI with things like accuracy will soon be solved.\u00a0<\/p>\n The Cornell study throws some cold water on all these assumptions\u00a0and challenges them head on.<\/p>\n The Cornell Study<\/strong><\/p>\n The study identifies two\u00a0fundamental LLM\u00a0flaws. The first we all know about: the propensity of the systems to hallucinate and provide inaccurate information. The study refers to this flaw as a reality flaw. It\u2019s a big problem in a profession like law where being wrong can have severe consequences. The second flaw identified\u00a0by the study it calls\u00a0a\u00a0transparency\u00a0one. We don\u2019t really know how these systems work.<\/p>\n The reality flaw,\u00a0says the study,\u00a0stems from the fact that generative systems \u201care not structurally linked to reality: namely factual accuracy\u2026a machine learning model does not learn the facts underlying the training data but reduces that data to patterns which it then ingests and seeks to reproduce.\u201d And the study notes that it\u2019s not just the public systems like ChatGPT that demonstrate this flaw, it\u2019s also the ones built for legal as well.<\/p>\n So, the study concludes, \u201cany output generated by AI must be verified if the user wishes to satisfy themselves as to the accuracy and connection to reality, of that output\u2014especially in legal practice.\u201d In other words, check\u00a0your\u00a0cites.<\/p>\n The second flaw,\u00a0one of transparency, is\u00a0the black box problem. It\u00a0in turn creates a\u00a0trust\u00a0issue,\u00a0says the study. If you don\u2019t know how a decision is made or a conclusion is reached, how can you trust it?\u00a0<\/p>\n For\u00a0a legal\u00a0system\u00a0that depends on reasoning and logic,\u00a0that\u2019s\u00a0a big issue.\u00a0I would phrase it this way:\u00a0how can you rely on something when you don\u2019t know how it works, how it reached the decision it reached, and you get different answers to the same questions.<\/p>\n Use of\u00a0AI\u00a0in legal hinges on the need\u00a0to\u00a0be able to explain how a decision was reached. That\u2019s a cornerstone of how legal processes and even the rule of law is based.<\/p>\n The study further concludes that neither of these flaws will be overcome anytime soon.<\/p>\n What Does This Mean?<\/strong><\/p>\n The study goes on to talk about what this means. It suggests that the plethora of cases where lawyers\u00a0have\u00a0failed\u00a0to check cites and end up having a hallucinated\u00a0or inaccurate\u00a0case or facts recited\u00a0in filings means lawyers\u00a0are\u00a0underestimating\u00a0the flaws. Or\u00a0have been\u00a0convinced by providers that the risks are negligible.\u00a0<\/p>\n These lawyers\u00a0have simply overrelied on a tool they believed or were led or lulled\u00a0into\u00a0believing\u00a0was more accurate than what it is.\u00a0The result so far has been a\u00a0great hue and cry by everyone that you\u00a0have to\u00a0check cites. Usually this is delivered with a wry grin that says it\u2019s just dumb and lazy lawyers to blame.\u00a0But the fact is the problem is not going away. In\u00a0fact,\u00a0it seems to be getting worse.<\/p>\n It may be that the guilty lawyers are dumb or lazy,\u00a0although as I have\u00a0written before<\/a>, that\u2019s not the whole story.\u00a0But what\u2019s left\u00a0unsaid\u00a0is something the study points out: \u201cthe net value of an AI model in legal practice can only be assessed once the efficiency gain (savings on time, salary costs, firm resource costs, etc.) is offset by the corresponding verification cost (cost to manually verify AI outputs for accuracy, completeness, relevance, etc.).\u00a0Those caught with hallucinated cases in their papers simply didn\u2019t take the time to verify relying on the AI tool.<\/p>\n Because the demand for accuracy in legal is so high, the study notes, the verification cost for many actions in legal is too high to offset the savings.\u00a0The study also concludes that this cost is not ameliorated by automated systems since the reality and transparency risks may still exist. Hence\u00a0what\u00a0the\u00a0study calls a\u00a0verification paradox.<\/p>\n And we see\u00a0the impact of this paradox\u00a0already with fines imposed by courts for hallucinated cases. We will no doubt see malpractice and ethical violation claims. The cost of being wrong in law is just too great to not verify and verify thoroughly.\u00a0<\/p>\n Granted, AI can do lots of things well where the risks of being wrong are not that great. It will have an enormous impact in business and maybe other professions. But for law, not so much: \u201cThe more important the output, the more important it is to verify its accuracy.\u201d<\/p>\n The study concludes:<\/p>\n The verification-value paradox suggests the net value of AI to legal practice is grossly\u00a0overestimated, due to an underestimation of the verification cost. A proper understanding\u00a0of\u00a0the\u00a0costly and essential nature of verification leads to the conclusion that AI\u2019s net value\u00a0<\/strong>will often be\u00a0negligible in legal practice: that is, in most cases, the value added will not be sufficient to justify\u00a0the corresponding verification cost.<\/p>\n<\/blockquote>\n The Reality<\/strong><\/strong><\/p>\n It\u2019s easy to see the economic impact of the verification paradox when you compare the cost of getting a piece of work done by an LLM with that done by a human. Let\u2019s assume you ask an LLM to do some legal research that would normally take you 10 hours. You get the result, but it\u2019s got some 25 case citations. Now you have to a) check to make sure every case exists and b) make sure that the case stands for the proposition the LLM says it does. By the time you do that, you could very well spend the eight hours, if not more.<\/p>\n A\u00a0<\/strong>Volcano About to Erupt<\/strong>?<\/strong><\/p>\n It may be too late to completely put\u00a0AI\u00a0back in the bottle. But where it takes just as long if not longer to verify the results of an AI tool you\u2019ve spent thousands of dollars on, you\u2019re not going be predisposed to buy more. Certainly, your clients aren\u2019t going to be wild about your use of a tool that not only fails to save them money but costs them more and exposes them to risk.<\/p>\n It\u2019s easy to envision the fundamental conclusion that using AI for many things is not worth the risk and the cost of validating its result. It\u2019s easy to\u00a0see how\u00a0this fact\u00a0will\u00a0temper\u00a0the\u00a0enthusiasm and reliance on AI.\u00a0<\/p>\n We\u00a0may rapidly conclude\u00a0the costs and risks of doing so are too high\u00a0and simply not worth it in the long- and perhaps even the short-run. When that happens, a lot of lawyers are going to be caught with expensive systems that they don\u2019t need. A lot of vendors may have to go in other directions. A lot of venture capital may go down the drain. The proverbial volcano may be about to erupt.<\/p>\n That\u2019s something worth considering before you buy the next shiny new AI toy\u00a0and\u00a0before you use AI shortcuts to\u00a0do\u00a0the hard work, before you blindly expect people you supervise to do the right thing and before you accept without question their work.<\/p>\n In the meantime, check your citations. Please.<\/p>\n Stephen Embry is a lawyer, speaker, blogger, and writer. He publishes\u00a0TechLaw Crossroads<\/a>, a blog devoted to the examination of the tension between technology, the law, and the practice of law<\/strong><\/em>.\u00a0<\/p>\n Melissa Rogozinski<\/a>\u00a0is CEO of the RPC Round Table and RPC Strategies, LLC, a marketing and advertising firm in Miami, FL.\u00a0<\/em><\/strong><\/p>\n The post Like Lawyers In\u00a0Pompeii:\u00a0Is\u00a0Legal Ignoring The Coming\u00a0AI Crisis? (Part II)<\/a> appeared first on Above the Law<\/a>.<\/p>\n We read about it every day. A lawyer uses a large language model (LLM) to do some research. They copy that research into a brief, but the research contains cases that don\u2019t exist. The lawyer is busted, the judge furious, and the client starts looking for a better lawyer.<\/p>\n It has everyone scratching their heads. I mean, everyone knows the AI systems will do this, so why does it keep happening? A new\u00a0Cornell University\u00a0study and paper<\/a>\u00a0sheds some light on this, the problem of overreliance, and why the volcano of serious AI flaws may be about to erupt. Quite simply, the cost of verifying the results of the AI tools exceeds any savings from their use. It\u2019s a paradox.<\/p>\n In Part I of an examination of the\u00a0why a volcano of AI problems may be about to erupt,\u00a0I looked at the dangers of overreliance on AI given the gaps in the underlying infrastructure. But there\u2019s more to the story.\u00a0The simple fact is that AI\u00a0tools have fundamental reality and transparency flaws is risky and downright foolhardy. Given the profound and breadth of the impact of these flaws and the corresponding\u00a0cost\u00a0to verify outputs, the use\u00a0and\u00a0role of AI in legal may\u00a0end up\u00a0being\u00a0more limited than many think.<\/p>\n The Assumptions<\/strong><\/p>\n As pointed\u00a0out in the study, the assumption fueling the explosion of AI use in legal is that will save gobs of time. This savings will inure to the benefit of lawyers and clients, will lead to fairer methods of billing like alternative fee structures, will get better results, improve access to justice, and lead to world peace. Well,\u00a0maybe even the vendors would not go so far as to guarantee the last one. But vendors do seem to be guaranteeing everything but that.\u00a0And pundits talk as if AI will transform legal from the ground up.\u00a0Law firms are buying into the hype, investing in expensive systems that do things they barely understand.\u00a0<\/p>\n But not so fast. All this hinges on the assumption that the time saved will vastly exceed the additional steps needed to verify the output and that any issues of AI with things like accuracy will soon be solved.\u00a0<\/p>\n The Cornell study throws some cold water on all these assumptions\u00a0and challenges them head on.<\/p>\n The Cornell Study<\/strong><\/p>\n The study identifies two\u00a0fundamental LLM\u00a0flaws. The first we all know about: the propensity of the systems to hallucinate and provide inaccurate information. The study refers to this flaw as a reality flaw. It\u2019s a big problem in a profession like law where being wrong can have severe consequences. The second flaw identified\u00a0by the study it calls\u00a0a\u00a0transparency\u00a0one. We don\u2019t really know how these systems work.<\/p>\n The reality flaw,\u00a0says the study,\u00a0stems from the fact that generative systems \u201care not structurally linked to reality: namely factual accuracy\u2026a machine learning model does not learn the facts underlying the training data but reduces that data to patterns which it then ingests and seeks to reproduce.\u201d And the study notes that it\u2019s not just the public systems like ChatGPT that demonstrate this flaw, it\u2019s also the ones built for legal as well.<\/p>\n So, the study concludes, \u201cany output generated by AI must be verified if the user wishes to satisfy themselves as to the accuracy and connection to reality, of that output\u2014especially in legal practice.\u201d In other words, check\u00a0your\u00a0cites.<\/p>\n The second flaw,\u00a0one of transparency, is\u00a0the black box problem. It\u00a0in turn creates a\u00a0trust\u00a0issue,\u00a0says the study. If you don\u2019t know how a decision is made or a conclusion is reached, how can you trust it?\u00a0<\/p>\n For\u00a0a legal\u00a0system\u00a0that depends on reasoning and logic,\u00a0that\u2019s\u00a0a big issue.\u00a0I would phrase it this way:\u00a0how can you rely on something when you don\u2019t know how it works, how it reached the decision it reached, and you get different answers to the same questions.<\/p>\n Use of\u00a0AI\u00a0in legal hinges on the need\u00a0to\u00a0be able to explain how a decision was reached. That\u2019s a cornerstone of how legal processes and even the rule of law is based.<\/p>\n The study further concludes that neither of these flaws will be overcome anytime soon.<\/p>\n What Does This Mean?<\/strong><\/p>\n The study goes on to talk about what this means. It suggests that the plethora of cases where lawyers\u00a0have\u00a0failed\u00a0to check cites and end up having a hallucinated\u00a0or inaccurate\u00a0case or facts recited\u00a0in filings means lawyers\u00a0are\u00a0underestimating\u00a0the flaws. Or\u00a0have been\u00a0convinced by providers that the risks are negligible.\u00a0<\/p>\n These lawyers\u00a0have simply overrelied on a tool they believed or were led or lulled\u00a0into\u00a0believing\u00a0was more accurate than what it is.\u00a0The result so far has been a\u00a0great hue and cry by everyone that you\u00a0have to\u00a0check cites. Usually this is delivered with a wry grin that says it\u2019s just dumb and lazy lawyers to blame.\u00a0But the fact is the problem is not going away. In\u00a0fact,\u00a0it seems to be getting worse.<\/p>\n It may be that the guilty lawyers are dumb or lazy,\u00a0although as I have\u00a0written before<\/a>, that\u2019s not the whole story.\u00a0But what\u2019s left\u00a0unsaid\u00a0is something the study points out: \u201cthe net value of an AI model in legal practice can only be assessed once the efficiency gain (savings on time, salary costs, firm resource costs, etc.) is offset by the corresponding verification cost (cost to manually verify AI outputs for accuracy, completeness, relevance, etc.).\u00a0Those caught with hallucinated cases in their papers simply didn\u2019t take the time to verify relying on the AI tool.<\/p>\n Because the demand for accuracy in legal is so high, the study notes, the verification cost for many actions in legal is too high to offset the savings.\u00a0The study also concludes that this cost is not ameliorated by automated systems since the reality and transparency risks may still exist. Hence\u00a0what\u00a0the\u00a0study calls a\u00a0verification paradox.<\/p>\n And we see\u00a0the impact of this paradox\u00a0already with fines imposed by courts for hallucinated cases. We will no doubt see malpractice and ethical violation claims. The cost of being wrong in law is just too great to not verify and verify thoroughly.\u00a0<\/p>\n Granted, AI can do lots of things well where the risks of being wrong are not that great. It will have an enormous impact in business and maybe other professions. But for law, not so much: \u201cThe more important the output, the more important it is to verify its accuracy.\u201d<\/p>\n The study concludes:<\/p>\n The verification-value paradox suggests the net value of AI to legal practice is grossly\u00a0overestimated, due to an underestimation of the verification cost. A proper understanding\u00a0of\u00a0the\u00a0costly and essential nature of verification leads to the conclusion that AI\u2019s net valuewill often be\u00a0negligible in legal practice: that is, in most cases, the value added will not be sufficient to justify\u00a0the corresponding verification cost.<\/p>\n<\/blockquote>\n The Reality<\/strong><\/p>\n It\u2019s easy to see the economic impact of the verification paradox when you compare the cost of getting a piece of work done by an LLM with that done by a human. Let\u2019s assume you ask an LLM to do some legal research that would normally take you 10 hours. You get the result, but it\u2019s got some 25 case citations. Now you have to a) check to make sure every case exists and b) make sure that the case stands for the proposition the LLM says it does. By the time you do that, you could very well spend the eight hours, if not more.<\/p>\n A\u00a0<\/strong>Volcano About to Erupt<\/strong>?<\/strong><\/p>\n It may be too late to completely put\u00a0AI\u00a0back in the bottle. But where it takes just as long if not longer to verify the results of an AI tool you\u2019ve spent thousands of dollars on, you\u2019re not going be predisposed to buy more. Certainly, your clients aren\u2019t going to be wild about your use of a tool that not only fails to save them money but costs them more and exposes them to risk.<\/p>\n It\u2019s easy to envision the fundamental conclusion that using AI for many things is not worth the risk and the cost of validating its result. It\u2019s easy to\u00a0see how\u00a0this fact\u00a0will\u00a0temper\u00a0the\u00a0enthusiasm and reliance on AI.\u00a0<\/p>\n We\u00a0may rapidly conclude\u00a0the costs and risks of doing so are too high\u00a0and simply not worth it in the long- and perhaps even the short-run. When that happens, a lot of lawyers are going to be caught with expensive systems that they don\u2019t need. A lot of vendors may have to go in other directions. A lot of venture capital may go down the drain. The proverbial volcano may be about to erupt.<\/p>\n That\u2019s something worth considering before you buy the next shiny new AI toy\u00a0and\u00a0before you use AI shortcuts to\u00a0do\u00a0the hard work, before you blindly expect people you supervise to do the right thing and before you accept without question their work.<\/p>\n In the meantime, check your citations. Please.<\/p>\n Stephen Embry is a lawyer, speaker, blogger, and writer. He publishes\u00a0TechLaw Crossroads<\/a>, a blog devoted to the examination of the tension between technology, the law, and the practice of law<\/strong><\/em>.\u00a0<\/p>\n\n
\n\n
\n