We talk a lot about the ethical duty of lawyers and legal professionals to understand the risks and benefits of relevant technology. But when it comes to using GenAI, that might not be enough. If we want to prevent the increasing number of hallucinations and inaccurate citations that are bedeviling lawyers and even judges, we need to understand how and why GenAI systems fail.<\/p>\n
That was the point of a recent paper<\/a> by a group of scientists and engineers: Dylan Restrepo<\/a>, Nicholas Restrepo<\/a>, Frank Huo<\/a>, and Neil Johnson<\/a>. The paper carried the lengthy title, When AI Output Trips to Bad but Nobody Notices: Legal Implications of AI\u2019s Mistakes<\/em>. In addition to their own calculations and analysis, the group also consulted a couple of lawyers: Daniela Restrepo<\/a> and Jean Paul Roekaert<\/a>. I can\u2019t vouch for the mathematical calculations but what they conclude squares with my own experience.<\/p>\n The Basic Premise<\/strong><\/p>\n The group concludes at the outset that rather than a random, unpredictable glitch, a physics-based analysis demonstrates that hallucination is a \u201cforeseeable engineering risk.\u201d Meaning, of course, the circumstances generating its occurrence can be at least a little predictable.<\/p>\n According to the paper, GenAI systems have \u201ca deterministic mechanism at its core that can cause output to flip from reliable to fabricated at a calculable step.\u201d\u00a0 And that step unfortunately comes when the lawyer\u2019s need is the greatest.<\/p>\n The group\u2019s analysis starts from the proposition that we should know by now: GenAI is \u201ca probabilistic text generator engineered to predict the next most plausible token in a sequence, without any internal concept of legal truth.\u201d It is not, argues the group, a database of verified legal authorities. (The group focused on the publicly available systems and not on the closed systems that claim to rely on verified legal authorities.)<\/p>\n What This Means<\/strong><\/p>\n Because it\u2019s predicting, not analyzing, GenAI does well when faced with inquiries about valid legal principles, logical-sounding arguments, undisputed case facts, procedural history, and the like. But when faced with something novel and complex, the tool is pushed \u201cinto a region where training data is sparse.\u201d In an effort to please and respond, it is then prone to, well, make stuff up.<\/p>\n The paper puts it this way:<\/p>\n The tool is therefore most prone to failure exactly when the lawyer\u2019s need is greatest: on a difficult point of law with sparse precedent. The act of researching an unsettled legal issue via an LLM becomes the principal trigger for the tipping instability.<\/p>\n<\/blockquote>\n These are important points since lawyers live in a world where a hallucination, an error, can have devastating consequences. So, as we have discussed<\/a>, given that risk, GenAI outputs must be checked over and over, often mitigating the cost savings of using the tools in the first place. But if we understand why the errors occur and more importantly when, we can better and more safely use the tools.<\/p>\n A Blessing\u2026And a Curse<\/strong><\/p>\n If true, then the group\u2019s findings are a blessing since it suggests a sliding scale of verification: less where the output focuses on well-known information and much more when it strays into the novel. Saves time and energy.<\/p>\n But for those uninformed of this predictability, the fact that failure can occur at a certain point can be a curse. Why: a lawyer with a legal project often starts with undisputed facts, then seeks information on what the law generally is with respect to the issues at hand. And then goes to more complex, ambiguous areas thinking it\u2019s okay.<\/p>\n The example given in the paper is a statute of limitations question. A lawyer starts their use of ChatGPT by plugging in undisputed facts. They then seek the general law with respect to the limitation period. All well and good: \u00a0the lawyer gets correct responses and then, in the words of the paper, \u201cgains confidence in the tool.\u201d\u00a0 So, the lawyer then begins asking for more ambiguous information about how that law can be used to leverage the facts or to develop arguments.<\/p>\n If the lawyer takes all the outputs and prepares a brief based on the information obtained, they (or their supervisor) might be tempted to spot check the first few paragraphs, find nothing amiss and, when pressed for time, conclude the rest of the outputs are also fine when they are not.<\/p>\n So, the blessing becomes a curse: \u201cAI\u2019s period of correct output increases rather than decreases the risk of harm, because it builds the user\u2019s trust just before the fabrication appears.\u201d<\/p>\n What To Do<\/strong><\/p>\n So, what do we make of all this? Again, I\u2019m no scientist but I do know from experience that the more general information I seek from GenAI, the more prone it is to be correct. When I stray into more ambiguous areas where there is less known about a subject, the more errors I tend to get.<\/p>\n For example, I once asked for information about a well-known painter. I got great information. But when I asked about another painter in the same school of painting who was relatively obscure, the tool just made up a name. Or when I asked what\u2019s the\u00a0 subway stop to take to catch the Q70 bus to LaGuardia Airport, it got it right. When I asked the best route from my hotel (which involves more ambiguity), it sent me to the wrong stop. It did say sorry when I pointed out the error (after some argument).<\/p>\n The point being for lawyers and legal professionals is to understand that \u201cAI possesses no independent legal agency: it is a computational tool.\u201d Granted, it is a computational tool with which you can converse like a human. It reacts in human ways. It\u2019s tempting to anthropomorphize it.<\/p>\n But that\u2019s where we go wrong. Instead, we need to start with thinking of it not as a person but a product with a foreseeable engineering risk. Like a sharp knife or an ATV. A risk that appears to materialize when faced with novelty and ambiguity.\u00a0 But it\u2019s that novelty and ambiguity that creates the greatest risk of hallucination, according to the paper.<\/p>\n For lawyers, that means if you are going to use this sharp knife, you better know how and in what circumstances. You need to know how to do that safely.<\/p>\n The paper says it the best: \u201cThe duty of technological competence, as expressed in ABA Model Rule 1.1 and its state-level counterparts, must evolve. It is no longer sufficient for a lawyer to know how to operate a piece of software. Competence now requires a practical understanding of how that software can fail.\u201d That it is clearly right about.<\/p>\n Want to use GenAI? Use it to access known information that would be time consuming or difficult to otherwise get. Ask it to do a lot of things where accuracy isn\u2019t that important. But don\u2019t ask novel or unsettled legal questions, without checking and double checking what you get back. Else you might get off at the wrong subway stop.<\/p>\n Or much worse.<\/p>\n Stephen Embry is a lawyer, speaker, blogger, and writer. He publishes\u00a0TechLaw Crossroads<\/a>, a blog devoted to the examination of the tension between technology, the law, and the practice of law<\/strong><\/em>.<\/p>\n The post Understanding AI Hallucinations: Making Sure You Don\u2019t End Up At The Wrong Stop<\/a> appeared first on Above the Law<\/a>.<\/p>\n We talk a lot about the ethical duty of lawyers and legal professionals to understand the risks and benefits of relevant technology. But when it comes to using GenAI, that might not be enough. If we want to prevent the increasing number of hallucinations and inaccurate citations that are bedeviling lawyers and even judges, we need to understand how and why GenAI systems fail.<\/p>\n That was the point of a recent paper<\/a> by a group of scientists and engineers: Dylan Restrepo<\/a>, Nicholas Restrepo<\/a>, Frank Huo<\/a>, and Neil Johnson<\/a>. The paper carried the lengthy title, When AI Output Trips to Bad but Nobody Notices: Legal Implications of AI\u2019s Mistakes<\/em>. In addition to their own calculations and analysis, the group also consulted a couple of lawyers: Daniela Restrepo<\/a> and Jean Paul Roekaert<\/a>. I can\u2019t vouch for the mathematical calculations but what they conclude squares with my own experience.<\/p>\n The Basic Premise<\/strong><\/p>\n The group concludes at the outset that rather than a random, unpredictable glitch, a physics-based analysis demonstrates that hallucination is a \u201cforeseeable engineering risk.\u201d Meaning, of course, the circumstances generating its occurrence can be at least a little predictable.<\/p>\n According to the paper, GenAI systems have \u201ca deterministic mechanism at its core that can cause output to flip from reliable to fabricated at a calculable step.\u201d\u00a0 And that step unfortunately comes when the lawyer\u2019s need is the greatest.<\/p>\n The group\u2019s analysis starts from the proposition that we should know by now: GenAI is \u201ca probabilistic text generator engineered to predict the next most plausible token in a sequence, without any internal concept of legal truth.\u201d It is not, argues the group, a database of verified legal authorities. (The group focused on the publicly available systems and not on the closed systems that claim to rely on verified legal authorities.)<\/p>\n What This Means<\/strong><\/p>\n Because it\u2019s predicting, not analyzing, GenAI does well when faced with inquiries about valid legal principles, logical-sounding arguments, undisputed case facts, procedural history, and the like. But when faced with something novel and complex, the tool is pushed \u201cinto a region where training data is sparse.\u201d In an effort to please and respond, it is then prone to, well, make stuff up.<\/p>\n The paper puts it this way:<\/p>\n The tool is therefore most prone to failure exactly when the lawyer\u2019s need is greatest: on a difficult point of law with sparse precedent. The act of researching an unsettled legal issue via an LLM becomes the principal trigger for the tipping instability.<\/p>\n<\/blockquote>\n These are important points since lawyers live in a world where a hallucination, an error, can have devastating consequences. So, as we have discussed<\/a>, given that risk, GenAI outputs must be checked over and over, often mitigating the cost savings of using the tools in the first place. But if we understand why the errors occur and more importantly when, we can better and more safely use the tools.<\/p>\n A Blessing\u2026And a Curse<\/strong><\/p>\n If true, then the group\u2019s findings are a blessing since it suggests a sliding scale of verification: less where the output focuses on well-known information and much more when it strays into the novel. Saves time and energy.<\/p>\n But for those uninformed of this predictability, the fact that failure can occur at a certain point can be a curse. Why: a lawyer with a legal project often starts with undisputed facts, then seeks information on what the law generally is with respect to the issues at hand. And then goes to more complex, ambiguous areas thinking it\u2019s okay.<\/p>\n The example given in the paper is a statute of limitations question. A lawyer starts their use of ChatGPT by plugging in undisputed facts. They then seek the general law with respect to the limitation period. All well and good: \u00a0the lawyer gets correct responses and then, in the words of the paper, \u201cgains confidence in the tool.\u201d\u00a0 So, the lawyer then begins asking for more ambiguous information about how that law can be used to leverage the facts or to develop arguments.<\/p>\n If the lawyer takes all the outputs and prepares a brief based on the information obtained, they (or their supervisor) might be tempted to spot check the first few paragraphs, find nothing amiss and, when pressed for time, conclude the rest of the outputs are also fine when they are not.<\/p>\n So, the blessing becomes a curse: \u201cAI\u2019s period of correct output increases rather than decreases the risk of harm, because it builds the user\u2019s trust just before the fabrication appears.\u201d<\/p>\n What To Do<\/strong><\/p>\n So, what do we make of all this? Again, I\u2019m no scientist but I do know from experience that the more general information I seek from GenAI, the more prone it is to be correct. When I stray into more ambiguous areas where there is less known about a subject, the more errors I tend to get.<\/p>\n For example, I once asked for information about a well-known painter. I got great information. But when I asked about another painter in the same school of painting who was relatively obscure, the tool just made up a name. Or when I asked what\u2019s the\u00a0 subway stop to take to catch the Q70 bus to LaGuardia Airport, it got it right. When I asked the best route from my hotel (which involves more ambiguity), it sent me to the wrong stop. It did say sorry when I pointed out the error (after some argument).<\/p>\n The point being for lawyers and legal professionals is to understand that \u201cAI possesses no independent legal agency: it is a computational tool.\u201d Granted, it is a computational tool with which you can converse like a human. It reacts in human ways. It\u2019s tempting to anthropomorphize it.<\/p>\n But that\u2019s where we go wrong. Instead, we need to start with thinking of it not as a person but a product with a foreseeable engineering risk. Like a sharp knife or an ATV. A risk that appears to materialize when faced with novelty and ambiguity.\u00a0 But it\u2019s that novelty and ambiguity that creates the greatest risk of hallucination, according to the paper.<\/p>\n For lawyers, that means if you are going to use this sharp knife, you better know how and in what circumstances. You need to know how to do that safely.<\/p>\n The paper says it the best: \u201cThe duty of technological competence, as expressed in ABA Model Rule 1.1 and its state-level counterparts, must evolve. It is no longer sufficient for a lawyer to know how to operate a piece of software. Competence now requires a practical understanding of how that software can fail.\u201d That it is clearly right about.<\/p>\n Want to use GenAI? Use it to access known information that would be time consuming or difficult to otherwise get. Ask it to do a lot of things where accuracy isn\u2019t that important. But don\u2019t ask novel or unsettled legal questions, without checking and double checking what you get back. Else you might get off at the wrong subway stop.<\/p>\n Or much worse.<\/p>\n Stephen Embry is a lawyer, speaker, blogger, and writer. He publishes\u00a0TechLaw Crossroads<\/a>, a blog devoted to the examination of the tension between technology, the law, and the practice of law<\/strong><\/em>.<\/p>\n The post Understanding AI Hallucinations: Making Sure You Don\u2019t End Up At The Wrong Stop<\/a> appeared first on Above the Law<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" We talk a lot about the ethical duty of lawyers and legal professionals to understand the risks and benefits of relevant technology. But when it comes to using GenAI, that might not be enough. If we want to prevent the increasing number of hallucinations and inaccurate citations that are bedeviling lawyers and even judges, we […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[16],"tags":[],"class_list":["post-148509","post","type-post","status-publish","format-standard","hentry","category-above_the_law"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts\/148509","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/comments?post=148509"}],"version-history":[{"count":0,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts\/148509\/revisions"}],"wp:attachment":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/media?parent=148509"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/categories?post=148509"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/tags?post=148509"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n\n
\n