{"id":108956,"date":"2025-02-21T15:55:27","date_gmt":"2025-02-21T23:55:27","guid":{"rendered":"https:\/\/xira.com\/p\/2025\/02\/21\/linklaters-put-ai-through-law-exams\/"},"modified":"2025-02-21T15:55:27","modified_gmt":"2025-02-21T23:55:27","slug":"linklaters-put-ai-through-law-exams","status":"publish","type":"post","link":"https:\/\/xira.com\/p\/2025\/02\/21\/linklaters-put-ai-through-law-exams\/","title":{"rendered":"Linklaters Put AI Through Law Exams"},"content":{"rendered":"
\"\"
<\/figcaption><\/figure>\n

While law firms struggle to keep a lid on AI security threats<\/a> and otherwise control lawyers using AI out of turn<\/a>, Linklaters has a forward-thinking solution that more firms should embrace. As RollOnFriday notes<\/a>, Linklaters puts its prospective robot lawyers through a UK law exam of their own design, crafting 50 questions from 10 different practice areas geared toward a \u201ccompetent mid-level lawyer specialised in that practice area\u201d and asked the AI to hash it out.<\/p>\n

It\u2019s the Baby Bar for robots. So, like, a Nano Bar? Whatever it is, it\u2019s working better than the California Bar<\/a>.<\/p>\n

It\u2019s not like Linklaters is alone on this. Tech professionals across Biglaw perform tests like these all the time. The distinction is that Linklaters is doing this out in the open. And senior lawyers graded the responses for substance, introducing important stakeholders to the AI evaluation process that can get overlooked when tools are considered exclusively by IT teams and firm tech committees. Which isn\u2019t a knock on the conscientious approach firm staff and tech-savvy attorneys bring to these decisions but\u2026 sometimes you need to take these issues outside the nerd circle.<\/p>\n

The good news for firms is that AI is getting better:<\/p>\n

\n

Linklaters noted there was a \u201csignificant improvement\u201d in the results when compared with tests it ran in October 2023. The firm said that the AI models are starting to perform at a level where they should be able to assist in legal research, such as providing a first draft or \u201cas a cross-check on an existing answer,\u201d and could also be useful \u201cfor tasks that involve summarising relatively well-known areas of law\u201d. \u00a0<\/p>\n<\/blockquote>\n

OpenAI delivered the best-performing AI model on the Linklaters test. That said, the tool only scored 6.4 out of 10 and bet-the-company lawyering doesn\u2019t grade on a curve. The report doesn\u2019t provide Grok\u2019s score, but based on its stated approach to legal reasoning<\/a>, we\u2019re guessing it made OpenAI look like William Blackstone crawled out of the grave for a spot of tea and a run at some contracts.<\/p>\n

But putting AI through the wringer in such a public way signals to lawyers that the firm both sees potential in AI while still making the clear case to everyone that AI isn\u2019t ready for prime time.<\/p>\n

This approach should push back against the \u201cshadow AI economy\u201d problem Hill Dickinson recently addressed. When firms downplay or outright shun AI, unruly lawyers are going to start experimenting with AI tools on their own. And that\u2019s how you end up with confidential client data being uploaded to random servers in hostile countries. Transparency keeps everyone on the same page when it comes to when and where AI fits into a modern legal practice.<\/p>\n

Linklaters makes AI sit law exams<\/a> [RollOnFriday]<\/p>\n

\n

\"Headshot\"Joe Patrice<\/a>\u00a0is a senior editor at Above the Law and co-host of Thinking Like A Lawyer<\/a>. Feel free to\u00a0email<\/a> any tips, questions, or comments. Follow him on\u00a0Twitter<\/a>\u00a0or Bluesky<\/a> if you\u2019re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search<\/a>.<\/em><\/strong><\/p>\n

The post Linklaters Put AI Through Law Exams<\/a> appeared first on Above the Law<\/a>.<\/p>\n

\"\"
<\/figcaption><\/figure>\n

While law firms struggle to keep a lid on AI security threats<\/a> and otherwise control lawyers using AI out of turn<\/a>, Linklaters has a forward-thinking solution that more firms should embrace. As RollOnFriday notes<\/a>, Linklaters puts its prospective robot lawyers through a UK law exam of their own design, crafting 50 questions from 10 different practice areas geared toward a \u201ccompetent mid-level lawyer specialised in that practice area\u201d and asked the AI to hash it out.<\/p>\n

It\u2019s the Baby Bar for robots. So, like, a Nano Bar? Whatever it is, it\u2019s working better than the California Bar<\/a>.<\/p>\n

It\u2019s not like Linklaters is alone on this. Tech professionals across Biglaw perform tests like these all the time. The distinction is that Linklaters is doing this out in the open. And senior lawyers graded the responses for substance, introducing important stakeholders to the AI evaluation process that can get overlooked when tools are considered exclusively by IT teams and firm tech committees. Which isn\u2019t a knock on the conscientious approach firm staff and tech-savvy attorneys bring to these decisions but\u2026 sometimes you need to take these issues outside the nerd circle.<\/p>\n

The good news for firms is that AI is getting better:<\/p>\n

\n

Linklaters noted there was a \u201csignificant improvement\u201d in the results when compared with tests it ran in October 2023. The firm said that the AI models are starting to perform at a level where they should be able to assist in legal research, such as providing a first draft or \u201cas a cross-check on an existing answer,\u201d and could also be useful \u201cfor tasks that involve summarising relatively well-known areas of law\u201d. \u00a0<\/p>\n<\/blockquote>\n

OpenAI delivered the best-performing AI model on the Linklaters test. That said, the tool only scored 6.4 out of 10 and bet-the-company lawyering doesn\u2019t grade on a curve. The report doesn\u2019t provide Grok\u2019s score, but based on its stated approach to legal reasoning<\/a>, we\u2019re guessing it made OpenAI look like William Blackstone crawled out of the grave for a spot of tea and a run at some contracts.<\/p>\n

But putting AI through the wringer in such a public way signals to lawyers that the firm both sees potential in AI while still making the clear case to everyone that AI isn\u2019t ready for prime time.<\/p>\n

This approach should push back against the \u201cshadow AI economy\u201d problem Hill Dickinson recently addressed. When firms downplay or outright shun AI, unruly lawyers are going to start experimenting with AI tools on their own. And that\u2019s how you end up with confidential client data being uploaded to random servers in hostile countries. Transparency keeps everyone on the same page when it comes to when and where AI fits into a modern legal practice.<\/p>\n

Linklaters makes AI sit law exams<\/a> [RollOnFriday]<\/p>\n

\n

\"Headshot\"Joe Patrice<\/a>\u00a0is a senior editor at Above the Law and co-host of Thinking Like A Lawyer<\/a>. Feel free to\u00a0email<\/a> any tips, questions, or comments. Follow him on\u00a0Twitter<\/a>\u00a0or Bluesky<\/a> if you\u2019re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search<\/a>.<\/em><\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

While law firms struggle to keep a lid on AI security threats and otherwise control lawyers using AI out of turn, Linklaters has a forward-thinking solution that more firms should embrace. As RollOnFriday notes, Linklaters puts its prospective robot lawyers through a UK law exam of their own design, crafting 50 questions from 10 different […]<\/p>\n","protected":false},"author":3,"featured_media":108918,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[16],"tags":[],"class_list":["post-108956","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-above_the_law"],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/xira.com\/p\/wp-content\/uploads\/2025\/02\/Headshot-300x200-cdrdkk.jpeg?fit=300%2C200&ssl=1","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts\/108956","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/comments?post=108956"}],"version-history":[{"count":0,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/posts\/108956\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/media\/108918"}],"wp:attachment":[{"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/media?parent=108956"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/categories?post=108956"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xira.com\/p\/wp-json\/wp\/v2\/tags?post=108956"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}