Picture the update at a legal department meeting: The AI pilot now has 40 users, access to live contracts, and a growing backlog of feature requests. Its creator is on vacation.
The slide still says “experiment.”
Somewhere between the promising demonstration and the vacation coverage problem, the prototype acquired a job. People began relying on it. Business processes adjusted around it. Turning it off became inconvenient.
Nobody remembers approving a permanent system.
This is the scenario legal leaders should keep in mind when someone proposes a quick proof of concept. A useful experiment can become an operational commitment through a series of small, individually reasonable decisions.
The way to manage that progression starts with a question specific enough to answer.
What Are We Trying to Learn?
“Let’s see what AI can do with our contracts” could keep a team occupied indefinitely.
A more useful question might be whether a tool can identify a defined group of provisions accurately enough to help procurement prepare for legal review. Another might ask whether lawyers can verify its answers quickly enough to reduce the total time spent on a matter.
Those are different experiments.
A tool can identify provisions accurately and still create additional work because its citations are difficult to check. Users can love an interface while consistently misunderstanding what its ratings mean. A technically impressive system can depend on documents the company cannot reliably obtain.
Choose the uncertainty that matters to the next business decision. Design the pilot around resolving it.
Give The Experiment A Fair Chance To Disappoint You
A demonstration usually shows the product under favorable conditions. An evaluation needs contact with the company’s actual mess.
Include the scanned agreement, the missing amendment, the exception buried in an exhibit, and the user who has seven minutes before a supplier call.
Decide how errors will be assessed. Missing a routine notice provision and missing a material liability exception may warrant very different responses. An overall accuracy number can hide that distinction.
Measure the human work around the output, too. Someone must prepare documents, check answers, correct mistakes, and handle escalations. Those minutes belong in the calculation.
For legal leaders, this is familiar territory. We already know that the significance of an error depends on its consequences. We should bring that judgment into the evaluation before anyone falls in love with the dashboard.
Decide What Each Result Will Authorize
A pilot should have an end date, a decision owner, and an agreed set of possible next steps.
The evidence might support buying the product. It might reveal a narrow capability worth building internally. It might show that an integration would solve the problem. It might establish that the company needs better data before proceeding.
Write those possibilities down while everyone is still relatively unattached.
Later, the conversation gets emotional. The team has worked hard. An executive has mentioned the initiative to the board. Someone has named the bot.
At that point, stopping can feel like an admission of failure. Agreeing on the decision rules early makes it easier to act on what the experiment actually reveals.
A company can retire the prototype and keep everything valuable it learned.
Find Out What The Value Claim Leaves Out
The debate about software pricing makes pilot evidence especially useful. Buyers considering fees tied to usage or outcomes need to understand what the proposed measure captures.
Suppose a contract tool reduces initial review time. Does the improvement survive legal verification? Do fewer matters reach outside counsel? Can the team handle more work? Does faster review change the time to signature?
Follow the benefit far enough to understand it.
Also identify the customer’s contribution. A vendor may depend on cleaner data, revised intake processes, or substantial staff training. Those investments affect the economics of buying, building, and continuing with the current process.
The pilot gives legal and procurement a chance to discover these dependencies before they negotiate around an optimistic spreadsheet.
Make Graduation An Actual Decision
Before expanding the tool, assign an operating owner, approve its permitted uses, establish support, and document how errors will be handled. Preserve the evaluation findings, including the situations in which the system performed poorly.
That record should survive the original project team.
The most valuable sentence at the end of a pilot may be: “We now know enough to make the decision.”
Give the experiment a question, a deadline, and someone authorized to act on the answer. Otherwise, that slide may say “pilot” long after the business has become dependent on it.
Olga V. Mack is the CEO of TermScout, where she builds legal systems that make contracts faster to understand, easier to operate, and more trustworthy in real business conditions. Her work focuses on how legal rules allocate power, manage risk, and shape decisions under uncertainty. A serial CEO and former General Counsel, Olga previously led a legal technology company through acquisition by LexisNexis. She teaches at Berkeley Law and is a Fellow at CodeX, the Stanford Center for Legal Informatics. She has authored several books on legal innovation and technology, delivered six TEDx talks, and her insights regularly appear in Forbes, Bloomberg Law, VentureBeat, TechCrunch, and Above the Law. Her work treats law as essential infrastructure, designed for how organizations actually operate.
The post A Proof Of Concept Is A Question, Not A Product appeared first on Above the Law.
Picture the update at a legal department meeting: The AI pilot now has 40 users, access to live contracts, and a growing backlog of feature requests. Its creator is on vacation.
The slide still says “experiment.”
Somewhere between the promising demonstration and the vacation coverage problem, the prototype acquired a job. People began relying on it. Business processes adjusted around it. Turning it off became inconvenient.
Nobody remembers approving a permanent system.
This is the scenario legal leaders should keep in mind when someone proposes a quick proof of concept. A useful experiment can become an operational commitment through a series of small, individually reasonable decisions.
The way to manage that progression starts with a question specific enough to answer.
What Are We Trying to Learn?
“Let’s see what AI can do with our contracts” could keep a team occupied indefinitely.
A more useful question might be whether a tool can identify a defined group of provisions accurately enough to help procurement prepare for legal review. Another might ask whether lawyers can verify its answers quickly enough to reduce the total time spent on a matter.
Those are different experiments.
A tool can identify provisions accurately and still create additional work because its citations are difficult to check. Users can love an interface while consistently misunderstanding what its ratings mean. A technically impressive system can depend on documents the company cannot reliably obtain.
Choose the uncertainty that matters to the next business decision. Design the pilot around resolving it.
Give The Experiment A Fair Chance To Disappoint You
A demonstration usually shows the product under favorable conditions. An evaluation needs contact with the company’s actual mess.
Include the scanned agreement, the missing amendment, the exception buried in an exhibit, and the user who has seven minutes before a supplier call.
Decide how errors will be assessed. Missing a routine notice provision and missing a material liability exception may warrant very different responses. An overall accuracy number can hide that distinction.
Measure the human work around the output, too. Someone must prepare documents, check answers, correct mistakes, and handle escalations. Those minutes belong in the calculation.
For legal leaders, this is familiar territory. We already know that the significance of an error depends on its consequences. We should bring that judgment into the evaluation before anyone falls in love with the dashboard.
Decide What Each Result Will Authorize
A pilot should have an end date, a decision owner, and an agreed set of possible next steps.
The evidence might support buying the product. It might reveal a narrow capability worth building internally. It might show that an integration would solve the problem. It might establish that the company needs better data before proceeding.
Write those possibilities down while everyone is still relatively unattached.
Later, the conversation gets emotional. The team has worked hard. An executive has mentioned the initiative to the board. Someone has named the bot.
At that point, stopping can feel like an admission of failure. Agreeing on the decision rules early makes it easier to act on what the experiment actually reveals.
A company can retire the prototype and keep everything valuable it learned.
Find Out What The Value Claim Leaves Out
The debate about software pricing makes pilot evidence especially useful. Buyers considering fees tied to usage or outcomes need to understand what the proposed measure captures.
Suppose a contract tool reduces initial review time. Does the improvement survive legal verification? Do fewer matters reach outside counsel? Can the team handle more work? Does faster review change the time to signature?
Follow the benefit far enough to understand it.
Also identify the customer’s contribution. A vendor may depend on cleaner data, revised intake processes, or substantial staff training. Those investments affect the economics of buying, building, and continuing with the current process.
The pilot gives legal and procurement a chance to discover these dependencies before they negotiate around an optimistic spreadsheet.
Make Graduation An Actual Decision
Before expanding the tool, assign an operating owner, approve its permitted uses, establish support, and document how errors will be handled. Preserve the evaluation findings, including the situations in which the system performed poorly.
That record should survive the original project team.
The most valuable sentence at the end of a pilot may be: “We now know enough to make the decision.”
Give the experiment a question, a deadline, and someone authorized to act on the answer. Otherwise, that slide may say “pilot” long after the business has become dependent on it.
Olga V. Mack is the CEO of TermScout, where she builds legal systems that make contracts faster to understand, easier to operate, and more trustworthy in real business conditions. Her work focuses on how legal rules allocate power, manage risk, and shape decisions under uncertainty. A serial CEO and former General Counsel, Olga previously led a legal technology company through acquisition by LexisNexis. She teaches at Berkeley Law and is a Fellow at CodeX, the Stanford Center for Legal Informatics. She has authored several books on legal innovation and technology, delivered six TEDx talks, and her insights regularly appear in Forbes, Bloomberg Law, VentureBeat, TechCrunch, and Above the Law. Her work treats law as essential infrastructure, designed for how organizations actually operate.
The post A Proof Of Concept Is A Question, Not A Product appeared first on Above the Law.
