CLM buying

How to Run a CLM Pilot That Settles the Buying Decision in a Week

Adira EditorialLegal AI desk10 min read

Most CLM pilots end without a decision. The team gets a trial login, uploads a few sample agreements the vendor suggested, watches a confident demo, and two weeks later the honest summary is "it looked good". That is not evidence anyone can put in front of a general counsel or a CFO. A pilot that decides the purchase is designed backwards from the decision. You write down, before anyone logs in, which result would make you buy and which would make you walk away. Then you test with your own contracts, your own reviewers and your own clock. This guide sets out a plan that fits inside a 7-day trial, with the preparation done beforehand. ## Name the one decision the pilot must settle Write a single sentence at the top of the pilot document. Examples: - "Can this tool cut first-pass review of inbound vendor agreements enough to free one lawyer-day a week?"

  • "Will sales-team NDAs go out without legal touching them, while staying inside our positions?"
  • "Can we find every renewal date across our signed customer contracts without opening each file?" If you cannot write that sentence, you are not ready for a pilot. You are still at the stage of deciding what problem to buy for, and a vendor demo will happily fill that gap with its own answer. ## Pick 15 to 25 contracts that reflect real work The sample is where most pilots go wrong. Vendors offer clean demo documents; real work is scanned PDFs, redlined Word files with tracked changes, and counterparty paper that ignores your template. Build the set yourself: | Slot | Count | What to include | |---|---|---| | Routine, your own paper | 5 to 7 | Recent NDAs, standard vendor agreements, simple order forms | | Counterparty paper | 5 to 8 | Drafts that arrived on the other side's template, ideally ones you already negotiated | | Messy formats | 2 to 4 | A scanned PDF, a document with heavy tracked changes, a long schedule-heavy agreement | | Known traps | 3 to 6 | Contracts where you already know the problem: an uncapped indemnity, a silent auto-renewal, a missing data clause | The "known traps" slot is the most valuable. Because your team already found the issue, you can check whether the tool finds it too, without debating what the right answer was. Keep a sealed answer key listing each trap; do not share it with the vendor. ## Time the current process before the trial starts A pilot without a baseline produces opinions. For at least five contracts from the sample, record how long the current process takes, measured the way you will measure during the pilot: - Touch time: minutes a lawyer or contract manager actually spends reading, marking up and writing comments.
  • Elapsed time: days from receiving the draft to sending your response.
  • Rework: how many times a senior reviewer sent it back. Use a stopwatch or a timesheet code, not memory. People routinely misjudge the time they spent on a review when asked afterwards, in both directions. ## Write the pass marks in advance Agree these with the decision-maker and put them in the pilot document before the trial begins. Changing them afterwards is how a disappointing pilot gets turned into a purchase. 1. Issue detection: the tool flags at least [n] of the [m] known traps in the answer key.
  1. False alarms: reviewers rate no more than [n] flags per contract as wrong or irrelevant.
  2. Draft quality: a senior lawyer would send the generated first draft after edits taking under [n] minutes.
  3. Touch time: median touch time on counterparty paper falls below [n] minutes.
  4. Usability: every participant completes the core task without vendor help by day 3.
  5. Data terms: written answers on training, model sub-processors and deletion received and accepted. Fill in the numbers from your baseline. A pass mark that every tool would clear is not a pass mark. ## Who takes part, and for how much time Keep the group small and real: - Two hands-on reviewers who do this work weekly, booked for about four hours each across the week.
  • One senior reviewer who scores output quality blind where possible, about two hours.
  • One owner who runs the schedule, keeps the scoring sheet and chases the vendor, about one hour a day.
  • A security contact who reviews the vendor's written answers in parallel. Leave out people who will only attend the demo. Their impressions will not match the scored results, and they tend to carry weight they have not earned. ## Seven days, planned day by day This plan assumes a trial of seven calendar days, which is what many self-serve CLM trials offer, and that the preparation above is finished. Day 1: Set up and calibrate. Create the workspace, invite participants, load company documents the tool uses to learn your style, and upload the sample set. Run two routine contracts together as a group so everyone uses the tool the same way. Day 2: Routine paper. Reviewers generate or review the routine slot. Record touch time per contract on the scoring sheet. Note anything the tool did that you would not accept. Day 3: Counterparty paper, first half. Review half the counterparty slot. This is the heaviest day; protect the reviewers' calendars. Day 4: Counterparty paper, second half, plus messy formats. Watch for failures on scanned files and tracked changes. A tool that cannot read your real files fails the pilot regardless of its review quality. Day 5: Known traps. Run the trap contracts. Compare flags against the sealed answer key. Record hits, misses and false alarms. Day 6: Blind quality scoring. The senior reviewer scores a mix of tool-assisted outputs and outputs produced the old way, without being told which is which. Day 7: Data, export and debrief. Export a few contracts to Word and PDF and check formatting. Confirm you can get your data out. Hold a one-hour debrief and complete the scoring sheet before anyone discusses price. ## The scoring sheet Score each criterion from 1 to 5 using the anchor definitions, then multiply by the weight. Agree the weights on day zero. | Criterion | Weight | 1 means | 5 means | |---|---|---|---| | Known traps caught | 3 | Missed most | Caught all, with the right reasoning | | False alarms | 2 | Many flags on every contract were noise | Flags were almost always worth reading | | First-draft quality | 2 | Needed rewriting | Sendable after light edits | | Touch time versus baseline | 2 | Slower than today | Clearly faster on counterparty paper | | Handles real file formats | 1 | Failed on scans or tracked changes | Read everything | | Ease of use without vendor help | 1 | Needed hand-holding | Self-explanatory by day 2 | | Data terms in writing | Pass or fail | Missing or vague | Clear and contractual | The data terms row is a gate, not a score. A tool that wins every other row but cannot state in writing how it treats your contracts does not proceed. ## Warning signs in a vendor-run demo Demos are useful for learning the tool and poor evidence for buying it. Watch for these: - The vendor brings the documents. Ask to use one of yours, live. A polished demo on a pre-loaded agreement shows the best case.
  • The presenter never types a new instruction. Pre-recorded or rehearsed flows hide how the tool handles an unexpected request.
  • Answers about AI quality turn into customer logos. You asked how often it misses an indemnity cap; a logo slide is not an answer.
  • "That is on the roadmap" for a pass-mark item. Score what exists today. Ask for the roadmap item in writing with a date if it matters.
  • Pressure to sign before the trial ends. A discount that expires before your pilot finishes is a reason to be careful, not to hurry. ## Comparing two vendors without favouring either If you pilot two tools, fairness takes a little planning: 1. Same sample, same answer key. Every contract goes through both tools.
  1. Swap the order. Reviewer A uses tool X first on half the set and tool Y first on the other half; reviewer B does the reverse. Otherwise the second tool benefits from the reviewer already knowing the document.
  2. Same setup effort. Give each vendor the same company documents and the same hour of onboarding.
  3. Back to back, not months apart. Run the trials in consecutive weeks so team workload and the contract mix stay comparable.
  4. Score before discussing price. Price conversations change how people remember quality. The NIST AI Risk Management Framework is a useful frame here: its Measure function asks you to test AI systems against criteria tied to your context, which is what the answer key and blind scoring do on a small scale. ## Confidentiality during the trial Before any real contract goes into a trial workspace, read the vendor's data terms. For US law firms, ABA Formal Opinion 512 addresses generative AI tools and, as summarised by commentators, treats client informed consent as necessary before confidential information goes into a self-learning tool, with boilerplate engagement-letter language not enough. In-house teams have fewer ethics constraints but the same commercial ones. If the terms are unclear, use your own company's paper or redact party names and prices. If a tool calls an outside model, ask which provider and on what terms; Anthropic's commercial terms, for instance, state that it may not train models on customer content from its services. ## Where Adira fits as a pilot candidate Adira offers a 7-day free trial, which is the reason the plan above is built around seven days. It is a reasonable candidate when your pilot sentence is about drafting in your own house style or reviewing counterparty paper: the review view marks positions as favourable, neutral or unfavourable to your company and lists red flags, key dates, obligations and missing clauses, and edits can be made clause by clause in plain English with a diff. Adira states that it does not train on customer contract corpora, and it is built on Anthropic's Claude. Honest limits for a pilot: SSO/SAML, audit logs and data residency controls are Enterprise features, so a trial will not show them, and Adira's legal depth is greatest for Indian law, with UK, Irish, Singapore and Australian support architected but to be checked contract type by contract type. If your sample is mostly US-law paper, the known-traps slot will tell you quickly whether it meets your pass marks. Adira's own measured result, from the 18 law firms that use it, is contract review and due diligence cut from 10 to 12 days to under 2 hours with only 2 to 3 rechecks; your pilot is the way to find out what it does on your contracts. ## When neither tool clears the bar A pilot where nothing passes is not wasted. Look at where the misses cluster. If both tools missed the same traps, the problem may be the trap itself: perhaps the issue only shows up when you read a schedule against the main body, which few tools do well, and a human check for that clause type should stay in your process whatever you buy. If both tools failed on scanned files, your real fix may be getting signed copies as native documents. If reviewers simply did not use the tools after day 2, the cause is often training time or a workflow that asked them to copy text between systems. Write those findings down too. They make the next pilot shorter and the next RFP sharper, and they stop the same vendor pitch from restarting the same debate six months later. ## After the pilot: the two-page result Write it up the same day while scores are fresh: the pilot sentence, the sample, baseline versus pilot touch time, traps caught, the weighted score for each vendor, the data-terms result, and a recommendation that refers back to the pass marks. If you are moving to implementation, the CLM implementation guide for US teams covers the rollout, and if the pilot exposed gaps in your requirements, revisit how to write a CLM RFP before the next round.

Frequently asked questions

How many contracts should a CLM pilot use?
Fifteen to twenty-five real contracts is enough to see patterns without turning the pilot into a project. Fewer than ten and one odd document can swing the result; more than thirty and reviewers stop scoring carefully by the end of the week.
Can a meaningful CLM pilot fit in a 7-day free trial?
Yes, if the preparation happens before the trial starts. Pick the contracts, take the baseline timings, write the pass marks and book reviewer time in advance, then use the seven days only for running and scoring. Teams that start the clock and then choose documents lose half the trial.
Should we use real client or company contracts in a pilot?
Real contracts give the only meaningful result, but check the vendor's data terms first and, for a law firm, consider client confidentiality duties. ABA Formal Opinion 512 says client informed consent is needed before confidential information goes into a self-learning AI tool. Redact or use your own company paper if the terms are not clear.
What is the difference between a CLM pilot and a proof of concept?
People use the words loosely. A proof of concept usually asks whether a tool can do a task at all; a pilot asks whether it does the task well enough, on your documents and with your people, to justify buying. For a purchase decision you want the pilot.
How do we stop the vendor from steering the pilot?
Choose the contracts yourself, run the tasks yourself in your own trial workspace, and score against criteria written before day one. Accept vendor help on setup and training, but do not let the vendor operate the tool during scored tasks.
Was this useful?

See how Adira drafts in your voice and reads contracts from your side.

Explore the showroom

Working through a contract like this? Weave is Adira’s free tool to read, mark up, and connect any contract in your browser — no account needed.

Try Weave — free

Book a demo

See Adira on your own contracts

A 30-minute walkthrough with our team. Tell us a little about you, then pick a time that suits you.

Prefer email? [email protected]