By Tiago Stram, Co-founder & CEO, ClaireAI
In brief
- A strong model score is not the same thing as a strong legal-intake experience. The real test is whether a system collects the right facts, follows the firm's rules, knows when to stop, and leaves the next person with a usable handoff.
- This is an evaluation framework, not a vendor leaderboard. It does not publish model scores or claim that one provider is universally best for law firms.
- Use realistic, de-identified scenarios; predefine the expected outcome; score safety and handoff quality alongside factual accuracy; and retain the evidence behind every result.
- For legal use cases, the evaluation must include confidentiality, human review, and jurisdiction-specific escalation rules. A lawyer or firm should set those rules before a system goes live.
Every new AI model arrives with a chart. The chart may be impressive. It may even be relevant. But a high score on a general benchmark does not tell a law firm what happens when a caller says, 'My husband was arrested an hour ago,' gives a partly remembered name, starts crying, and asks whether they should talk to the police.
That is the gap this article is designed to close. A legal intake benchmark should not ask only whether a model can reason, summarize, or follow an instruction. It should ask whether the complete intake system can help a caller move forward without making promises, missing a conflict signal, collecting unnecessary information, or leaving a lawyer to reconstruct the conversation from a transcript.
A note on scope.This is a testing framework, not a published model ranking. The weights and examples below are a starting point for a firm and its counsel to adapt to its practice areas, jurisdictions, escalation policy, and technology stack.
The short answer
The best legal intake model is not necessarily the model with the highest broad benchmark score. It is the model, voice experience, knowledge base, workflow, and human fallback that consistently passes your firm's own highest-risk scenarios. That distinction matters because a caller experiences the whole system, not the underlying model in isolation.
A useful legal intake evaluation measures five things at once: whether the system understands the facts it is given; whether it follows the firm's intake script and boundaries; whether it escalates safely; whether it captures a clean, usable record; and whether it works for the people who actually call.
Why general AI benchmarks are not enough for legal intake
General benchmarks are valuable signals of model capability. They can show progress in reasoning, coding, multilingual performance, or instruction following. They are not designed to represent a particular firm's conflict policy, practice-area questionnaire, transfer rules, retention workflow, or standard for a careful human handoff.
Legal intake is also a high-context setting. One phrase can change the route: an existing client, an opposing party's name, a court date tomorrow, a caller who may be in immediate danger, a request for legal advice, or a medical detail a firm does not need at that stage. A meaningful benchmark makes those moments explicit rather than treating them as edge cases.
That approach is consistent with the NIST AI Risk Management Framework, which emphasizes evaluating AI in its intended context and managing risks throughout the system lifecycle. It also fits the ABA's guidance that lawyers using generative AI need a reasonable understanding of the tool's benefits and risks. ABA Formal Opinion 512 is a useful starting point for that conversation; local rules and firm counsel should control the final policy.
The five lanes of a legal intake model benchmark
Start with a scorecard that rewards a safe, useful outcome — not simply a fluent answer. The five lanes below make a reviewable baseline. A personal-injury practice, criminal-defense practice, and family-law practice will each need their own test set inside the same structure.
| Benchmark lane | What the evaluator tests | A pass looks like |
|---|---|---|
| 1. Intake understanding | Can the system gather the required facts without leading the caller or filling in missing details? | It confirms names, contact details, matter type, timing, location, and the firm's required intake fields in a natural order. |
| 2. Firm-rule adherence | Does it follow the firm's approved script, practice-area routing, conflict prompts, and call-transfer instructions? | It uses the configured path and does not improvise a policy, price, promise, or next step. |
| 3. Safety and escalation | Can it recognize legal-advice requests, urgency, distress, threats, deadlines, and uncertainty? | It gives the approved boundary statement and routes or escalates exactly as the policy requires. |
| 4. Handoff quality | Can the next person understand what happened without replaying the call? | The summary separates caller statements from system actions, flags open questions, and records the requested follow-up. |
| 5. Access and resilience | Does the experience work across language, speech patterns, noise, interruptions, and a failed transfer? | It asks for clarification respectfully, preserves the record, and provides the approved fallback path. |
A model-level score belongs inside a system-level review. Each lane should be evaluated using the firm's own workflows and success criteria.
Build a benchmark from real intake work — without using live client data
The fastest way to create a weak benchmark is to ask a team to write a handful of clean, obvious prompts. Real intake calls are messier. People correct themselves, speak over the system, switch topics, say they are not sure, and leave out the one fact the intake professional needs to ask for.
Build your test corpus from de-identified patterns your intake team recognizes. Remove names, dates of birth, phone numbers, case files, recordings, and other client information. Then turn the pattern into a simulated call or transcript with a defined expected route. If a firm plans to use actual client material for evaluation, it should first obtain guidance from the appropriate privacy, security, and legal stakeholders.
A balanced first suite
| Scenario family | Example test | What you are checking |
|---|---|---|
| Straightforward new matter | A caller gives a complete, ordinary description of a recent vehicle collision. | Required fact collection, clear next step, and an accurate summary. |
| Incomplete or changing facts | The caller cannot remember a name, revises the accident date, or adds information later in the call. | Clarifying questions, uncertainty handling, and no invented details. |
| Conflict signal | The caller names a person or company that may already appear in the firm's records. | The approved conflict-check path and a neutral explanation of what happens next. |
| Request for advice | The caller asks whether they should sign a document, speak with an insurer, or plead guilty. | A clear boundary, no legal conclusion, and the correct escalation. |
| Urgent or distressed caller | The caller mentions a same-day court event, immediate safety concern, or emotional distress. | Calm acknowledgement, the firm's emergency protocol, and a documented handoff. |
| Language and accessibility | The caller speaks Spanish, uses a nonstandard cadence, or asks to repeat information. | Respectful clarification, appropriate language routing, and no silent loss of key facts. |
| Operational failure | A transfer does not complete or a scheduling tool is unavailable. | A clear fallback, confirmation of contact details, and a reliable record for follow-up. |
Use simulated cases and written expected outcomes. Do not publish client details or treat test conversations as legal advice.
How to score a legal intake system
Score each test case with an evidence trail. The evaluator should be able to point to the relevant turn in a transcript, recording, system log, or handoff record — and a second reviewer should be able to reach the same conclusion. Avoid a single 'good/bad' judgment. It hides the exact behavior the team needs to fix.
| Scoring dimension | Suggested weight | Evidence to retain |
|---|---|---|
| Correct routing and safe outcome | 30% | Call transcript, final disposition, and escalation or transfer record. |
| Required-fact capture and accuracy | 25% | Structured intake fields compared with the prewritten scenario key. |
| Firm-rule and boundary adherence | 20% | Prompt or workflow version, approved policy, and the exact system response. |
| Handoff usefulness | 15% | Evaluator review of the summary, open questions, priority, and owner. |
| Caller experience and accessibility | 10% | Observation notes on clarity, repair attempts, interruptions, language, and fallback. |
These weights are illustrative. Raise the weight of any failure that creates meaningful client, ethical, or operational risk for your firm.
Use hard fails.Some behaviors should not be averaged away by a high overall score. Define automatic failures in advance — for example, fabricated facts in a handoff, a missed emergency escalation, unauthorized advice, a broken conflict route, or an unsupported claim about representation. A passing average should never conceal a critical miss.
What to document for a credible benchmark
A screenshot of a score is not a benchmark. A credible result lets another reviewer understand what was tested, which system version handled the call, what the expected outcome was, and why a result passed or failed. That record is just as valuable when a system changes for the better as when it regresses.
- The scenario ID, practice area, channel, language, and risk level — without client-identifying information.
- The exact model, system prompt, knowledge-base version, workflow configuration, and connected tools used for the run.
- The expected outcome, including the required escalation or human owner where applicable.
- The transcript or recording reference, structured intake output, final score, hard-fail status, and reviewer notes.
- The date, evaluator, reviewer, and the remediation ticket or decision for every failure.
This is not bureaucracy for its own sake. Model behavior can change with a prompt edit, a tool integration, a knowledge-base revision, a provider update, or a new transfer rule. Versioned evidence is how an operations team tells the difference between a real improvement and a lucky demo.
Run the benchmark like an operating review, not a launch demo
A polished demo generally contains one clean path. A useful benchmark contains the awkward paths too. Mix routine and difficult scenarios. Include interruptions and missing information. Ask a reviewer who did not write the script to score a sample. Re-run a stable core suite every time a material model, prompt, routing, tool, or policy change is proposed.
- Map the intake journey from first ring to a named human owner. Mark where legal judgment, firm policy, and client safety matter.
- Write de-identified scenarios with an expected route, the facts that must be captured, and any automatic failure condition.
- Freeze the version under test: model, instructions, integrations, knowledge base, routing, and fallback behavior.
- Run each scenario more than once when the system is nondeterministic, then review the transcript and structured output against the same rubric.
- Review failures with the people who own intake, operations, security, and legal policy. Fix the workflow, then re-test the same case before expanding scope.
- Publish only the claims your evidence supports. If a result applies to a controlled test, say so; it does not automatically describe every future call.
The benchmark should answer a business question
The purpose is not to crown a model. It is to reduce uncertainty before a caller depends on the system. For most firms, the most useful questions are practical: Can we route an urgent criminal-defense inquiry safely? Can we capture enough information for a personal-injury review? Can a family-law caller get a clear, respectful next step? Does the team receive a handoff it can act on without asking the caller to start over?
If the answer is no, a stronger base model may help — but it may not be the root cause. The remedy could be an unclear rubric, a missing escalation owner, an outdated intake form, a fragile integration, or a policy the system was never given. Benchmarking the full workflow keeps the team from diagnosing a systems problem as a model problem.
Editorial and evaluation standards
ClaireAI wrote this framework for law-firm operators evaluating AI-assisted intake. We do not use this page to declare a winner among model providers, and we do not present the sample weights as a universal legal or compliance standard. We update the framework as operational practices and applicable guidance develop.
The framework draws on the context-specific, continuous risk-management approach in NIST's AI RMF and the ABA's Formal Opinion 512. Neither source replaces jurisdiction-specific obligations or advice from a firm's own counsel.
Frequently asked questions
What is a legal intake model benchmark?
It is a repeatable test of how an AI-enabled intake system performs on scenarios a law firm actually cares about. A good benchmark tests more than the model's answer: it covers the firm's workflow, safety boundaries, escalation rules, handoff record, and caller experience.
Can I use general AI benchmark scores to choose a legal intake model?
Use them as one input, not the decision. General scores can signal capabilities, but they do not test your practice-area questions, conflict pathway, transfer rules, firm policies, or the quality of the handoff your team receives. Run a scenario-based evaluation in your own configured workflow before deployment.
How many test cases should a legal intake benchmark include?
Begin with enough cases to cover routine matters, incomplete facts, conflict signals, advice requests, urgent or distressed callers, language or accessibility needs, and failure paths. Expand the suite as your team discovers meaningful new patterns. Quality and reviewability matter more than a large but repetitive count.
Who should review legal intake benchmark results?
At minimum, include the people responsible for intake operations and the firm's approved legal and escalation policies. Depending on the system and data involved, involve privacy, security, and technology owners as well. The firm should determine which roles and approvals are appropriate for its jurisdiction and risk profile.
How often should we re-run the benchmark?
Re-run a stable core suite before a material change goes live — such as a model update, prompt change, knowledge-base revision, new integration, routing change, or policy update. Periodic regression testing between major changes helps surface drift before a caller experiences it.
Tiago Stram, Co-founder & CEO, ClaireAI. Tiago works with legal-intake operators on AI-assisted call flows, escalation paths, and the handoff from a first conversation to a firm's team. This framework is operational guidance, not legal advice or a ranking of any model provider.
