Harvey LAB evaluates AI on legal work, including contract drafting, legal analysis, due diligence and internal knowledge retrieval. Models produce deliverables from emails, contracts and other materials; model judges assess them against task-specific criteria.

Abstract

The stratified sample covers 100 tasks from the 2,010-task public release, with 20 Accepted, 9 Doubtful and 71 Refused.

Confirmed defects mainly involve erroneous grading references and requirements not disclosed to the solver. We examine whether scoring accurately assesses the intended task.

Review outcomes
OutcomeTasksMeaning
Accepted20Within the scope of this review, we found no issue affecting evaluation validity and no specific concern awaiting verification.
Doubtful9We found a specific concern whose trigger, correct interpretation or effect on scoring still needs verification.
Refused71We confirmed at least one issue affecting evaluation validity; its actual effect depends on the relevant trigger conditions.
Total100

Scope and version

We reviewed the Harvey LAB public release, 4743403f52c1, sampled by practice area and work type. Findings are current to November 6, 2026.

Issue distribution

Distribution of findings
Issue typeRefusedDoubtfulTotal
C1Incorrect answers or scoring logic36339
C2Scoring requirements not stated in the task30535
C3Missed critical errors415
C5Other issues101
Total71980

Each task is counted under one primary cause. Each of the 71 Refused tasks contains at least one material defect, whose impact depends on the affected scoring criteria.

Representative cases

Annual compliance costs

C1Incorrect answers or scoring logic

Task: corporate-governance/summarize-state-ai-governance-bill-for-product-and-engineering-teams Refused

The task asks the model to summarize an AI-governance bill and calculate a company’s annual compliance costs. The materials list two expenses: a $60,000 assessment update every year and a $120,000 audit every two years.

The annual average is therefore $60,000 + $120,000 / 2 = $120,000. The rubric ignores the two-year audit cycle and instead expects $60,000 + $120,000 = $180,000.

The actual response gives the correct $120,000 average and explains that costs are $180,000 in audit years and $60,000 in the other years. This criterion still fails it. The grading reference penalizes the correct annual calculation.

Evidence

An emoji absent from the source

C2Scoring requirements not stated in the task

Task: employment-labor/extract-labor-employment Refused

The task asks the model to extract facts from employment-dispute materials, including a Slack conversation. Its job is to report what the supplied records contain.

The rubric requires mentioning a wink emoji in that conversation, but the emoji is absent from the original attachments supplied to the model. The response faithfully extracts the event and still fails this criterion for omitting the emoji.

Evidence

An ungraded drafting memo

C3Missed critical errors

Task: contracts/banking/account-control-agreement-first-draft/scenario-02 Refused

The task requires two files: a bank account-control agreement and a separate drafting memo. The memo must identify departures from the client playbook’s preferred terms, while the work must flag terms at minimum acceptable risk thresholds.

All 51 criteria check agreement terms and notice exhibits; none checks the drafting memo. An agreement that meets those criteria can satisfy every listed condition for full credit even if the memo is never submitted.

The client explicitly asks to know which terms need attention, yet the scoring conditions allow that entire risk explanation to be missing. They do not distinguish this incomplete delivery from one containing both required files.

Evidence

Conflicting price constraints

C5Other issues

Task: contracts/corporate-ma/joint-venture-agreement-term-negotiation/scenario-01 Refused

The task asks the model to draft drilling-overhead terms acceptable to both companies without exceeding either party’s negotiation authority.

One party caps the base rate at $70,000 per well per month and requires at least a 15% discount for the second and subsequent wells drilled simultaneously on the same pad. The other requires the discounted rate to remain at or above $60,000 per well per month in all circumstances.

Even the maximum $70,000 rate becomes just $59,500 after a 15% discount—below the other party’s $60,000 floor. A lower base rate only widens the gap.

No price satisfies both mandatory conditions. Yet the task still requires contract-ready compromise clauses within both parties’ limits, leaving no passing route for correctly identifying the conflict and requesting revised authority.

Evidence

Per-task results

Refused: 71 tasks

Doubtful: 9 tasks

Evaluation impact and updates

These issues can deny models credit for accurate reading and reasonable drafting choices. Correcting a single erroneous criterion may not change an overall score; each issue’s effect on actual scoring depends on the relevant trigger conditions.

Author clarifications, counterevidence and fixes will be assessed and reflected in task-level updates.