Benchmarking Legal AI on Ambiguity: A New Evaluation Framework for Edge-Case Reasoning

Abstract

Legal AI systems are increasingly deployed in contexts that demand nuanced interpretive judgment, yet existing evaluation benchmarks remain anchored to deterministic accuracy measures that reward single correct answers and penalize appropriate epistemic hedging. This misalignment between evaluation design and the realities of legal practice produces systems that perform well on benchmarks while failing in precisely the situations where professional legal judgment is most consequential.

This article proposes an Ambiguity-Aware Evaluation Framework for legal AI that moves beyond top-1 accuracy as the primary performance criterion. The framework is developed through a structured analysis of the structural limitations of current legal AI benchmarks, a taxonomy of legal question types, and the specification of novel evaluation metrics and architectural requirements suited to ambiguity-rich legal reasoning. The framework distinguishes three categories of legal questions: deterministic questions governed by clear statutory rules and uniform doctrine; contested questions characterized by legitimate interpretive disagreement across jurisdictions or scholarly opinion; and indeterminate questions requiring policy judgment, fact-sensitive reasoning, or analogical extension into unsettled legal territory. For each category, the framework specifies appropriate evaluation criteria and defines three novel metrics: ambiguity detection rate, which measures whether a system correctly identifies when a legal question lacks a settled answer; interpretation completeness, which evaluates whether a system enumerates the full space of legally viable positions; and qualification appropriateness, which assesses whether expressed confidence is calibrated to the underlying degree of legal certainty. The article further identifies the architectural components required to operationalize the framework, including retrieval-based ambiguity detection mechanisms, structured interpretation enumeration modules, and uncertainty calibration training objectives that reward hedged reasoning on contested and indeterminate questions rather than penalizing it. The findings indicate that current benchmark designs systematically incentivize overconfidence, fail to distinguish genuine legal reasoning from surface pattern matching, and inadequately prepare systems for deployment in professional legal contexts. The proposed framework provides a more practically valid standard for evaluating legal AI, one that reflects the epistemic demands of skilled legal practice and rewards systems capable of recognizing and communicating the limits of deterministic legal analysis.