Kurate.org
Rankings
Back to Rankings

What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency

Unnati Shah, Samruddhi Kakade, Aum Patel

Oct 31, 2026arXiv:2611.01311v1
cs.AIcs.CL
Share
Scorecard· 16/16
6.5/10 impact

Carefully controlled diagnostic showing LLM judges lose catches when a silent record is present, with a reusable harness, but limited to one corpus, 24 gap faults per draw and four judges.

Abstract

LLM judges often check, against an amendment record, whether the documents an answer cites are still in force. Such records are rarely complete. From a table of replacements, our test generates matched answers citing a replaced document, its successor, the replaced document before its replacement, and an older document that a newer one updated without replacing; it removes links from the record and scores each finding by its stated reason. On 48 replacements from the IETF RFC index, judges given a correct record caught nearly every replaced citation, as a lookup would. On the 24 faults whose link we removed, every judge caught more from the texts alone than with the record: Gemini 3.1 Pro 23/24 against 1/24, GPT-5.5 23/24 against 9/24, Claude Opus 4.5 17/24 against 9/24 and Gemma 4 26B 13/24 against 0/24. The drop recurred on a second draw of removed links. Most misses carried no finding; in an exploratory run with a conflict code, only Gemini used it often. Rewording the gap as "no replacement recorded" did not help; presenting the record as an unverified note, or warning that it was incomplete, restored most lost catches for Gemini, GPT-5.5 and Claude (significant after correction for the first two) but at most 5 of 13 for Gemma; where catches returned, so did false alarms on updates. With both texts, judges mostly told replacements from updates; every update flagged had a near-identical title. A descriptive case study on Indian rooftop-solar rules is appended; we release the harness.

AI Impact Assessments

(1 model)

Core Contribution

The paper builds a small test harness for one error: an answer citing a document that has since been replaced. From a table of replacements it generates matched items (a fault, a control citing the successor, a date control, and a decoy citing an older document that was only updated). It then deletes half the links from the amendment record, presents the damaged record in several ways, and scores findings by their stated reason rather than by whether anything was flagged. The main empirical result concerns 24 "gap faults", where the record is silent about a replacement the texts reveal. On these, every judge (Gemini 3.1 Pro, GPT-5.5, Claude Opus 4.5, Gemma 4 26B) caught fewer faults with the record than from the texts alone. Gemini fell from 23/24 to 1/24. Most misses were bare passes with no finding. The drop persisted when the line said only "no replacement recorded", which is a true statement. Framing the record as an unverified note or adding a warning restored catches for three judges, but false alarms on decoys returned with them. The paper's message is that a judge given a correct record mostly acts as a lookup, and that evaluation with a correct record plus any-finding scoring hides this.

Methodological Rigor

  • Design. It is unusually careful for a small study. The gold is the publisher's own RFC index, so no annotation is needed. Matched controls and decoys target specific shortcuts. Scoring by reason is shown to matter: any-finding scoring would make judges look like they object to about two in five valid controls. The unit of analysis is the link, with exact McNemar tests and Holm correction.
  • Transparency. Hypotheses were committed before the runs (internally, not publicly). Deviations, post-hoc analyses and exploratory runs are labelled. The authors disclose that a first analysis reused pilot replies and that rerunning changed only a handful of scores. They also disclose that decoys were not matched on abstract overlap, and that the AI assistant used to build the harness comes from the same developer as one judge.
  • Weaknesses. There is a single corpus. The 48 links give only 24 gap faults per draw, and the second draw shares 12 of them with the first. Hosted models ran once, through a proxy that hides temperature and seed. The decoys are only a partial control, since abstract overlap confounds the title-similarity reading. The conflict-code run is exploratory. The presentation conditions use one prompt variant each and were run on one draw only. The mechanism is explicitly left unresolved.

    • Potential Impact

  • Practical value. The finding that a correct-record evaluation says little about a judge, and that robustness to record gaps must be measured separately, is useful to builders of legal, regulatory and RAG verification pipelines. The reason-coded scoring and date-control/decoy design are easily reusable.
  • Reach. The harness is general to any table of replacements and is released with replies and scripts. The result fits a broader literature on deference to supplied signals (anchoring, tool over-reliance, sycophancy) but adds a new twist: the signal errs only by omission, and even a true "none recorded" line suppresses catches. Scope is modest, however. It covers one error type and one corpus (RFCs, where replacements repeat titles and abstracts, making text-only detection comparatively easy). It does not cover partial amendments or chains, and it is not a large benchmark.
  • Case study. The appended Indian rooftop-solar case study is explicitly descriptive because its answer key failed quality checks. It adds little evidentiary weight and clutters the paper, though it models good honesty about negative results.

    • Timeliness & Relevance

    LLM-as-judge and retrieval-grounded verification are heavily deployed. Temporal validity and record incompleteness are real production problems. The finding that judges silently defer to metadata is directly actionable and fits current concerns about over-trust in context. The use of the newest frontier models (GPT-5.5, Gemini 3.1 Pro) increases relevance.

    Strengths

  • The core result is large, consistent across judges and across two draws, and has zero reverse flips.
  • The "true statement still hurts" condition is a clean, informative control.
  • Results are paired against a deterministic lookup baseline.
  • Reason-coding exposes bare passes versus misses with irrelevant findings.
  • The paper is honest about limits and the harness is released.
  • It reports a useful trade-off: the warning restores catches but also restores decoy false alarms, suggesting a reversion to text-only behaviour rather than better integration of the record.

    • Limitations

  • Small n, with differences between judges explicitly provisional.
  • One domain, and no tests of whether the pattern generalises to contracts or regulations with dissimilar amendments.
  • Random rather than realistic record degradation.
  • No mechanism or causal explanation.
  • Some prose is dense, with many conditions and post-hoc splits.
  • The effect may partly reflect prompt framing ("amendment record lookup"), which the authors acknowledge.
  • The paper is not a methodological breakthrough. It is a careful diagnostic study.

    • Overall

    This is a solid, honest, well-controlled diagnostic paper with a practically relevant finding. Its impact is bounded by scale and by narrow scope, so it is likely to be cited in the judge-reliability, temporal-RAG and legal-AI evaluation literatures rather than to reshape a field.

    Rating:6.5/ 10
    Significance 6Rigor 7.5Novelty 6Clarity 6.5

    Generated Nov 3, 2026

    Comparison History (0)

    No comparisons yet.