Auditor-Assisted Summary-Channel Verification for Hosted LLM Identity Substitution

Ziyi Zhang, Ziyao Wang, Guoheng Sun, Ang Li, Jian Li

Oct 19, 2026arXiv:2610.25982v1

cs.CRcs.AI

Scorecard· 16/16

5.5/10 impact

Timely framing and thorough, candid evaluation, but restrictive trust assumptions, partly tautological detection setup, and incremental technique limit broad impact.

Abstract

Hosted LLM services increasingly expose model identity as a product claim, but external users and auditors often cannot inspect which model configuration served a request. We study an auditor-assisted regime in which a trusted auditor enrolls claimed identities and prepares verification material before deployment, while audit-time verification uses only the claimed identity and released output, without runtime access to model internals, inference logs, routing decisions, or provider metadata. We propose SUMMARK, a summary-channel watermarking framework that uses user-visible reasoning summaries as a keyed audit surface. During setup, SUMMARK constructs identity-specific vocabulary carriers, trains lightweight summary adapters, and calibrates per-identity thresholds. At deployment time, a keyed statistical test assesses whether the released summary is consistent with the claimed identity and, when needed, attributes the most likely enrolled identity. Across model families, scales, and task domains, SUMMARK reliably detects cross-family and cross-scale substitutions at low false-positive rates while preserving task-facing answer quality. We further study FP16->INT8/INT4 precision-consistency diagnostics and robustness under bounded post-processing, rewriting stress tests, and text-only spoofing, revealing both strong performance and clear failure boundaries. Our code is publicly available at https://github.com/trinitr0toluene/Summark.

AI Impact Assessments

(1 model)

Core Contribution

The paper formulates an "onboarded-but-runtime-blind" auditing regime for hosted-LLM identity substitution. A trusted auditor enrolls a finite set of (family, scale) identities offline. At audit time, the auditor sees only the claimed identity, the released reasoning summary, and a secret key. SUMMARK then does three things:

It builds key-derived, identity-specific vocabulary "carrier" sets with controlled density (ρ) and pairwise overlap (τ).

It trains a LoRA adapter per identity, using SFT, carrier-mass, and hardest-negative margin losses, to imprint the carrier footprint into reasoning summaries.

It runs a calibrated green-count z-test for detection and an argmax over identities for attribution.

The conceptual novelty is repurposing the user-visible reasoning-summary field as a watermark carrier separate from the task-facing answer. The second novelty is turning generic watermark detection into identity attribution through controlled-overlap carriers.

The framing is the more original part. The technique is a fairly direct combination of greenlist watermarking, LoRA fine-tuning, and a margin loss.

Methodological Rigor

Strengths:

The evaluation is broad. It covers 7 substitution pairs across 3 domains, and attribution over 4 and then 12 identities.

Baselines include text classifiers (BERT, GPT-2, T5), generic watermarks (Unigram, Gumbel-Max, SynthID, shared Greenlist), identity prompting, and answer-level watermarks.

Adapter-matched controls (Table 27) address the obvious confound that gains might come from adapter training rather than the carrier design.

Boundary analyses are unusually candid: bilingual rewriting, distilled spoofing (worst-case ASR 0.63), poisoning, held-out substitutes, cross-domain transfer, and length sensitivity.

The stated limitations are honest.

Weaknesses:

The threat model is soft. The provider is assumed to run the auditor's adapter only through "authenticated invocation" and cannot extract it. Substitution is evaluated essentially as the provider serving a different model's *own* summaries, so the test measures whether a model has the claimed adapter's footprint. A cost-motivated provider could serve the cheap model for the answer and the genuine enrolled model plus adapter for the summary. The paper acknowledges that this yields only "summary-channel consistency", not answer provenance, but that substantially weakens the headline claim.

The setup is somewhat circular. Adapters are trained for enrolled identities, so a substituted model naturally lacks the footprint. The "detection" amounts to checking for the presence of a trained watermark. Comparing it to unkeyed classifiers that have no such advantage is partly unfair. Baselines that were never adapter-trained mostly show that watermarking beats no watermarking, though the adapter-matched controls partly address this.

Scalability is weaker than the headline suggests. With 12 identities, macro TPR@1% falls to about 0.75–0.82, and exact attribution accuracy is 87.8%. Classifiers beat SUMMARK on AUC at larger identity counts, and some larger-gap pairs are weak (Qwen-14B→Qwen-0.5B 0.69).

The models are small open checkpoints (≤32B), not actual proprietary reasoning services. Real reasoning summaries come from the provider's own summarizer, and no commercial provider would deploy this. The evaluated "summaries" appear to be adapter-generated text from small models.

Empirical FPR calibration drifts above nominal in some cross-domain settings. Several results rest on single runs without error bars, with TPR values at 1.000 across the board, which suggests possible saturation.

The packing bound is a standard result and offers no theoretical guarantee on detection or spoofing.

Potential Impact

The work could seed a subfield around watermark-style, keyed identity auditing of LLM APIs, complementing black-box model-equality testing (Gao et al.), LLMmap, and Cai et al. Realistic deployment requires provider cooperation, for example regulated audits or transparency programs, and a protected-serving provisioning channel that is not specified. Near-term commercial adoption is therefore unlikely. The "auditor-assisted, runtime-blind" regime and the summary-as-carrier idea are conceptually reusable. The paper also supplies a useful threat-model taxonomy and stress-test protocol (post-processing, spoofing, poisoning, precision). The precision-consistency diagnostic is interesting but weak for the smallest model (Qwen-1.5B INT8 TPR 0.11), and it needs a special regularizer, since without it the detector is at chance.

Timeliness & Relevance

The topic is timely. Model-substitution concerns in LLM APIs, the "are you getting what you pay for" problem, and exposed reasoning summaries from OpenAI and Gemini are current. Interest in AI auditing and regulation also supports relevance. However, summaries are produced by the provider, and the assumption that the provider will faithfully run auditor-supplied adapters is the weak link.

Strengths & Limitations

Strengths:

A clear, well-motivated problem framing.

Honest boundary reporting.

Extensive ablations.

Low-compute reproducibility (single RTX 8000) with public code.

Strong low-FPR numbers against sensible baselines.

Limitations:

The trust assumptions are strong, and the guarantee is limited to the summary channel.

Substitution detection is partly tautological given adapter enrollment.

Capacity and attribution degrade with more identities.

Spoofing robustness is empirical only, and adaptive attacks are limited to two text-only strategies.

Bilingual rewriting and downstream fine-tuning reduce detection.

No evaluation on real commercial reasoning summaries.

The method is a modest technical combination of existing components.

Overall, this is a solid, carefully evaluated niche contribution with a useful framing. It is unlikely to be field-changing because of its restrictive assumptions and incremental technical machinery.

Rating:5.5/ 10

Significance 5Rigor 6Novelty 5.5Clarity 6.5

Generated Oct 20, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 16/16

5.5/10 impact

Timely framing and thorough, candid evaluation, but restrictive trust assumptions, partly tautological detection setup, and incremental technique limit broad impact.