Kurate.org
Rankings
Back to Rankings

Real Computation or Merely Naming? A Pre-Registered Mechanism Audit of 25 Cognitive-Architecture Modules in an LLM-Based Agent System

Peter Chen, Ni Zhang, Ying Xu, Siyuan Zhang

Oct 30, 2026arXiv:2610.41604v1
cs.AI
Share
Scorecard· 16/16
5.0/10 impact

Carefully reported, honest audit with reusable protocol and positive controls, but limited by an easy single target, two-system case-level depth, and largely expected findings.

Abstract

Agent systems built on local large language models are increasingly marketed through the vocabulary of cognitive architecture: modules named global workspace, theory of mind, causal reasoning, or active inference are listed as features, and the presence of the name is taken as the presence of the mechanism. We ask the prior question -- does the named computation actually happen -- and contribute a transferable verification method for the claim-implementation gap: a four-question static audit protocol with line-number evidence and call-chain tracing, demonstrated on two systems. We audit a system claiming 25 cognitive mechanisms, pre-register falsifiable predictions for 11 modules and execute the interventions, replicate the protocol on the IIT-phi core of a second, independent system with its own pre-committed falsification criteria, and run two repair experiments as positive controls. In 18 of 25 modules the computed value never reaches any production output; three idle-wiring shapes new to the dead-code and code-smell literatures emerge (connected-but-never-called, called-but-inert, called-but-content-blind). Of 61 frozen criteria across 12 interventions, 53 were confirmed, two partially adjudicated, five falsified, one adjudication-ambiguous. Repairs raised planner task success from 0.028 to 1.000 and cut state-controller error by 52%. Module names and module computations must be audited as independent claims; the protocol and the five-class failure taxonomy are reusable for the claim-implementation gap at two-system, case-level depth.

AI Impact Assessments

(1 model)

Core Contribution

The paper audits whether "cognitive-architecture" module names in an LLM-agent repository correspond to computation that reaches output. It offers four things:

  • A four-question static audit protocol with line-number evidence and value-to-output tracing.
  • A scorecard from 12 pre-registered interventions covering 61 frozen criteria.
  • A five-class failure taxonomy, plus three "idle-wiring" shapes: connected-but-never-called, called-but-inert, and called-but-content-blind.
  • Two repair experiments used as positive controls, and a cross-system replication on the IIT-φ core of a second system.

    • The headline findings are that 18 of 25 modules never affect production output, and that none of the 7 partial modules influences control flow. Repairs raised planner success from 0.028 to 1.000 and cut controller error by 52%. The framing is useful: a module name and the computation behind it are independent claims. The authors openly say the findings are mostly visible from reading code, so the contribution is quantification and protocol rather than surprise.

    Methodological Rigor

    Strengths:

  • Pre-registration with frozen criteria.
  • Falsifications reported prominently, including five falsified criteria and one ambiguous adjudication, disclosed rather than reworded.
  • Positive controls, which give the audit discriminant validity.
  • A second system with its own pre-committed criteria, which tests whether the taxonomy transfers.
  • Concrete, checkable interventions such as bit-identical outputs under perturbation, 200 distinct contents mapping to a single score, and RNG pollution in 100% of runs.

    • Weaknesses:

  • The 25-module static audit was done by LLM agents with no inter-rater reliability check. Only 11 modules received dynamic verification, so 14 failing verdicts rest on static tracing alone.
  • The object system is a single, anonymous repository, a low-quality hobbyist-style project that is a soft target. Anonymizing it hurts external verification, although a content hash and a Zenodo archive are provided.
  • "Pre-registration" is self-registered and was revised for some sutures. Many criteria are close to tautological (a function returning a constant is shown to return a constant), so a 53/61 confirmation rate says little about predictive power.
  • The repairs are weakly controlled. The planner suite is self-authored and deterministic. The controller task is matched to the mechanism, and its result is a single-run value with no seed sweep.
  • The taxonomy's "repair prospect" claims for classes 3–5 are untested predictions.
  • The second system is covered only one module deep (the IIT-φ core). The audit ran on CPU with OS stubs, so the winning-substrate path may differ from the intended Apple Silicon deployment.
  • The scale of the generalizing claim ("system class") is unsupported by two systems.

    • Potential Impact

    The protocol is simple and portable, and the call for mechanism-level checks in artifact evaluation is sensible. It could be cited in discussions of AI-agent hype, artifact evaluation, and consciousness-claim skepticism. The idle-wiring shapes are somewhat useful vocabulary, though their claimed novelty relative to dead-code and data-flow analysis is modest. The most likely path to influence is as a case study or template within the reproducibility and agent-evaluation communities. The paper is unlikely to shift mainstream agent research, since the audited systems are fringe, but it models good practice for negative-result reporting.

    Timeliness

    Agent systems with anthropomorphic module names, and consciousness-related claims about AI, are a current concern. Claim-versus-implementation gaps are a real problem, and a cheap audit method is relevant. However, an automated or LLM-assisted static audit of this kind will probably be superseded by general code-analysis tools. The paper also reads somewhat like a prolific self-citing series, with companion audits cited as preprints on Zenodo.

    Strengths & Limitations

    Strengths:

  • Transparency and honest reporting, including self-acknowledged limitations.
  • Falsification discipline.
  • Positive controls, which are rare in audit papers.
  • Concrete, line-level evidence with released artifacts.
  • A cross-system check.

    • Limitations:

  • Mostly confirms the obvious; the target is an easy one.
  • Two-system, case-level evidence only.
  • No inter-rater reliability for the LLM auditors.
  • Single-run repair statistics.
  • Self-authored benchmarks.
  • Authors from a single small institution with an unnamed target, and some writing that is dense with internal identifiers ("suture19") and hard to follow.
  • Generalization claims ("reach beyond cognitive-architecture vocabulary") outrun the evidence.

    • Overall

    This is a competent, honest, and modestly useful methodological case study. Its scientific impact is limited by the weak target, the narrow scope, and the near-tautological nature of many findings, but it has value as a template and as a cautionary data point.

    Rating:5/ 10
    Significance 4.5Rigor 5.5Novelty 4.5Clarity 5

    Generated Nov 2, 2026

    Comparison History (0)

    No comparisons yet.