Can neurons speak? Semantic narration of vision at single-cell resolution
Arnau Marin-Llobet, Richard Hakim, Sara Matias, Venkatesh N. Murthy, Na Li, Demba Ba
Abstract
Identifying what individual neurons encode in higher-order visual cortex is an open problem. Responses resist intuitive parameterization, and the deep-network embeddings used in their place are black boxes. Here, we introduce NEURRATOR, a framework that decodes spiking activity into free-form natural-language narration of the viewed scene at single-neuron resolution. A learned encoder maps spike trains from arbitrary subsets of simultaneously-recorded neurons into the patch-embedding space of a frozen CLIP, from which a multimodal language model and sparse autoencoder generates and validates a description with no language-side training. Applied to Neuropixel recordings of mouse visual cortex during natural-movie viewing, NEURRATOR narrates from thousands of neurons, singular cortical regions, local populations, or from a molecularly-defined cell-types. We use this property to (i) quantify how decoding fidelity scales with population size and cortical region, and (ii) "neurrate", in plain language, what individual neurons and genetically-tagged inhibitory cell-types contribute to visual representation. This recasts cell identity from a classification target into a functional probe of the visual system, providing a new unit of biological insights in neural systems.
AI Impact Assessments
(1 models)Scientific Impact Assessment: "Can neurons speak? Semantic narration of vision at single-cell resolution"
1. Core Contribution
NEURRATOR introduces a pipeline that maps single-neuron spike trains from mouse visual cortex into the patch-embedding space of a frozen CLIP vision-language model, enabling a frozen LLaVA multimodal language model to generate natural-language descriptions of the visual scene being viewed — all without any language-side training on neural data. The key architectural insight is that by training only a neural encoder to predict CLIP patch embeddings (576 tokens × 1024 dimensions), the entire downstream language generation machinery can be borrowed off-the-shelf. This is extended with sparse autoencoders (SAEs) pretrained on CLIP to decompose cell-type-specific neural representations into interpretable visual concept features.
The most novel aspect is the combination of three capabilities in a single framework: (1) free-form language generation from spikes, (2) subset-query flexibility allowing the same trained model to be restricted to arbitrary neuron subsets at inference, and (3) concept-level decomposition via SAEs revealing what different genetically-defined cell types encode.
2. Methodological Rigor
The approach is technically sound in its core architecture. The dual MSE + cosine loss for patch regression is well-motivated, and the PatchInjector hook is an elegant engineering solution that avoids any language-model fine-tuning. The evaluation includes several important controls:
However, several methodological concerns warrant attention. The SBERT cosine similarities, while statistically significant, are modest in absolute terms (0.367 for contiguous-middle, 0.170 for front-only). The paper acknowledges decoded narrations describe "gross scene content rather than fine perceptual detail." The pseudo-mouse construction — concatenating neurons across animals — is a pragmatic but biologically questionable workaround that conflates inter-animal variability with cell-type effects. The authors acknowledge they have not tested whether cell-type contrasts hold within single animals, which is a significant limitation for the biological claims.
The baseline comparisons (Table A1) show NEURRATOR substantially outperforms CEBRA + LLaVA and Ridge + LLaVA, but these are relatively simple baselines. No comparison is made against more sophisticated neural decoding approaches or against recent brain-to-image reconstruction methods adapted for electrophysiology.
3. Potential Impact
The framework opens several promising research directions:
The practical impact may be limited by the current performance level — the decoded narrations capture scene-level gist but not the detail needed for many neuroscience questions. The reliance on CLIP's representational biases means the system can only "see" concepts CLIP has learned, potentially missing neural representations outside this prior.
4. Timeliness & Relevance
This work sits at a timely intersection of three active areas: large-scale electrophysiology (Neuropixels), vision-language foundation models, and mechanistic interpretability (SAEs). The tension between predictive power and interpretability in neural encoding models is well-recognized, and using vision-language spaces as a bridge is a natural solution that the field has been approaching from the fMRI side. Bringing this to single-unit resolution fills a clear gap.
The application of SAEs — developed primarily for AI interpretability — as probes of biological neural representations is creative and timely, representing cross-pollination between AI safety/interpretability research and neuroscience.
5. Strengths & Limitations
Strengths:
Limitations:
Notable gap: The paper's title asks "Can neurons speak?" but the scaling analysis suggests individual neurons cannot produce meaningful narrations — the answer is closer to "populations can narrate, and we can interrogate subsets." The "single-cell resolution" framing, while technically accurate (the encoder processes individual spike trains), overpromises relative to what the decoding quality supports.
Overall, this is an innovative framework paper that introduces a creative architectural solution to a real problem in systems neuroscience. The biological findings are preliminary but hypothesis-generating, and the approach has clear potential for broader application. The main scientific contributions are the framework itself and the demonstration that cell-type identity can serve as a functional probe rather than a classification target.
Generated Jun 18, 2026
Comparison History (15)
Paper 2 pioneers a highly innovative intersection of modern AI and neuroscience by translating neural spikes into natural language. Its broad applicability across systems neuroscience, AI interpretability, and brain-computer interfaces, combined with the extreme timeliness of multimodal foundation models, gives it a wider potential scientific and real-world impact compared to the more domain-specific, though mechanistically rigorous, findings in Paper 1.
Paper 1 has higher likely impact due to a concrete, data-driven methodological contribution (NEURRATOR) demonstrated on large-scale Neuropixels recordings, enabling interpretable single-neuron and cell-type functional readouts with immediate applicability to systems neuroscience and neuroAI. It offers clear real-world utility (neural decoding, interpretability, probing cell-type contributions), strong timeliness (leveraging CLIP/LLMs), and broad appeal across neuroscience and ML. Paper 2 is conceptually ambitious and integrative, but is more theoretical with key open mathematical/biological gaps and less immediate validation, reducing near-term impact despite potentially wide long-term relevance.
Paper 1 has higher likely impact due to its scale (1,000+ hours fMRI, 720 subjects), tri-modal foundation model framing, and broad applicability: a unified predictor of human brain responses across stimuli/tasks/subjects enabling in-silico neuroscience and potentially accelerating experimental design across cognitive, sensory, and clinical domains. Its methodological claim (generalization + several-fold improvements + recovery of classic findings) suggests strong rigor and timeliness given foundation-model trends. Paper 2 is novel and interpretable for systems neuroscience, but is narrower (mouse visual cortex, decoding/narration) and likely less cross-field transformative than a general human multimodal brain model.
Paper 1 introduces a groundbreaking paradigm by decoding single-neuron activity directly into natural language. This highly innovative approach revolutionizes brain-computer interfaces and neural decoding by providing an interpretable, semantic window into micro-scale brain computation. While Paper 2 offers valuable macro-scale insights into cortical topography, Paper 1's ability to bridge single-cell biology with multimodal foundation models has broader, more transformative implications for neuroscience, neuroengineering, and AI, representing a major leap in understanding neural representations.
NEURRATOR introduces a genuinely novel framework bridging neuroscience and large language models, enabling natural-language decoding of neural activity at single-cell resolution. This represents a paradigm shift in how we characterize neural representations—moving from abstract embeddings to interpretable language descriptions. Its broad applicability (any recorded neuron, population, or cell type), methodological innovation (combining CLIP, LLMs, and sparse autoencoders), and potential to transform systems neuroscience give it wider cross-disciplinary impact. Paper 1 is rigorous and insightful but addresses a more specialized question about connectome dynamics with more incremental conceptual advancement.
Paper 2 introduces a highly novel framework bridging AI and neuroscience by decoding single-neuron activity into natural language. This has transformative implications for Brain-Computer Interfaces (BCIs) and neural decoding. While Paper 1 provides rigorous mechanistic insights into DCNN adversarial robustness, it primarily offers a clarifying, negative result regarding spatial frequencies. Paper 2's innovative use of multimodal language models to translate visual cortex activity into plain text creates a new functional probe for biology, giving it broader cross-disciplinary appeal, real-world application potential in neuroprosthetics, and higher expected scientific impact.
Paper 2 introduces a groundbreaking framework translating single-neuron activity into natural language. This presents immense novelty and transformative potential for brain-computer interfaces and systems neuroscience. While Paper 1 provides valuable insights into human-LLM semantic alignment, Paper 2 creates a highly reusable tool that addresses the 'black box' problem of neural embeddings. Its capability to decode vision at a single-cell resolution using multimodal foundation models positions it for broader, more disruptive impact across neuroengineering, biology, and AI.
NEURRATOR introduces a highly novel framework bridging neuroscience and large language models to decode neural activity into natural language descriptions at single-neuron resolution. This represents a paradigm shift in how we characterize neural coding—moving from classification to semantic narration. Its breadth of impact spans computational neuroscience, AI interpretability, and systems neuroscience. While Paper 2 makes a solid methodological contribution to biophysical modeling with neural ODEs, it addresses a more incremental improvement in an established field. Paper 1's novelty, timeliness (leveraging foundation models for neuroscience), and potential to reshape neural coding analysis give it higher impact potential.
Paper 2 introduces a fundamentally novel paradigm—translating neural spiking activity into natural language descriptions—that bridges neuroscience and AI in a highly creative way. It addresses the long-standing problem of characterizing neural tuning in higher visual cortex, offers broad applicability (single neurons to populations, cell-type-specific analysis), and leverages cutting-edge foundation models (CLIP, LLMs) in a neuroscience context. Its interdisciplinary nature, intuitive interpretability, and potential to transform how neuroscientists characterize neural function give it broader impact than Paper 1, which, while rigorous, addresses a more specialized signal-processing problem in EEG analysis.
Paper 1 is more novel and potentially higher-impact: it proposes a new, broadly usable framework to translate single-neuron spiking into natural-language scene narration via CLIP-space decoding, enabling interpretable functional probing of neurons and genetically defined cell types. This could influence systems neuroscience, neural decoding/BCI, representational analysis, and ML interpretability, with clear timeliness given multimodal foundation models. Paper 2 is valuable and rigorous, but its contribution (adding think-aloud constraints to LLM-based cognitive model discovery) is more domain-specific and likely narrower in cross-field impact.