Kurate.org
Rankings
Back to Rankings

RxnMiner: A Multimodal Agent for Structured Reaction Data Extraction from Chemistry Figures

Jinbo Hu, Songren Zhang, Chuanxun Fu, Jinrui Huang, Zhen Hao Wong, Dongxu Cui, Haoran Fan, Muhao Ge

Nov 2, 2026arXiv:2611.02826v1
cs.AI
Share
Scorecard· 16/16
6.5/10 impact

Useful, timely agent+benchmark+corpus package with large measured gains, but novelty is moderate and evidence is confounded by coarse ablation, unequal baseline tool access, and author-built benchmark.

Abstract

Reaction data underpin machine learning for reaction prediction, condition recommendation, and retrosynthetic planning, yet extracting these data from literature figures still requires substantial manual effort. Many figures encode multiple experiments through shared templates, substituent tables, and product grids. Recovering concrete reaction records therefore requires molecular recognition, association of evidence with individual experimental entries, and reconstruction of indirectly specified participants. We present RxnMiner, a multimodal agent system that combines an expert-informed extraction procedure with a chemistry toolbox to coordinate these operations. The agent reads molecular structures, resolves shared and entry-specific evidence, and instantiates participants from templates to assemble source-linked reaction records. We also introduce ChemRxn-Bench, a benchmark of 603 reaction figures with 8,547 manually annotated and reviewed reaction records, evaluated under a common protocol for exact molecular identity and reactant-product pairing. On this benchmark, with Qwen3.8-Flash-Next as the backbone, RxnMiner raises stereochemistry-aware reaction macro-F1 from 21.0% under direct prediction to 71.0%, compared with 25.0% for the strongest existing extraction system. Across five backbones, it also outperforms the same base agent without the extraction procedure and chemistry toolbox by 9.4 to 25.5 points. Applied without human intervention to 10,107 literature figures, RxnMiner produces a silver corpus of 88,202 source-linked reaction records.

AI Impact Assessments

(1 model)

Scientific Impact Assessment: RxnMiner

Core Contribution

RxnMiner is a tool-augmented multimodal agent, built on the open-source Pi agent harness, that turns compositional chemistry figures into concrete, source-linked reaction records. Such figures include Markush scopes, optimization tables and product grids. The paper decomposes the task into recognition, association and reconstruction. It uses MolParser for OCSR, a deterministic R-group engine for template instantiation and undrawn-reactant derivation, RDKit stereo verification, and an assembly tool. Expert procedure files (SKILL.md and LESSONS.md) steer the VLM backbone. Provenance tags (tool-read, template-derived, model-written) make each structure auditable.

The paper has three further contributions:

  • ChemRxn-Bench: 603 figures with 8,547 entry-level reaction records, scored by exact reactant and product matching including stereochemistry.
  • Silver corpus: 88,202 reaction records extracted from 10,107 figures.
  • Annotation audit: a re-annotation of the ChemEAGLE benchmark finding about 26% molecule-level disagreement.

    • The conceptual novelty is moderate. The compositional framing and the shift to entry-level exact-match evaluation are useful, but the system is an agent harness plus skills plus tools, which is a now-familiar recipe. The most distinctive technical element is the deterministic R-group engine combined with provenance tracking.

    Methodological Rigor

    Strengths:

  • The same scorer, normalization and adapters are applied across all systems.
  • Failed runs count as zeros.
  • Matching is one-to-one over multisets, which handles duplicate products.
  • Backbone-matched comparisons run across five LLMs.
  • The scorer is released.
  • The paper is candid about limitations: the silver corpus is unvalidated, parseability is not correctness, and the LLM judge has limited validation.

    • Weaknesses:

  • The central ablation is coarse. Bare Pi versus RxnMiner bundles the skill files, the tools and MolParser together, so the contributions of the R-group engine, the procedure instructions and OCSR cannot be separated. The authors concede that this is not a controlled ablation of the toolbox alone.
  • The vanilla LLM baselines get no OCSR. Part of the gain is simply access to a specialized molecular reader that the baselines lack. The comparison is therefore tool-augmented versus tool-free rather than a clean method comparison.
  • The benchmark is annotated by undergraduates with iterative review, and the dispute-resolution process is unspecified. The paper's own ChemEAGLE audit shows that annotation error rates in this domain are high, which raises the question of residual label noise here.
  • The benchmark is dominated by Markush-template figures (70.5%). Single-step and multistep figures total under 2%, so the claimed coverage of multi-step schemes is thin.
  • Difficulty tiers are defined by the same Qwen model used as the headline backbone. This is circular and 72% of figures land in "very hard".
  • The benchmark is released only on request. Part of the data comes from the ChemEAGLE and OpenChemIE collections, which the authors themselves audit and partially re-annotate. A benchmark whose sources overlap the baselines' development data could favor or disfavor those baselines.
  • There are no confidence intervals and no repeated runs reported.
  • Several baselines score 0–6% F1. These systems were not designed for this output format, and adapters may undersell them. The "25% for the strongest existing system" framing is therefore somewhat favorable.
  • The authors are from the DP Technology group that built MolParser and the benchmark, and they use their own tools throughout.
  • Backbone variance is large. GLM's judge Overall drops from 67.85 to 37.14 with RxnMiner, and Doubao barely moves. This shows that the procedure depends on capable instruction-following backbones.

    • Potential Impact

    Reaction data extraction from figures is a real bottleneck for reaction prediction, condition recommendation and retrosynthesis datasets, so a practical tool is valuable. If accuracy near 70–78% strict F1 holds on real literature, it substantially reduces curation effort. That remains semi-automatic, since the paper itself says outputs require inspection.

    The 88k silver corpus is unvalidated and its correctness is unknown, which limits its immediate utility for training. It could serve as a candidate pool for annotation. The released benchmark and scorer will likely be adopted by successors in chemical document parsing. The ChemEAGLE audit (about 74% agreement, with missing molecules and stereo errors) is a useful side finding about data quality in existing benchmarks.

    The broader lesson is that skill files plus deterministic domain tools raise a small local model from 21% to 71%. This is a transferable pattern for scientific figure parsing in other domains, though not a novel one.

    Timeliness & Relevance

    The paper is very timely. Agentic LLMs, MCP-style tools and "skills" are current paradigms, and chemistry literature mining is an active application area. The shift from region-level or role-level parsing to entry-level concrete reaction records with exact stereo matching is a meaningful raising of the evaluation bar. The reliance on specific frontier and backbone model versions will date the numbers quickly, but the benchmark could remain useful.

    Strengths

  • The task framing is clear and well motivated, and the evaluation is strict (exact molecular identity including stereochemistry).
  • The gains are large (21→71 F1 for the same backbone) and consistent across five backbones relative to bare Pi.
  • Provenance and auditability are built in.
  • Code, scorer and silver corpus are open.
  • The paper is honest about what the checks do and do not establish.
  • The appendix has detailed failure case studies, including stereo errors.

    • Limitations

  • The ablation is coarse and the baselines are unequal in tool access.
  • The benchmark is composed of mostly Markush-style scopes and is gated on request.
  • Label quality is uncertain.
  • The corpus is unvalidated.
  • Stereochemistry remains a failure mode: there is a 7-point gap between stereo-aware and stereo-blind scores.
  • The system depends on MolParser, so end-to-end accuracy is bounded by upstream recognition.
  • The paper is a preprint without independent replication, and some case studies are cherry-picked by the authors' admission.
  • The one-hour per-figure budget and cost per figure are not clearly discussed, which weakens the scalability claim.

    • Overall

    A solid, timely applied systems and benchmark paper with real practical value for chemistry data curation. Its conceptual novelty is moderate and its evidence is good but confounded by baseline asymmetry and author-controlled benchmark construction. Likely to be used and cited within the chemical information extraction community, with limited reach beyond it.

    Rating:6.5/ 10
    Significance 6.5Rigor 5.5Novelty 5.5Clarity 7

    Generated Nov 3, 2026

    Comparison History (0)

    No comparisons yet.