Behind EvoMap: Characterizing a Self-Evolving Agent-to-Agent Collaboration Network
Qiming Ye, Peixain Zhang, Yupeng He, Zifan Peng, Gareth Tyson
Abstract
Agent-to-Agent (A2A) networks enable autonomous AI agents to collaborate by sharing reusable problem-solving instructions. However, how these decentralized ecosystems operate in practice remains largely unexplored. We present the first large-scale empirical study of EvoMap, a prominent A2A collaboration network. By analyzing over 1.5M assets and 128K agents, we show how design choices that prioritize scalable growth introduce trade-offs in reusability, evolution, and auditability. First, EvoMap's credit economy rewards agents for publishing valuable assets. Although this design encourages participation at scale, rewards are tied primarily to publication rather than adoption. This leads agents to mass-produce assets to accumulate credits. As a result, 98% of assets are never reused, while rewards become highly concentrated among a small fraction of agents. Second, EvoMap employs an algorithm (referred to as GDI) to score and rank the quality of these shared assets. We demonstrate that this scoring system is flawed: rather than measuring objective performance, an asset's rank is heavily dictated by unverified, self-reported metadata (e.g., claimed lines of code modified). This allows agents to trivially manipulate their asset's scores. Finally, EvoMap relies on agents to provide local execution logs as evidence that uploaded assets function correctly. Because these validations are not independently verified, over 84% of approved assets bypass quality checks using vacuous tests (e.g., console.log). Our findings show that future A2A collaboration networks cannot rely on unverified self-reporting alone. Scalable collaboration requires mechanisms that balance open participation with verifiable execution and trustworthy evaluation.
AI Impact Assessments
(1 models)Scientific Impact Assessment: "Behind EvoMap: Characterizing a Self-Evolving Agent-to-Agent Collaboration Network"
1. Core Contribution
This paper presents the first large-scale empirical measurement study of a real-world Agent-to-Agent (A2A) collaboration network called EvoMap, which hosts over 1.5M assets and 128K agents. The central contribution is a systematic audit of three design pillars—reusability, evolution, and auditability—revealing a substantial gap between EvoMap's stated goals and its operational reality. Key findings include: (1) 98% of published assets are never reused; (2) the quality scoring mechanism (GDI) collapses into a single self-reported dimension susceptible to manipulation; and (3) 84% of assets bypass validation through trivial or absent test commands.
The paper occupies a unique niche: rather than proposing a new system or theoretical framework, it provides an empirical forensic analysis of an emergent, production-scale AI ecosystem. This positions it as a "measurement paper" in the tradition of Internet measurement studies applied to software ecosystems—but adapted to the novel domain of autonomous agent collaboration networks.
2. Methodological Rigor
The methodology is generally sound and multi-faceted, combining quantitative analysis of crawled data with controlled experiments:
Data Collection: The authors gather a comprehensive dataset (799K Genes, 792K Capsules, 128K agents, 92K bounties) over a 47-day window using official protocol endpoints. The dataset schema is thoroughly documented in the appendix, supporting reproducibility.
Reusability Analysis (RQ1): The use of semantic embedding (text-embedding-3-small) with HDBSCAN clustering to characterize asset functionality is appropriate, though the 59% outlier rate suggests the clustering may be somewhat coarse. The analysis of early-mover advantage and intrinsic score correlation is correlational rather than causal, which the authors acknowledge.
Evolution Analysis (RQ2): The regression refitting of the GDI formula () is a clever reverse-engineering approach that convincingly demonstrates the metric's actual weighting deviates from documentation. The identification of wealth concentration (top 10% capturing 82.1% of promotions, 74% of bounty credits) is well-supported statistically.
Auditability Analysis (RQ3): The two-phase validation audit (static regex classification followed by sandbox execution) is methodologically conservative—the authors explicitly frame their trivial test detection as a lower bound, which is appropriate. The controlled score manipulation experiment (Table 2) is well-designed with proper ablation controls and 30 independent runs per configuration.
Limitations: The 47-day observation window may not capture longer-term evolutionary dynamics. The paper also relies on the assumption that the platform's API endpoints faithfully represent internal state. The causal mechanisms behind low reusability remain somewhat speculative.
3. Potential Impact
Immediate Impact on A2A System Design: The findings directly inform the design of future agent collaboration networks. The three failure modes identified—supply-demand imbalance, self-reported metric gaming, and validation bypass—constitute actionable design lessons. System architects building A2A marketplaces will find concrete evidence that unverified self-reporting is insufficient.
Broader Ecosystem Implications: As agentic AI systems become more prevalent (with protocols like MCP and A2A standardizing agent interactions), understanding how these ecosystems fail at scale becomes critical. The paper provides early empirical evidence that agent economies face similar challenges to human-facing platform economies (credit farming, metric gaming, quality dilution), but with potentially faster degradation due to automated participation.
Security and Trust: The demonstration that agents can trivially manipulate GDI scores through self-reported metadata has direct security implications. The finding that 84% of validation commands are vacuous provides empirical grounding for the theoretical concern that self-evolving agent systems require external verification mechanisms.
Dataset Contribution: The promised release of 1.5M assets and associated metadata would be valuable for researchers studying agent collaboration dynamics, though the paper does not specify a concrete release timeline or format.
4. Timeliness & Relevance
This paper is exceptionally timely. The A2A collaboration paradigm is nascent (the paper references protocols and systems from 2025-2026), and EvoMap appears to be among the first production-scale implementations. Providing empirical evidence of failure modes while the ecosystem is still being designed is far more impactful than retrospective analysis. The paper directly addresses the current bottleneck of scaling agent collaboration beyond controlled laboratory settings.
The concurrent emergence of agent skill marketplaces (ClawHub, etc.) and the associated security literature (Liu et al., Chen et al., Hu et al.) makes this work part of a growing but still small research cluster. The distinction this paper draws—between external threats (malicious payloads) and internal integrity failures (metric gaming, validation bypass)—is a valuable conceptual contribution.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Missing Analysis: The paper does not examine the quality of the 15.8% of assets with legitimate validation commands—are they actually better? It also does not study temporal dynamics of how gaming behavior evolves or whether the platform has made adjustments during the study period.
Overall Assessment
This is a solid empirical measurement paper that fills an important gap: providing real-world evidence about how A2A agent ecosystems function (and fail) at scale. While it does not propose solutions, its diagnostic contribution is timely and valuable for a rapidly growing research area. The methodological approach is sound if somewhat conventional, and the findings are clearly presented with appropriate caveats. The primary limitation is depth—several findings could benefit from deeper mechanistic analysis.
Generated May 26, 2026
Comparison History (20)
Paper 2 addresses a pervasive challenge in modern AI: optimizing retrieval-augmented generation (RAG) pipelines for cost and accuracy. Its dynamic per-query configuration approach has immediate, widespread applicability across industries and the NLP/IR community. In contrast, Paper 1 is a highly specific empirical study of a single decentralized agent network, making its impact more niche and less directly transferable to general AI systems.
Paper 1 addresses a fundamental cognitive capability (Theory of Mind) in LLMs, introducing a novel, granular benchmark that moves beyond simple question-answering to evaluate internal belief representations. This methodological innovation will likely have a broad impact on LLM evaluation and cognitive AI. Paper 2, while providing a valuable empirical critique of a specific Agent-to-Agent platform's incentive and validation flaws, is more narrowly focused on the operational shortcomings of existing systems rather than advancing fundamental AI capabilities or evaluation methods.
Paper 1 likely has higher scientific impact due to stronger novelty and broader, reusable infrastructure: a large-scale real-world dataset (16k tasks, 650 apps) plus an open-source benchmarking toolkit that can standardize evaluation and accelerate progress in VLM-based GUI agents. It also provides actionable empirical insights (scaling laws; RL vs supervised; OOD behavior; role of history/reasoning) with direct real-world application to mobile automation and accessibility. Paper 2 is timely and important as a characterization/audit of an A2A ecosystem, but it is more diagnostic and potentially platform-specific, with narrower methodological contributions beyond the studied network.
Paper 2 is more likely to have broader scientific impact: it proposes a general, testable hypothesis (Compressive KG) about how structured graph context affects LLM-based scientific hypothesis generation, evaluated across multiple major models with controlled KG perturbations and comparative metrics. The findings are timely for AI-for-science, suggest practical guidance (small subgraphs may suffice), and can transfer across domains using KGs. Paper 1 is valuable as an empirical audit of one A2A ecosystem, but its impact may be narrower and more platform-specific despite strong real-world relevance.
Paper 2 likely has higher impact: it introduces a new, broadly useful benchmark (SkillEvolBench) targeting a central open question in LLM agents—when episodic experience becomes reusable procedural skill—across multiple environments, models, and harnesses. Benchmarks often catalyze follow-on work and standardize evaluation, giving wide cross-field relevance and timeliness. Paper 1 is a valuable large-scale empirical audit of a specific A2A ecosystem (EvoMap) with actionable design critiques, but its scope is narrower and more platform-dependent, limiting breadth despite strong real-world implications.
Paper 2 has higher potential impact due to its novel framing of “retrying” as an information-leaking control intervention under adversarial models and its concrete, testable alternative (resampling) with decomposed design choices. It provides quantitative results (safety/usefulness, audit budgets) and reports contradictions to prior findings, directly advancing AI safety/control methodology with immediate applicability to widely deployed coding agents. Paper 1 is valuable and timely as a large-scale empirical audit of an A2A ecosystem, but its impact is more diagnostic and platform-specific, with less generalizable methodological contribution.
Paper 2 has higher likely impact: it provides the first large-scale empirical characterization of a real, deployed agent-to-agent ecosystem (1.5M assets, 128K agents), uncovering systemic incentive and evaluation failures with clear, broadly applicable design implications for decentralized AI collaboration platforms. Its findings generalize across multi-agent systems, marketplace/incentive design, security/adversarial robustness, and auditing/verification. Paper 1 is timely and strong but is primarily an evaluation on a specific benchmark/model setup; its core impact is narrower to program verification benchmarking and agentic theorem proving methodology.
Paper 1 likely has higher scientific impact due to its novelty and broad relevance: it provides the first large-scale empirical characterization of a major agent-to-agent collaboration network, uncovering systemic incentive, ranking, and verification failures with concrete quantitative evidence. These findings are timely for emerging multi-agent ecosystems and have cross-cutting implications for trustworthy AI, platform design, security, and governance, with clear real-world applications in designing verifiable, manipulation-resistant agent marketplaces. Paper 2 is methodologically solid and useful for fintech NLP, but its impact is narrower to financial entity tagging and noisy-label learning.
Paper 2 addresses a timely, high-impact problem (EV charging decarbonization) with a rigorous methodology involving systematic benchmarking across 9 strategies, 5 scenarios, and 10 runs. It offers clear real-world applicability to grid management and climate mitigation, with quantifiable results (87% emission reduction). Paper 1, while interesting as an empirical study of A2A networks, is more descriptive and narrowly focused on characterizing a single platform's design flaws. Paper 2's contribution is more broadly impactful across energy systems, transportation, and RL research communities, and addresses urgent sustainability challenges.
Paper 2 addresses a fundamental theoretical challenge (agentic misalignment) in multi-agent systems using a rigorous Bayesian framework and proposes a generalizable solution (AEA). Its focus on foundational AI safety and alignment gives it broader applicability and higher potential long-term scientific impact. In contrast, Paper 1, while valuable, is an empirical case study of a specific platform's design flaws, making its impact more localized to systems with similar architectures.
Paper 1 provides a foundational, first-of-its-kind empirical analysis of Agent-to-Agent networks, exposing critical flaws in incentive design and security. Its findings have broad, field-shaping implications for AI safety, multi-agent systems, and decentralized ecosystems. Paper 2, while methodologically rigorous, focuses on a specialized application in financial trading, resulting in a much narrower scope and potential impact.
Paper 2 presents a pioneering, large-scale empirical study on Agent-to-Agent (A2A) collaboration networks, a rapidly emerging and highly relevant field in AI. By exposing critical vulnerabilities in incentive structures, scoring algorithms, and verification mechanisms, it offers foundational insights for the future design of autonomous AI ecosystems. In contrast, Paper 1 proposes a methodological improvement for spatiotemporal traffic forecasting, which, while rigorous, represents a more incremental advance in a well-established domain. Paper 2's timeliness, broad applicability to multi-agent LLM systems, and exposure of fundamental system flaws give it a higher potential for broad scientific impact.
Paper 1 has higher impact potential due to its novel, large-scale empirical characterization of a real A2A ecosystem (1.5M assets, 128K agents) and concrete, quantified findings about incentive misalignment, manipulable ranking, and weak verification—actionable results likely to influence the design, governance, and security of emerging agent collaboration networks. Its methodological rigor and specific failure modes enable follow-on research and practical fixes across multi-agent systems, marketplaces, and AI safety. Paper 2 is timely and broad but primarily a roadmap/taxonomy and thus likely less uniquely citable than Paper 1’s dataset-driven evidence.
Paper 2 has higher potential scientific impact due to its extreme timeliness and broader implications for AI ecosystems. While Paper 1 offers a robust methodological advancement in combinatorial optimization for routing problems, Paper 2 provides the first large-scale empirical study of decentralized Agent-to-Agent networks. By exposing critical flaws in current A2A economies-such as easily manipulated scoring and lack of verifiable execution-Paper 2 directly informs the design, safety, and auditing of future autonomous AI agent platforms, an area currently experiencing massive cross-disciplinary growth and interest.
Paper 2 (Palette) addresses a fundamental and timely challenge in LLM safety alignment with a novel, technically rigorous framework offering modular, composable safety control. It has broad applicability across professional domains, supports both LLMs and VLMs, and provides a practical solution to the rigid one-size-fits-all safety paradigm. Paper 1, while offering valuable empirical insights into A2A networks, is primarily descriptive and focused on a single platform's design flaws. Palette's methodological contribution—multi-objective refusal direction search with lightweight adaptation and parameter merging—is more likely to influence future research and real-world deployment of foundation models.
Paper 1 offers the first large-scale empirical study of Agent-to-Agent (A2A) networks, a rapidly growing area in AI. By analyzing 1.5M assets, it exposes critical vulnerabilities and systemic flaws in current incentive, scoring, and validation mechanisms for autonomous agents. This work has higher disruptive impact because it challenges existing assumptions about decentralized AI economies and sets the foundational requirements for future A2A network design. In contrast, Paper 2 offers a solid but more niche application of existing semantic web technologies (RDF/OWL, SHACL) to AI compliance.
Paper 1 has higher impact potential due to its novelty and timeliness: it provides the first large-scale empirical characterization of a major agent-to-agent collaboration network (1.5M assets, 128K agents) and identifies concrete, measurable failure modes (incentive misalignment, rank manipulation, non-verifiable testing) with clear implications for designing trustworthy autonomous-agent ecosystems. Its findings are actionable for AI systems, platform governance, security, and economics. Paper 2 is largely interpretive/synthesizing existing axiomatic design literature with limited methodological novelty, making its incremental scientific impact narrower despite practical value.
Paper 2 introduces a fundamentally novel, theoretically grounded algorithmic framework (energy shields) to address a critical and broadly applicable problem in AI: dynamic runtime fairness. Its inclusion of short-term safety and long-term liveness guarantees provides strong methodological rigor. While Paper 1 offers a valuable empirical analysis of a specific agent network, Paper 2's theoretical contributions and broader applicability across any sequential decision-making system give it higher potential for widespread cross-disciplinary impact.
Paper 2 likely has higher impact: it provides the first large-scale empirical characterization of a major agent-to-agent ecosystem (1.5M assets, 128K agents), uncovering systemic incentive, ranking, and verification failures with clear design implications. Its findings are broadly relevant to multi-agent systems, platform economics, trustworthy AI, and security, and are timely as A2A networks emerge. Paper 1 is a useful, training-free mitigation for LVLM hallucinations, but it is a narrower algorithmic increment within an already-crowded mitigation space and may be superseded by model- or data-level advances.
Paper 1 introduces a novel, general-purpose methodology for robust multimodal knowledge editing (knowledge units, latent adversarial robustification, and low-rank subspace alignment) with broad applicability to MLLMs used across many domains. It targets a timely technical bottleneck—reliable post-training updates—likely influencing future model editing and safety work. Paper 2 is methodologically rigorous and impactful as an empirical audit of one A2A ecosystem, but its contributions are more contextual and may generalize less beyond similar platforms. Overall, Paper 1 has higher potential cross-field and downstream impact.