TitanCA: Lessons from Orchestrating LLM Agents to Discover 100+ CVEs
Ting Zhang, Yikun Li, Chengran Yang, Ratnadira Widyasari, Yue Liu, Ngoc Tan Bui, Phuc Thanh Nguyen, Yan Naing Tun
Abstract
Software vulnerabilities remain one of the most persistent threats to modern digital infrastructure. While static application security testing (SAST) tools have long served as the first line of defense, they suffer from high false-positive rates. This article presents TitanCA, a collaborative project between Singapore Management University and GovTech Singapore that orchestrates multiple large language model (LLM)-powered agents into a unified vulnerability discovery pipeline. Applied in open-source software, TitanCA has discovered 203 confirmed zero-day vulnerabilities and yielded 118 CVEs. We describe the four-module architecture, i.e., matching, filtering, inspection, and adaptation, and share key lessons from building and deploying an LLM-based vulnerability discovery solution in practice.
AI Impact Assessments
(3 models)Scientific Impact Assessment: TitanCA
1. Core Contribution
TitanCA presents a four-module pipeline that orchestrates multiple LLM-powered agents for automated vulnerability discovery in open-source software. The key architectural idea is decomposing vulnerability detection into specialized stages—matching (VulCoCo), filtering (R2Vul), inspection (VulTrial), and domain adaptation (PairVul)—rather than relying on a single monolithic model. The system's tangible output is impressive: 203 confirmed zero-day vulnerabilities and 118 assigned CVEs across 127,000+ GitHub repositories. This represents a meaningful real-world contribution to open-source security.
The core novelty is not in any single module (each is published or under review separately) but in the orchestration philosophy: composing specialized agents into a precision-maximizing pipeline with cost-aware ordering. This is more of a systems/engineering contribution than a fundamental algorithmic advance.
2. Methodological Rigor
This is where the paper has notable weaknesses. The paper is structured as a "lessons learned" article for IEEE Security & Privacy Magazine rather than a rigorous research paper, and this framing limits the depth of evaluation:
The individual modules reference separate publications (some accepted at ICSE 2026 and ACL 2026), which likely contain more rigorous evaluations. However, the pipeline-level evaluation—arguably TitanCA's main contribution—remains largely anecdotal.
3. Potential Impact
The practical impact is substantial and multifaceted:
The work could influence how security teams think about deploying LLMs—not as drop-in replacements for SAST tools, but as orchestrated pipelines with specialized roles.
4. Timeliness & Relevance
The paper addresses a highly timely intersection of two major trends: the explosion of LLM capabilities and the persistent challenge of software vulnerability management. As AI-generated code proliferates (the paper acknowledges this in its future work), automated vulnerability detection becomes increasingly critical. The just-in-time analysis approach aligned with CI/CD pipelines reflects current DevSecOps practices.
The multi-agent orchestration paradigm is also timely, riding the wave of interest in agentic AI systems while providing a concrete, deployed example rather than a theoretical framework.
5. Strengths & Limitations
Strengths:
Limitations:
Additional Observations
The paper's framing as a "lessons learned" article is both a strength and weakness. It makes the practical insights accessible but obscures the scientific contribution. The most impactful claim—that orchestration outperforms monolithic models—deserves rigorous empirical backing that this paper does not provide. The upcoming Phase 2 (expanding to cross-function analysis, repair suggestions, and developer behavior studies) suggests a productive research trajectory, but the current contribution is primarily a well-executed engineering effort with useful but preliminary scientific insights.
The collaboration model between university and government agency is noteworthy and could serve as a template for translational cybersecurity research.
Generated Apr 21, 2026
Comparison History (32)
Paper 2 likely has higher scientific impact due to its strong demonstrated real-world outcomes (203 confirmed zero-days, 118 CVEs) and timeliness in LLM-agent security automation, making it immediately relevant to both research and practice. Its multi-agent architecture and deployment lessons can transfer across vulnerability research, secure software engineering, and AI systems. Paper 1 is valuable as a SoK that can reshape NIDS evaluation norms, but its impact is more indirect and depends on community uptake; it offers less immediate measurable application and narrower domain reach.
TitanCA demonstrates immediate, concrete real-world impact with 203 confirmed zero-day vulnerabilities and 118 CVEs discovered in open-source software. It addresses a critical cybersecurity need with a practical, deployed system combining LLM agents for vulnerability discovery—a timely topic with broad implications for software security. While ZK-PoP is technically innovative in combining zero-knowledge proofs with authorship attestation, it addresses a more niche problem. TitanCA's validated results, practical deployment with a government agency, and relevance to the LLM-for-security trend give it broader and more immediate scientific impact.
TitanCA demonstrates concrete, measurable real-world impact by discovering 203 confirmed zero-day vulnerabilities and 118 CVEs in open-source software, representing a significant practical contribution to cybersecurity. While Paper 1 offers valuable theoretical insights into multi-turn safety failures with a novel state-space framework, Paper 2 combines methodological innovation (multi-agent LLM orchestration) with immediate, verifiable real-world applications. The practical deployment with GovTech Singapore and tangible security improvements give it broader impact across both AI and cybersecurity communities.
AEGIS addresses a fundamental scalability bottleneck in privacy-preserving AI inference—a problem of growing importance as LLMs are deployed in sensitive domains. It offers a novel co-design of application-level and encryption-level parallelism with strong quantitative results (3.86x speedup, 96.62% scaling efficiency). This opens a new research direction for practical FHE-based deep learning on multi-GPU systems. While TitanCA demonstrates impressive practical vulnerability discovery (118 CVEs), it is more of an engineering contribution with incremental novelty in LLM agent orchestration. AEGIS's methodological depth and broader applicability to privacy-preserving computation give it higher long-term impact.
TitanCA demonstrates higher scientific impact through its concrete, verified real-world results: 203 confirmed zero-day vulnerabilities and 118 CVEs in actual open-source software. This represents a paradigm shift in automated vulnerability discovery with immediate practical significance for cybersecurity. While GuardPhish addresses an important LLM safety gap with rigorous methodology, TitanCA's novel multi-agent LLM architecture for vulnerability discovery is more innovative, has broader cross-field impact (software security, AI agents, DevSecOps), and its validated real-world deployment distinguishes it from benchmark-oriented research.
Paper 2 demonstrates exceptional real-world impact by already discovering over 100 CVEs using an innovative LLM-agent architecture. Its application to general software vulnerability discovery offers much broader utility and timeliness compared to Paper 1, which focuses on a more narrow, hardware-specific vulnerability in UAVs. The large-scale practical validation and relevance to the rapidly growing field of LLM-assisted cybersecurity give Paper 2 a significantly higher potential for widespread scientific and industry impact.
TitanCA demonstrates substantial real-world impact by discovering 203 confirmed zero-day vulnerabilities and 118 CVEs in open-source software, providing concrete, measurable contributions to cybersecurity. Its practical deployment through a government-academic collaboration and actionable pipeline architecture gives it broader immediate impact. While Paper 1 introduces a novel jailbreak technique (HMNS) with interesting interpretability-informed methodology, it primarily advances adversarial attack capabilities rather than defense, and its impact is more narrowly focused on the AI safety research community. Paper 2's demonstrated real-world vulnerability discovery affects the entire software ecosystem.
TitanCA demonstrates higher scientific impact through concrete, measurable real-world results: 203 confirmed zero-day vulnerabilities and 118 CVEs discovered using LLM-powered agents. This represents a novel, practical application of LLM orchestration to cybersecurity with immediate real-world impact on software security. The collaboration between academia and government adds deployment credibility. Paper 2, while methodologically sound as a systematization of knowledge with a useful framework, is primarily a survey/analysis paper with incremental experimental contributions (synthetic perturbation experiments) and narrower applicability to the niche intersection of propaganda detection and privacy.
While Paper 1 offers a highly novel methodological intersection of cryptography and AI interpretability, Paper 2 demonstrates profound, measurable real-world impact by discovering over 100 confirmed zero-day vulnerabilities (118 CVEs). This massive empirical validation of LLM agents in automated cybersecurity provides immediate, broad-reaching implications for software security and AI application, likely driving extensive future research in agent orchestration.
Paper 1 demonstrates phenomenal real-world impact by discovering over 100 actual CVEs in open-source software using a novel orchestration of LLM agents. Its practical validation and high timeliness in the intersection of generative AI and cybersecurity give it a much higher potential scientific and industrial impact compared to Paper 2, which offers a solid but more incremental improvement tested on a single dataset.