Not In My Git Yard: Catching Backdoors at Commit and Release Time
Dimitri Kokkonis, Michaël Marcozzi, Stefano Zacchiroli
A rigorous, timely, deployable CI-integrated backdoor detector addressing a hot supply-chain security gap, but an incremental extension of the authors' prior Rosa work with best-effort guarantees.
Abstract
Code-level backdoors-stealthy code changes that grant hidden privileges via secret triggers-pose a persistent threat to opensource software. Known attempts to inject such backdoors into widely used projects through malicious commits, tampered release packages, or compromised third-party dependencies, were stopped only by luck and manual review. Existing Continuous Integration (CI) pipelines cannot detect these attacks, and downstream binary analysis tools require substantial manual effort. In this work, we present Lily, an automated approach that strengthens open-source development and release processes against backdoor injection. Lily integrates a backdoor detection mechanism into (1) CI pipelines to block malicious commits, and (2) release vetting workflows to prevent tampered releases or compromised dependencies from entering large ecosystems, such as Linux distributions. Lily offers two key contributions. First, it enhances CI-compatible fuzzing with the capability to detect triggers of suspicious behavior based on historical and current software executions. This enables fast, precise backdoor detection suitable for both CI and update validation workflows. Second, it combines code change analysis with fuzzing data to precisely point maintainers to backdoor-revealing code regions, even when release updates modify millions of lines of code. We also outline five strategies attackers could use to evade Lily, and evaluate corresponding defenses. Our experiments across hundreds of benign and backdoored commits and releases show that Lily achieves high detection accuracy with low false alarm rates, reliably identifies malicious code, resists adversarial attempts, and would have prevented real-world backdoor incidents.
AI Impact Assessments
(1 models)Scientific Impact Assessment
1. Core Contribution
The paper introduces *Lily*, an automated system for detecting code-level backdoors at two critical junctures in the open-source lifecycle: (1) commit time within CI pipelines, and (2) release/dependency vetting time (e.g., Linux distribution package intake). The central technical novelty is a two-condition detection oracle operating on system-call profiles collected during short (10-minute) graybox fuzzing campaigns: a code change is flagged only when a fuzzer-generated input produces behavior that is both (i) *novel* relative to the pre-revision code and (ii) *atypical* relative to the revised code's normal execution profile. This conjunctive design is the key insight — each condition prunes the false positives generated by the other, which the ablation study shows is decisive (dropping false-alarm rates from 10–58% to 0.2–4.3%). A second contribution is the "suspicious code tracer," which correlates static diffs with dynamic traces to localize backdoors to a handful of lines even in multi-million-line release diffs. A thorough adversarial analysis of five evasion strategies (with a hardened "LilySelective" mode against corpus poisoning) rounds out the work.
2. Methodological Rigor
The evaluation is careful and follows current fuzzing evaluation norms (Schloegel et al. SoK): 545 version pairs across 13 diverse projects, 20 repeated trials per pair to address fuzzing non-determinism (>3600 CPU-hours), CI-realistic resource budgets (4 cores/16 GiB/10 min), a proper ablation isolating each oracle component, and Mann–Whitney U significance tests. The adversarial section is unusually mature for a systems-security tool paper — it enumerates concrete attacks, reproduces the multi-commit PHP evasion, and quantifies the poisoning attack and its mitigation. Weaknesses: only 3 of 13 backdoors are real-world incidents; the remaining 10 are synthetic (drawn from the authors' own Rosarum benchmark), which raises benchmark-authorship concerns. The comparison to the most relevant baseline (Ganz et al.) is only partial because the authors could not run that tool. Coverage limitations (avg. 32% reachable line coverage) mean the reported detection rates are contingent on harness maturity — a caveat the authors honestly foreground.
3. Potential Impact
The applied relevance is high. Supply-chain backdoors (XZ Utils, PHP, ProFTPD/vsFTPd) are among the most prominent security threats of the past few years, and the paper directly targets the CIFuzz/OSS-Fuzz ecosystem where deployment is plausible. The design constraints (fully automated, low false alarms, fits within existing fuzzing budgets) are exactly what would be needed for real adoption by OSS-Fuzz or distro maintainers. If integrated, it could meaningfully raise the cost of an entire attack class. That said, the approach is best-effort (no guarantees), constrained by harness reach, and only catches *future* injections — significantly narrowing the deployment ceiling.
4. Timeliness & Relevance
Extremely timely. The XZ Utils incident (2024) galvanized the community, and this paper explicitly positions itself in that moment. Backdoor detection under CI constraints is a genuine, currently-unsolved bottleneck, and the ASE'26 venue is appropriate.
5. Strengths & Limitations
Strengths: an elegant, well-motivated oracle design validated by a convincing ablation; strong reproducibility (code on GitHub, Software Heritage SWHID, Zenodo replication package, released seeds); thoughtful threat modeling; clear writing with an illustrative worked example (vsFTPd). Limitations: heavy conceptual reliance on the authors' prior Rosa work (this is an extension, not a fresh paradigm); dependence on system calls as the behavioral signal creates a theoretical evasion surface (attackers reusing existing syscall profiles) that the authors acknowledge cannot be fully ruled out; synthetic-benchmark dominance; no live-project deployment; harness-coverage dependence. The novelty is a smart recombination and adaptation of known ingredients (differential fuzzing, multi-version execution, syscall tracing) rather than a fundamentally new technique.
Other observations: The work is a system/tool paper of clear practical orientation. Its foundationality is moderate — the Rosarum benchmark and Lily tool are reusable, and the novel+atypical oracle concept may be adopted by follow-up regression-fuzzing work. Reproducibility is a genuine strength that should aid uptake.
Overall, this is a solid, well-executed, and timely security engineering paper with clear deployment potential, but its incremental relationship to Rosa and its best-effort, coverage-limited guarantees temper its ceiling.
Generated Jul 30, 2026
Comparison History (0)
A rigorous, timely, deployable CI-integrated backdoor detector addressing a hot supply-chain security gap, but an incremental extension of the authors' prior Rosa work with best-effort guarantees.