OCCAM: Open-set Causal Concept explAnation and Ontology induction for black-box vision Models
Chiara Maria Russo, Simone Carnemolla, Simone Palazzo, Daniela Giordano, Concetto Spampinato, Matteo Pennisi
Abstract
Interpreting the decisions of deep image classifiers remains challenging, particularly in black-box settings where model internals are inaccessible. We introduce OCCAM, a framework for open-set causal concept explanation and ontology induction in vision models. OCCAM discovers visual concepts in an open-set manner, localizes them via text-guided segmentation, and performs object-level interventions by removing concepts to measure changes in class confidence, estimating each concept's causal contribution. Beyond local explanations, OCCAM aggregates interventional evidence across a dataset to induce a structured concept ontology that captures how classifiers globally organize visual concepts. Reasoning over this ontology reveals consistent dependencies between concepts, exposes latent causal relations, and uncovers systematic model biases. Experiments on Broden and ImageNet-S across multiple classifiers show that OCCAM improves explanation quality in open-set black-box settings while providing richer global insight than per-image attribution methods.
AI Impact Assessments
(1 model)Scientific Impact Assessment: OCCAM
1. Core Contribution
OCCAM introduces a framework for explaining black-box vision classifiers through three interconnected mechanisms: (1) open-set concept discovery via multimodal LLMs, (2) causal contribution estimation through input-level interventions (concept removal via inpainting), and (3) ontology induction by aggregating interventional evidence across a dataset into a structured knowledge graph. The key novelty lies in combining these three elements — open-set discovery, interventional causality, and global ontology construction — into a unified pipeline that operates under strict black-box constraints.
The problem addressed is genuine: most concept-based explanation methods require either white-box access, predefined concept vocabularies, or both. OCCAM removes both constraints simultaneously, which is a meaningful practical advance for deployed systems and API-accessible models.
2. Methodological Rigor
Strengths in methodology:
Weaknesses in methodology:
3. Potential Impact
The framework addresses a real need in deployed AI systems where model internals are inaccessible. The ontology induction component is conceptually appealing — moving from per-image explanations to a global, queryable knowledge structure could be valuable for model auditing, bias detection, and regulatory compliance.
The extension to multimodal classifiers (Table 4) demonstrates breadth, and the finding that SigLIP distributes reliance more broadly than CLIP variants is an interesting architectural insight.
However, practical impact may be limited by computational cost (running an MLLM + SAM3 + LaMa inpainting per concept per image is expensive) and by the reliance on the quality of foundation model components. The explanation quality is fundamentally bounded by the MLLM's ability to propose relevant concepts and SAM3's segmentation accuracy.
4. Timeliness & Relevance
The paper is timely in leveraging the current ecosystem of foundation models (LLMs, SAM, inpainting models) for explainability. The black-box constraint is increasingly relevant as models are deployed behind APIs. The ontology induction angle connects XAI to knowledge representation, which is an underexplored direction.
However, the dependence on multiple large foundation models (Gemma 27B, SAM3, LaMa) means the "explainer" is arguably more complex than many models being explained, raising questions about the explanation's own transparency and reliability.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations
The volleyball example (Fig. 2) where classifiers identify "hair" and "smiling faces" as important for volleyball classification is presented as revealing model biases — but this could also reflect limitations of the MLLM's concept proposals rather than genuine classifier behavior. Disentangling the two is not addressed.
The paper is from a single institution (University of Catania) with six authors, suggesting a focused effort. The writing is generally clear but occasionally conflates evaluation-only use of white-box methods with the black-box claims.
Generated May 19, 2026
Comparison History (22)
Meta-Agent addresses a critical and timely problem in AI—reliable multi-agent system construction and execution—with a comprehensive framework featuring novel error attribution and recovery mechanisms. The multi-agent systems space is rapidly growing with broad applications, giving it wider impact potential. While OCCAM contributes meaningfully to explainability with its causal concept ontology approach, it addresses a more specialized niche within interpretability. Meta-Agent's integration of planning, grounding, and verification offers a more broadly applicable architectural contribution with immediate practical utility across diverse AI deployment scenarios.
Paper 2 (OCCAM) likely has higher scientific impact due to broader, timely relevance: interpretable AI for black-box vision models is a major cross-domain need (trust, safety, auditing). Its combination of open-set concept discovery, causal interventions, and ontology induction offers novel global+local explanatory structure with clear real-world applications (bias detection, compliance, model debugging) across many vision systems. Paper 1 is technically strong and valuable for KG reasoning, but its impact is more specialized to multi-variable logical query answering and niche benchmarks.
OCCAM addresses the broadly important problem of explainability in deep learning with a novel combination of open-set concept discovery, causal interventions, and ontology induction for black-box vision models. It has wider applicability across computer vision, XAI, and model auditing, with empirical validation on standard benchmarks. Paper 2 offers an elegant formalization of trust calibration as preference learning but is more niche, primarily theoretical, and lacks empirical validation. OCCAM's methodological contributions and breadth of impact give it higher potential scientific influence.
Paper 1 addresses the urgent and highly relevant issue of safety and vulnerabilities in emerging Large Reasoning Models (LRMs). Given the rapid deployment of reasoning LLMs, understanding and mitigating their jailbreak vulnerabilities is critical. The novel use of attention-guided reinforcement learning provides a rigorous approach to AI safety, likely leading to broader immediate impact and real-world application in securing foundational models compared to the more mature field of vision model explainability addressed in Paper 2.
Paper 1 addresses the highly timely intersection of LLMs and personalized healthcare, with a large-scale evaluation (2,257 queries, 1,945 PHRs) and both automated and clinician ratings. It introduces a novel evaluation framework for PHR-specific error modes, has direct real-world healthcare applications affecting millions of patients, and tackles the critical challenge of making complex health records actionable. Paper 2 contributes meaningfully to XAI with causal concept explanations, but explainability methods face a more crowded field. Paper 1's broader societal impact potential and timeliness in the LLM-health space give it the edge.
Paper 2 tackles a fundamental and pressing challenge in AI—interpreting black-box vision models via causal concept explanation and ontology induction. Its focus on open-set discovery and causal intervention offers broad implications for AI safety, trust, and explainability across multiple domains. In contrast, while Paper 1 presents a highly practical and commercially valuable application (NL2SQL), its multi-agent orchestration approach represents a more incremental methodological advancement with a narrower scientific scope compared to the foundational XAI contributions of Paper 2.
Paper 2 presents a concrete, technically novel framework with demonstrated empirical results: open-set concept discovery, causal interventions in black-box vision models, and dataset-level ontology induction. This combination is methodologically substantive and likely to be reusable across interpretability, robustness, and responsible AI, with clear real-world applicability to auditing deployed vision systems. Paper 1 is a vision/conceptual piece about trust in agent-to-agent networks; timely and potentially broad, but it appears less methodologically rigorous and offers fewer immediately testable contributions, which typically lowers near-term scientific impact.
Paper 2 addresses a critical real-world problem—predicting challenging behaviors in children with profound autism in actual classroom settings—with clear translational potential for safety and education. It bridges wearable sensing, foundation models, and special education in a novel real-world deployment context, moving beyond controlled lab settings. While Paper 1 makes solid contributions to XAI with causal concept explanations, the field is increasingly crowded. Paper 2's direct humanitarian impact, interdisciplinary reach (healthcare, education, ML, sensor technology), and practical applicability to an underserved population give it higher potential impact.
Paper 1 addresses a fundamental challenge in AI explainability (XAI) for black-box vision models using a novel causal framework and ontology induction. Its methodology for exposing systematic biases and latent relations has broad implications across multiple domains utilizing computer vision, such as healthcare and autonomous driving. In contrast, Paper 2 focuses on a more specialized, domain-specific application (cyber threat intelligence knowledge graphs), limiting its broader scientific impact compared to the fundamental AI transparency advances proposed in Paper 1.
Paper 1 addresses the critical, foundational problem of black-box model interpretability using causal concept explanation and ontology induction. Its theoretical depth and focus on uncovering causal relations and systematic biases offer broader implications for AI safety and trustworthy machine learning. While Paper 2 presents a highly practical systems-level optimization for web agents, Paper 1's contributions to explainable AI and causal inference are likely to spur a wider range of foundational follow-up research and have a more profound scientific impact.