Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles
Xiao Li, Yixuan Jia, Zekai Zhang, Xiang Li, Lianghe Shi, Jinxin Zhou, Zhihui Zhu, Liyue Shen
Abstract
Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners, yet the connection between these two abilities remains less explored. Drawing inspiration from self-supervised learning (SSL), we introduce a framework for jointly evaluating the representation and generation capabilities of diffusion models. Specifically, we decompose features into invariant and residual components and derive the Invariant Contamination Ratio (ICR), a Fisher-based metric that quantifies how residual variation contaminates invariant signal in feature space. We use this framework to analyze both discriminative and generative behavior of diffusion models. On the representation side, we find that invariance peaks at intermediate noise levels, which also yield the best downstream classification performance. On the generative side, we study how training transitions from genuine generalization to memorization in data-limited regimes, and show that ICR serves as a sensitive training-time indicator of early learning: increasing residual energy along Fisher directions marks the onset of memorization, detectable from training features alone without external evaluators or held-out test sets. Overall, our results show that diffusion models can be monitored from a self-supervised perspective through the geometry of their learned representations.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper bridges diffusion models and self-supervised learning (SSL) by proposing the Invariant Contamination Ratio (ICR), a label-free metric that quantifies how much augmentation/noise-sensitive residual variation contaminates the stable invariant signal in diffusion model feature spaces. The key idea is to decompose diffusion representations into an invariant component s (conditional mean over augmentations/noise) and a residual component ξ, then solve a generalized eigenvalue problem (Fisher-style) between their covariances to obtain a scalar diagnostic. ICR serves dual purposes: (1) identifying optimal noise levels for downstream representation tasks, and (2) detecting the onset of memorization during training in data-limited regimes without requiring generation or external evaluators.
Methodological Rigor
The framework is mathematically well-grounded. The invariance-residual decomposition is clean, leveraging the law of total covariance to separate Σ_h = Σ_s + Σ_ξ. The Fisher generalized eigenvalue formulation provides a principled way to measure directional signal-to-noise ratios, and the connection to Fisher Linear Discriminant Analysis (where each image is its own "class") is natural and well-motivated.
The two-view approximation for practical estimation is clearly derived, showing how sum and difference statistics recover the two covariance components. The paper includes extensive robustness analyses: sensitivity to augmentation design (Figure 14), layer selection (Figure 15), and sample complexity (Figure 13), all of which strengthen confidence in the metric's reliability.
However, some limitations exist. The theoretical result (Proposition 1) establishes monotonicity only in a simplified linear Gaussian setting, which is far from the nonlinear deep network regime where the metric is applied. The gap between theory and practice is acknowledged implicitly but not deeply discussed. Additionally, the memorization detection capability is demonstrated correlatively rather than with formal guarantees—the U-shaped ICR trajectory *precedes* memorization ratio increases, but the causal mechanism is not rigorously established.
Potential Impact
Practical utility: ICR's most compelling application is as a training-time diagnostic for diffusion models, particularly in data-limited settings. Unlike FID (which requires generation and external networks) or memorization ratios (which require generating thousands of samples and nearest-neighbor searches), ICR is computed purely from training features. This could reduce computational overhead for model monitoring significantly.
Noise level selection: The "semantic window" finding—that ICR minima coincide with optimal classification noise levels—provides a practical, label-free alternative to expensive grid searches over noise schedules for representation extraction.
Broader connections: The paper strengthens the conceptual bridge between SSL and diffusion models, which has implications for both communities. The observation that diffusion representations naturally satisfy SSL desiderata (invariance and expansion) at intermediate noise levels validates recent REPA-style approaches and could guide future hybrid training objectives.
Limitations in scope: The experiments are conducted on relatively standard benchmarks (CIFAR10/100, ImageNet at moderate resolutions) using EDM and SiT architectures. The applicability to large-scale latent diffusion models (Stable Diffusion, Flux) or text-conditioned settings remains unverified. The metric also requires choosing augmentation pipelines, which introduces some design decisions.
Timeliness & Relevance
This work addresses a timely need. Diffusion models are increasingly deployed in data-limited domains (medical imaging, scientific applications) where memorization is a serious concern. Existing memorization detection tools are expensive and post-hoc. A cheap, intrinsic training-time signal like ICR fills a genuine practical gap.
The SSL-diffusion connection is also highly topical, with concurrent works like REPA and spectral alignment approaches showing that aligning diffusion features with SSL objectives improves training. This paper provides analytical tools to understand *why* such alignment helps, complementing the engineering-oriented concurrent work.
Strengths
1. Elegant formulation: The invariance-residual decomposition is simple, interpretable, and connects naturally to classical statistical tools (Fisher discriminant, generalized eigenvalues).
2. Label-free and efficient: ICR requires no labels, no generation, and no external networks—a significant practical advantage over alternatives.
3. Dual utility: The same metric serves both representation evaluation (across noise levels) and training monitoring (across epochs), providing a unified diagnostic.
4. Thorough empirical validation: The paper includes multiple architectures (EDM, SiT), datasets, data regimes, and ablation studies on augmentation, layer choice, and sample complexity.
5. Clear presentation: The paper is well-organized with informative figures (especially Figure 1 as overview and Figure 2 for qualitative validation).
Limitations
1. Theory-practice gap: Proposition 1 covers only linear Gaussian models; the connection to deep nonlinear networks is purely empirical.
2. Limited scale: No experiments on modern large-scale models (latent diffusion, text-conditioned generation).
3. Augmentation dependence: While robustness to augmentation strength is shown, the framework still requires choosing a "sufficiently rich" augmentation pipeline, which is somewhat circular.
4. Memorization detection is correlative: ICR anticipates memorization onset but doesn't explain *why* or provide formal detection criteria (e.g., thresholds).
5. No comparison with other intrinsic metrics: While alignment/uniformity and silhouette scores are discussed, a more systematic comparison with RankMe, α-ReQ, and other SSL representation metrics would strengthen the contribution.
6. The metric doesn't suggest corrective action: ICR diagnoses problems but doesn't prescribe solutions (e.g., regularization strategies).
Overall Assessment
This is a solid analytical contribution that provides useful conceptual and practical tools for understanding diffusion models through the SSL lens. The ICR metric is well-motivated, practically efficient, and demonstrates clear empirical utility across multiple settings. The main limitations are the restricted experimental scale and the gap between the theoretical analysis and practical usage. The work is likely to influence how practitioners monitor diffusion model training and select noise levels for representation extraction.
Generated Jun 9, 2026
Comparison History (20)
Paper 2 likely has higher scientific impact due to broader cross-domain relevance and timeliness: it proposes a general evaluation framework and metric (ICR) for diffusion models that connects representation learning, generalization, and memorization—issues central across ML, vision, and generative modeling. Its potential applications include training diagnostics and model selection without external evaluators, which could influence many diffusion-based pipelines. Paper 1 is rigorous and impactful within neural population modeling/BCI, but its scope is narrower and depends on a specific benchmark and setting.
Paper 1 introduces a novel, rigorous theoretical framework (Invariant Contamination Ratio) that bridges representation and generation in diffusion models. Crucially, it provides a method to detect the onset of memorization using only training features, without needing held-out test sets. While Paper 2 is a highly relevant and timely survey on LLM efficiency, Paper 1 presents original, fundamental research that solves an open problem in generative AI. Its innovative approach to evaluating representation space offers deeper methodological advancements and foundational scientific impact compared to a synthesis of existing literature.
Paper 2 introduces a novel theoretical framework connecting diffusion models and self-supervised learning, with broader conceptual contributions including the ICR metric and insights into memorization detection. It addresses fundamental questions about representation learning in generative models, impacting multiple research areas (generative modeling, SSL, representation learning). Paper 1, while technically solid, presents an incremental optimization improvement for post-training quantization—a well-studied problem with narrower scope. Paper 2's interdisciplinary bridging and diagnostic tools have wider potential influence.
Paper 1 introduces a novel framework connecting diffusion models' generative and representational capabilities through self-supervised learning principles, with practical applications like detecting memorization onset during training. This addresses a timely and broadly impactful topic given the widespread adoption of diffusion models. Paper 2, while technically rigorous with tight bounds for ReLU regression with queries, addresses a more specialized theoretical learning theory problem with narrower immediate impact. The breadth of relevance and timeliness of diffusion model analysis gives Paper 1 higher potential impact.
Paper 1 introduces a novel theoretical framework connecting diffusion models' generative and representational capabilities through self-supervised learning principles, with the ICR metric offering broad utility for understanding and monitoring diffusion model training (e.g., detecting memorization onset). This addresses fundamental questions about widely-used generative models with broad cross-field implications. Paper 2, while technically solid, addresses a more specialized engineering problem (communication compression in pipeline parallelism) with narrower impact scope and incremental improvements over existing methods.
Paper 1 offers a more novel, timely contribution by linking diffusion models’ generative and representation-learning behavior via a principled Fisher-based metric (ICR) and providing a training-time signal for memorization onset without external evaluators. This is methodologically grounded and directly relevant to today’s widespread diffusion-model deployment, with potential broad impact across generative modeling, SSL, and model monitoring/interpretability. Paper 2 is useful for curriculum learning, but its reliance on pre-existing model series may limit generality and immediate adoption compared to Paper 1’s broadly applicable diagnostic framework.
Paper 2 offers broader scientific impact by providing a unifying theoretical framework, a novel benchmark, and a new method (CoSAE) for concept alignment across diverse models and modalities. This addresses a critical, widespread challenge in multimodal AI and interpretability. While Paper 1 provides valuable insights into diffusion models and memorization, its scope is narrower. Paper 2's ability to bridge disparate alignment methods and demonstrate high sample efficiency promises foundational utility across a wider range of machine learning disciplines.
Paper 1 has higher impact potential: it introduces a clear, causally grounded validation criterion (“closure”) for circuit discovery from co-activation clustering, directly addressing a widely used but weakly validated interpretability workflow. It demonstrates the method across multiple 1B dense models and input distributions, and provides a salient negative result in an MoE setting plus training-time analyses showing metric/function decoupling—findings likely to reshape best practices across mechanistic interpretability and model analysis. Paper 2 offers a useful evaluation metric for diffusion representations, but is more incremental within existing SSL-style probing and may have narrower cross-field methodological implications.
Paper 1 addresses fundamental challenges in understanding diffusion models, a highly active and cross-disciplinary field. The introduction of a metric to detect memorization without a test set offers broad, foundational utility for generative AI and representation learning. Paper 2, while offering important contributions to safety in offline RL, addresses a more specialized problem and is likely to have a narrower overall scientific impact.
Paper 1 addresses a critical, practical failure mode in federated learning (knowledge resurfacing), which has immediate real-world implications for privacy, security, and legal compliance (e.g., GDPR's right to be forgotten). By ensuring unlearned data remains forgotten during subsequent training, it solves a pressing bottleneck in deploying ML models safely. While Paper 2 offers valuable theoretical insights into diffusion models, Paper 1 provides a highly timely, robust solution to an urgent privacy challenge, likely leading to broader real-world adoption and higher overall scientific impact.