Fourier fractal dimension to predict the generalization of deep neural networks
Joao B. Florindo, Davi Wanderley Misturini
Abstract
Predicting the generalization performance of deep neural networks without relying on hold-out validation data is a fundamental challenge in machine learning. While Stochastic Gradient Descent (SGD) drives the optimization of these highly parameterized models, its heavy-tailed, non-Gaussian dynamics induce complex, scale-invariant trajectories in the parameter space. In this paper, we propose a novel generalization measure based on the Fourier fractal dimension of the network's weight variations. By analyzing the characteristic function of the Lévy-driven stochastic differential equations in the frequency domain, we extract a metric that robustly captures the geometric complexity of the learning process. Furthermore, we introduce a customized Fourier-based optimizer designed to actively regularize this fractal dimension during training. Extensive empirical evaluations on the CIFAR-10, SVHN, and MNIST datasets demonstrate that our proposed Fourier generalization measure exhibits a strong correlation with the actual generalization gap. Our method achieves state-of-the-art Kendall rank correlation coefficients, outperforming a wide array of existing norm-based, margin-based, and PAC-Bayesian measures. Ultimately, this work highlights the potential of frequency-domain fractal analysis as both a powerful predictor for model generalizability and a principled foundation for developing more stable optimization algorithms.
AI Impact Assessments
(1 models)Scientific Impact Assessment: "Fourier fractal dimension to predict the generalization of deep neural networks"
1. Core Contribution
The paper proposes using the Fourier fractal dimension of neural network weight variations as a predictor for the generalization gap. The theoretical motivation connects the heavy-tailed dynamics of SGD (modeled as Lévy-driven SDEs) to frequency-domain fractal analysis via the characteristic function. The key insight is that the stability index α of the Lévy process—which governs the ability to escape narrow minima—can be estimated from the power-law decay of the Fourier transform of weight variations. Additionally, a "Fourier-based optimizer" is proposed that regularizes this fractal dimension during training.
The conceptual bridge between Lévy process characteristic functions and Fourier fractal dimension is the most interesting aspect, building upon prior work by Simsekli et al. (2019, 2020) on heavy-tailed SGD dynamics and fractal geometry in generalization.
2. Methodological Rigor
Theoretical development: The mathematical exposition is relatively thin. The connection from the Lévy-Khintchine formula to |ϕ(u)|² = exp(-2|σu|^α) and then to fractal dimension estimation via log-log regression is straightforward but underdeveloped. The paper acknowledges that modeling W_t as a d-dimensional α-stable process is a "reasonable approximation" (supported by Figure 1), but provides no formal justification for when this approximation breaks down or how robust the measure is to deviations.
Experimental evaluation: This is the paper's weakest point. The evaluation is conducted on only three relatively simple datasets (CIFAR-10, SVHN, MNIST) with what appears to be a single architecture (modified AlexNet). Critical experimental details are missing:
The claimed state-of-the-art Kendall coefficients (0.680, 0.672, 0.551) cannot be properly evaluated without knowing whether the experimental setup matches the benchmark conditions.
Optimizer description: Section 4.3 is alarmingly vague. There is no algorithm pseudocode, no mathematical formulation of the update rule, and no clear explanation of what "enforces general reduction on the magnitude of the Fourier transform" means in practice. The sentence acknowledging that computing the dimension over parameter evolution across epochs "would be impractical" and that they instead compute it over the parameter tensor spatially represents a significant approximation that fundamentally changes what is being measured—yet this receives minimal discussion.
3. Potential Impact
The core idea—connecting frequency-domain fractal analysis to generalization—is genuinely interesting and could inspire further research. If properly validated, a Fourier-based generalization measure could be computationally attractive compared to trajectory-based fractal dimension estimates. However, the limited scope of experiments and missing details significantly undermine confidence in the practical utility of the method.
4. Timeliness & Relevance
Predicting generalization without validation data remains an important open problem, particularly for AutoML and neural architecture search. The paper addresses a real need. However, the comparison baseline is exclusively from Jiang et al. (2019), ignoring more recent developments in generalization prediction from 2020-2025 (e.g., persistent homology approaches, data-dependent fractal dimensions, and other recent measures). This makes the claimed state-of-the-art status questionable.
5. Strengths & Limitations
Strengths:
Limitations:
Overall Assessment
The paper presents an intriguing conceptual contribution—connecting Fourier fractal analysis to generalization prediction via Lévy process theory. However, the execution falls substantially short of what would be needed to convincingly establish this as a state-of-the-art method. The experimental evaluation is too narrow, the optimizer is inadequately described, comparisons with recent work are absent, and several critical methodological details are missing. This reads as an early-stage exploration of a promising idea rather than a mature contribution ready to influence the field.
Generated Jun 9, 2026
Comparison History (18)
Paper 1 addresses a fundamental and ubiquitous challenge in deep learning: predicting generalization without validation data. Its novel use of Fourier fractal dimensions to analyze SGD dynamics offers profound theoretical insights and practical optimization benefits. In contrast, Paper 2 presents a more incremental, albeit useful, methodological improvement to Bayesian particle flow by integrating Deep Sets. Paper 1 has significantly broader applicability across the entire machine learning community and a higher potential to influence foundational deep learning theory.
Paper 2 has broader potential impact: a generalization predictor and optimizer applicable across many deep-learning domains, addressing a widely relevant and timely problem. Its approach is more novel (frequency-domain fractal analysis tied to Lévy/SGD dynamics) and could influence theory, diagnostics, and optimization practice. Paper 1 targets an important but narrower niche (offline safe RL robustness to data poisoning) with strong applied value, yet likely impacts a smaller community and depends on specific threat models/benchmarks.
Paper 1 tackles a fundamental and widespread challenge in deep learning: predicting generalization without validation data. Its novel approach using Fourier fractal dimension offers both a new theoretical metric and a practical optimization tool, potentially impacting a vast array of deep learning applications. Paper 2, while methodologically rigorous and practically useful for recommendation systems, focuses on a narrower variant of multi-armed bandits. The broader applicability and fundamental theoretical insights of Paper 1 give it a significantly higher potential for widespread scientific impact across the machine learning community.
Paper 1 tackles a fundamental theoretical challenge in deep learning (predicting generalization) with a highly novel mathematical approach, directly advancing AI theory and optimization. Paper 2 presents a valuable software engineering tool for checkpoint management, which, while highly practical for researchers, offers less fundamental scientific innovation and theoretical impact compared to Paper 1.
Paper 1 has higher potential impact due to timeliness and real-world applicability: adaptive tensorization for weight/KV-cache compression directly targets deployment bottlenecks of LLMs (memory, latency, cost) and could be broadly adopted. Its novelty—discovering low-rank structure via index ordering—offers a practical compression lever for foundation models. Paper 2 is conceptually interesting (fractal/Fourier generalization metric and optimizer) but risks narrower applicability and weaker external validity since results are on small vision benchmarks; generalization measures often struggle to transfer to modern large-scale settings.
Paper 1 addresses a fundamental and highly relevant challenge in machine learning (predicting DNN generalization) with a novel theoretical approach and practical optimizer. Its impact spans the broad and rapidly growing field of AI. In contrast, Paper 2 is a replication and methodological critique focused on a specific, niche dataset regarding airline profit cycles, which offers significantly less novelty and a much narrower breadth of impact.
Paper 2 likely has higher impact due to immediate real-world applicability and breadth: it operationalizes a repeatable, evolving benchmark-generation pipeline for enterprise Text2Cypher, addressing governance, executability, privacy/redaction, and evaluation at scale. The methodology includes audited ablations, human-judge calibration, and downstream model evaluation, strengthening rigor and reproducibility. Its timeliness aligns with enterprise LLM deployment needs and benchmarking gaps across domains using property graphs. Paper 1 is novel but narrower in scope (generalization proxy/optimizer) and may face challenges in theoretical validation and adoption beyond studied settings.
Paper 2 likely has higher scientific impact due to strong real-world applicability (early detection of diverse treatment complications using routine labs), large-scale longitudinal dataset, external validation across independent systems (MIMIC-IV, CoMMpass), and breadth across clinical categories and cancers. The transformer-based trajectory modeling is timely and directly translatable to clinical surveillance without new infrastructure, increasing adoption potential. Paper 1 is novel methodologically, but impact may be narrower and dependent on broader acceptance/validation of fractal metrics for generalization beyond the evaluated benchmarks.
Paper 2 has higher likely impact: it targets a timely, widely relevant problem (cost/performance optimization in LLM deployment) with clear real-world applications and immediate adoption potential. The meta-learning + contextual bandit formulation for personalized routing generalizes across users and model sets, giving broad systems/ML/industry relevance. Paper 1 is innovative but more niche (generalization measures via fractal/Fourier analysis) and its practical utility and rigor may be harder to validate beyond limited benchmarks, reducing near-term cross-field uptake.
Paper 2 likely has higher impact due to clear real-world utility (real-time CFD surrogates for indoor environment optimization), strong timeliness (fast generative alternatives to diffusion), and broader adoption potential in engineering workflows where inference speed is critical. Its methodological contribution (conditional drifting in VAE latent space with boundary-condition alignment and a path toward unseen geometries) is directly actionable and could influence both ML-for-physics and applied CFD communities. Paper 1 is novel but more speculative, with impact hinging on robustness across architectures/tasks and theoretical validation beyond benchmark correlations.