Sampling and Curvature in First-Exit Comparisons for Momentum SGD
Haiyi Li
Rigorous, honest obstruction results and a clean persistence-to-telegraph bridge, but narrow regime, loose certificates, few-seed neural evidence and a sprawling presentation limit impact.
Abstract
First-exit prediction for momentum SGD depends on sampling information that pointwise gradient laws omit. Strongly convex finite pools with identical pointwise laws and matched iid momentum dynamics can have different reshuffled exits; even analytic pools with uniformly close pointwise laws can have separated exit probabilities. For persistent sampling in one dimension, we establish an explicit mean-exit comparison with a finite-rate switching process under nonvanishing component velocities and step-size conditions; iid sampling instead averages to an ODE. Retaining realized minibatches and sampled curvature yields finite-window exit brackets beyond this regime. In 32 trained ridge-logistic runs, 29 meet prespecified targets with rounding-certified brackets; an enclosure control explains most of the width advantage. Exact-real certificates establish finite means at the original tested radii for three ResNet-18 checkpoints and give local finite-mean conditions for tanh and ViT networks; their upper bounds remain loose, and original-radius ViT mean guarantees remain unresolved despite 128 fresh continuation paths. A two-network replication gives 32/32 successful seeds in each of four finite-path persistence comparisons. Independent equal-length controls retain slower exits after refresh resets than without resets, but the refresh-versus-random ordering reverses with update length at both networks. These interventions do not causally decompose raw momentum SGD. Neural step-size scaling does not validate the switching limit.
AI Impact Assessments
(1 model)Core Contribution
The paper asks what a first-exit prediction for momentum SGD must retain beyond the one-step minibatch gradient law. It makes four linked claims.
1. Information. Explicit finite-pool constructions with identical pointwise gradient laws (and identical iid dynamics) have different two-step reshuffled exit probabilities (1/2 vs 0). Analytic pools with arbitrarily close pointwise W∞ laws also remain separated. So no calibration from one-step noise statistics can determine reshuffled exits.
2. Continuum limit. For persistent samplers (data echoing) in one dimension with sign-definite component velocities, Theorem 1 gives an explicit O(h) mean-exit bracket against a finite-rate switching (telegraph/PDMP) process. iid sampling instead averages to an ODE, and linear Gaussian momentum recursions cannot converge to a telegraph.
3. Computable comparisons. Retaining realized minibatches and sampled curvature yields pathwise finite-window CDF/RMST brackets. 29 of 32 trained ridge-logistic runs meet the prespecified targets, with rounding-certified brackets. The paper is candid that an enclosure control closes about 90% of the width gap.
4. Neural networks. Persistent sampling with moderate echo counts exits faster than iid (ratios 0.4–0.7), while heavy echoing exits slower. The effect needs heavy-ball momentum and is tied to practical step sizes, not the switching limit. Reset-intervention and replication studies are included.
The conceptual message, that sampler memory matters for escape-time modeling and that SDE or telegraph models calibrated on marginal noise are structurally incomplete, is clear and useful. The paper also reports a number of negative or limiting findings.
Methodological Rigor
The theory is careful. The counterexamples are exact rational enumerations. The bridge theorem has explicit constants and is checked numerically: 82 applicable cells, with the bounds conservative by a median factor of 8.2. The pathwise tube argument is sound.
The empirical work is unusually pre-registered, but it rests on 3–4 training seeds per network. The paper concedes the bootstrap intervals are descriptive only. Several claims are explicitly limited:
The counterexamples are small, hand-built pools, and the 1D bridge assumptions (no stationary points within any component) exclude realistic minibatch regimes. This restricts the link between the theory and the neural experiments.
Potential Impact
The conceptual point (pointwise laws are insufficient for reshuffled exits) will interest the SGD-dynamics and escape-time communities. The data-echoing/persistence finding, with a non-monotone dose response, is a practical empirical observation. The certified-bracket machinery is niche: it is low-dimensional, costs about 40 times direct simulation, and offers no new estimator. The finite-mean certificates for neural networks are of mostly theoretical value given their looseness.
Likely uptake is modest, mainly among theorists of SGD continuum models and sampling-aware analyses. The paper does not offer a usable predictive model: its surrogates have RMST errors above 100%.
Timeliness
Continuum SDE models of SGD are widely used, and critiques of them are timely. Non-iid sampling (shuffling, echoing) is topical. However, the paper's heavy emphasis on first-exit times in toy and local settings limits its connection to mainstream concerns such as generalization.
Strengths
Limitations
Overall this is a competent, careful but narrow and over-extended paper, with moderate expected impact.
Generated Oct 20, 2026
Comparison History (0)
Rigorous, honest obstruction results and a clean persistence-to-telegraph bridge, but narrow regime, loose certificates, few-seed neural evidence and a sprawling presentation limit impact.