LLC estimates are only as good as their hyperparameters

This post and [Linear does not mean regular] are two halves of one investigation. That one asks how the local learning coefficient scales with width in the lazy and rich regimes; this one is about why the first answer I got was backwards, and what has to be true of the estimator for the second one to be believed. Either can be read first.

My initial configuration reported the µP network 16.9× higher at width 1024 — the ordering reversed relative to the correct answer, and the magnitude inflated by more than an order of magnitude.

That was not a bug, and I should be plain that it was not standard practice either. I had set the localization strength to $\gamma = 5000$, roughly two orders of magnitude above the range the LLC literature actually recommends [Lau et al.], and I had let the sampler inherit the model's own per-tensor learning-rate scaling, which nobody suggests doing. Both were my choices. What makes them worth a post is not that they were conventional but that neither is visible in a reported result: the estimator returns a number indexed by a scale the practitioner picks, and that scale is rarely written down.

It is structurally the same problem as damping in influence functions, which is where I first met it, so the argument starts there.

Some years ago I worked on why influence functions are fragile in deep networks. [Epifano et al.] The influence of a training point on a test loss is

[Koh & Liang] and that expression requires $H$ to be invertible, which in practice requires positive definiteness, which in turn requires a unique non-degenerate minimum. Deep networks satisfy none of these. The Hessian has negative eigenvalues, the stochastic inverse-Hessian-vector product does not converge, and the standard remedy is damping: $(H + \lambda I)^{-1}$, with $\lambda$ set to a small constant such as 0.01. We found the resulting estimates degraded sharply with parameter count and argued that part of the observed fragility originated in the validation procedure rather than the method.

What I did not appreciate then is that this is a singular-model problem, and that SLT names it precisely. Kreer et al. make the same diagnosis in that language: degenerate loss landscapes have non-invertible Hessians, which violates the implicit function theorem conditions influence functions require, and damping "adds hyperparameter complexity without principled justification." [Kreer et al.] Their remedy is to avoid inversion entirely and take a covariance under a localized posterior,

which is the same localized posterior the learning coefficient estimator samples from.

This is not an analogy. In the quadratic regime the estimator computes

and $\sum_i h_i/(h_i+c)$ is the effective degrees of freedom of ridge regression at penalty $c$, a standard quantity in classical regression analysis. The learning coefficient estimate at finite $\gamma$ is therefore a damped-Hessian statistic, the same family of object as the damped influence function, computed by a different route.

That identification is checkable. Computing $\tfrac12\,\mathrm{tr}\!\left[H (H + cI)^{-1}\right]$ directly, via Hutchinson probes with conjugate-gradient solves and no sampling, reproduces the SGLD estimate to three significant figures on the µP network at width 1024: 3.80 against 3.80 at $\kappa = 1$, and 16.14 against 16.29 at $\kappa = 10$.

The convention is to calibrate the sampler's hyperparameters once per experiment and hold them fixed across the family so that cells remain comparable. This is sound when the family is homogeneous. Mine is not. Top curvature varies by about 400× across my cells, from 0.82 for the NTK network at width 1024 to 346 for the µP network at the same width. Since a direction registers only when $n\beta h \gg \gamma$, a single $\gamma$ occupies a different position on the response curve in every cell.

My first pass used $\gamma = 5000$ throughout, two orders of magnitude above the range the LLC literature recommends. [Lau et al.] At that setting the NTK network's stiffest direction gives $n\beta\lambda_{\max}/\gamma = 0.025$, so its entire spectrum sat below the damping and the estimate reported a rescaled Hessian trace. The µP network at the same setting sat above it.

A second and subtler version of the same error was also present. My sampler inherited the model's parametrization, so the µP network's chain used $\varepsilon/\texttt{fan\_in}$ on hidden weights, a 1024× smaller step than the NTK network's, in the layer holding 56% of the parameters. The localization mode relaxes in $\tau \approx 2/(\epsilon_t \gamma)$ steps, so the two models equilibrated on timescales three orders of magnitude apart, and at a shared step budget the µP chains had not converged. An unequilibrated chain underestimates the coefficient, inflating the gap being measured.

Together these produced an incorrect result: the µP network 16.9× higher at width 1024, with the ordering reversed relative to the reference computation.

A dimensionless correction

Both faults have the same structure, a quantity carrying units held numerically constant across cells whose curvature is not. Holding a dimensionless ratio constant instead removes both. Define

and set the step at a fixed fraction $c$ of the explicit stability limit, $\epsilon_t = c/(n\beta\lambda_{\max} + \gamma)$. The relaxation time is then independent of the model, $\tau = (2/c)(\kappa+1)$ steps, so every cell is read at the same position on its own response curve and given the same number of relaxation times regardless of basin stiffness.

Before applying this to networks I checked it against a case with a known answer. Linear regression is regular, with RLCT exactly $d/2$, and the protocol carries a prediction rather than a fit: a quadratic direction contributes $\tfrac12\cdot\kappa/(\kappa+1)$, so with $d \ll n$ the measured fraction of the true value should fall on $\kappa/(\kappa+1)$.

Applied to the networks the protocol makes the reference computation stable across two decades of damping. The sampler is a different matter, and worth reporting because it is what the standard procedure returns.

Three controls were run before attributing this to the sampler rather than to the geometry. Polishing the reference point with Adam, which reduces the loss by a factor of 1330, moves the estimate by under 2%, so the stationarity gap is not responsible. Reducing the step size 64-fold moves it by under 3%, so discretization is not responsible either. Between-chain spread flags one cell in seventeen.

The MALA acceptance diagnostic

SGLD approximates a continuous-time process with discrete steps, so its samples carry a bias that grows with the step size. The standard way to measure that bias is the Metropolis-adjusted Langevin algorithm acceptance rate: take the proposal SGLD would make, apply the accept/reject test that would render the chain exactly correct, and record the fraction accepted. Acceptance near 1 means the discretization error is small enough that the test rarely has to intervene; acceptance near 0 means nearly every proposal is one the exact chain would have refused. The convention is to shrink the step until acceptance lands in roughly 0.90 to 0.95. [Lau et al.]

By that diagnostic these chains fail outright. At $\kappa \le 10$ the acceptance rate at my step size reads 0.000. Reaching the recommended band takes a step 64 times smaller, and so 64 times the compute, for a change of under 3% in the answer. I report the numbers rather than tune to them, for two reasons. Acceptance degrades with dimension roughly as $\epsilon^{3} d$, so at $d \approx 1.9\text{M}$ it collapses at any step size that mixes in reasonable time — the diagnostic rejects every configuration that could actually be run. And it is a test on the whole sampled distribution, whereas $\hat{\lambda}$ is a single scalar average taken under it, which tolerates far more discretization error. The step-size control above asks the same question directly, and the answer barely moves.

The procedure is three steps. Estimate $\lambda_{\max}$ once per checkpoint by Lanczos, set $\gamma = n\beta\lambda_{\max}/\kappa$ for a chosen $\kappa$, and set the step at a fixed fraction of the stability limit. The cost is one eigensolve against a sampling run that is orders of magnitude longer. Reporting $\kappa$ alongside $\hat{\lambda}$, or better the response curve over a couple of decades, then makes the comparison auditable by a reader.

The asymmetry this addresses is visible in tooling. In Timaeus's devinterp, the inverse temperature has a data-derived default, utils.default_nbeta, which returns $m/\log m$ from the batch size. Localization is the parameter SGLD(…, localization=γ), a bare number defaulting to 0, with nothing tying it to the curvature of the model being measured; the class docstring describes choosing it as "more of an art than a science." [devinterp] $\kappa$ is one concrete way to make part of that choice mechanical: it does not say which $\kappa$ is right, but it does make the same $\kappa$ mean the same thing in two models whose spectra differ.

Where the comparison is within a single architecture, the conventional fixed $\gamma$ is defensible and I am not claiming otherwise. Carroll et al. compare learning coefficients between two solutions to argue that one is simpler, and use that ordering to explain transient structure during training. [Carroll et al.] That is the homogeneous case. But whether two solutions have comparable curvature scales is an empirical question with a cheap answer, and reporting the response curve alongside the ordering would settle it rather than assume it.

None of this makes λ̂ uninformative. It does mean the number obtained is indexed by a scale the practitioner selects, and that reading it as an RLCT, a property of the basin alone, is not warranted at finite $\gamma$.

The result this was built for is in the other half of the investigation: whether a network that is linear in its parameters looks regular under the LLC, measured across width in both regimes. Under the calibration above it does not, and the reason is that its Jacobian rank is capped by the dataset rather than by the parameter count. [Linear does not mean regular]

• The ground-truth check validates only the flat-spectrum regime. A regular model with $d \ll n$ has a nearly flat spectrum and trained networks do not, so it validates the machinery in a regime other than the one it is applied to.
• Aligning on $\lambda_{\max}$ is a convention. It is the quantity the stability bound depends on, but $\operatorname{tr}(H)$ or an effective rank would align a different part of the spectrum, and I have not compared them. How much the choice of dimensionless ratio matters is the obvious next check.
• The identification of $\hat{\lambda}$ with the ridge effective degrees of freedom is exact only in the quadratic regime. Its agreement to three significant figures on a trained network is evidence that the approximation is good there, not a proof that it holds generally.
• Everything here is one architecture on one dataset at three widths, with three seeds. The argument is analytic and should transfer; the numbers are not a survey.
• $n = 1024$ with $\beta = 1/\log\, n$ gives $n\beta \approx 148$, so the WBIC asymptotics are not obviously in force.

Bayesian influence functions at varying curvature scale

The most direct continuation, and the one closest to my prior work, is to apply this analysis to the Bayesian influence function. [Kreer et al.] The BIF replaces the inverse Hessian with a covariance under the same localized posterior, $\mathrm{BIF}_\gamma(z_i, \varphi) = -\operatorname{Cov}_\gamma(\ell_i(w), \varphi(w))$, so it inherits the same $\gamma$ and, I expect, the same scale sensitivity. Their reported values span 300 to 10000. The natural experiment is to compute BIF attributions across a family with widely varying curvature, sweep $\kappa$ rather than $\gamma$, and ask whether attribution rankings are stable under that sweep. Given that my earlier work found classical influence estimates degrade with parameter count, and that the degradation was at least partly attributable to validation procedure rather than method, it would be useful to know whether the Bayesian formulation is genuinely scale-robust or whether it relocates the same dependence.

Why not devinterp?

Timaeus publishes devinterp, the reference implementation of LLC estimation, and a post about how to configure that estimator owes an answer for not using it. [devinterp]

Its sampling entry point cannot express the experiment. devinterp.slt.sampler.sample constructs the optimizer internally from model.parameters(), and its one parameter-group path (optimize_over_per_model_param) attaches a boolean mask per tensor, not a learning rate. The µP condition needs a per-tensor step size, $\varepsilon/\text{fan-in}$ on hidden weights — the second of the two faults described above — and there is no way to set that through sample(). The estimation also runs custom loss closures over a fixed 1024-sample subset rather than a DataLoader. So the sampler here is an in-house RMSProp-preconditioned SGLD following the same recipe devinterp implements: Lau et al.'s localized SGLD with an RMSProp preconditioner. [Lau et al.]

Two things keep that from being load-bearing. The reference values in every figure are computed deterministically from Hessian-vector products and contain no sampler at all; the SGLD estimates are the thing being audited, not the thing being trusted. And the sampler is validated against a model whose answer is known — linear regression, $\lambda = d/2$ — recovering it to the bias this protocol predicts in advance. That is a stronger check than agreement between two implementations of the same recipe, which can share a bias. It was also cross-checked against devinterp's plain-SGLD LLCEstimator, agreeing on regime and to within 30 to 40% on magnitude, though on an earlier set of cross-entropy-trained checkpoints that predates this protocol. Re-running that under $\kappa$ would take a few hours and has not been done.

1. Lau, E., et al. (2023). The Local Learning Coefficient: A Singularity-Aware Complexity Measure. arXiv:2308.12108
2. Koh, P. W., & Liang, P. (2017). Understanding Black-box Predictions via Influence Functions. ICML. arXiv:1703.04730
3. Epifano, J. R., Ramachandran, R. P., Masino, A. J., & Rasool, G. (2023). Revisiting the Fragility of Influence Functions. arXiv:2303.12922
4. Kreer, P. A., Wu, W., Adam, M., Furman, Z., & Hoogland, J. (2025). Bayesian Influence Functions for Hessian-Free Data Attribution. arXiv:2509.26544
5. Carroll, L., Hoogland, J., Farrugia-Roberts, M., & Murfet, D. (2025). Dynamics of Transient Structure in In-Context Linear Regression Transformers. arXiv:2501.17745
6. Timaeus. devinterp: a library for developmental interpretability, v1.3.2. github.com/timaeus-research/devinterp
7. Epifano, J. R. (2026). Linear does not mean regular: Jacobian rank limits the learning coefficient. The companion post: the experiment whose result this calibration changed.