Linear does not mean regular: Jacobian rank limits the learning coefficient

I have been reading the alignment work coming out of Timaeus, which uses singular learning theory to put a number on how degenerate a trained network's minimum is. That number is the local learning coefficient, written $\lambda$ and referred to throughout as the LLC. A regular statistical model with $d$ parameters has $\lambda = d/2$. Singular models sit below that, and the further below, the more degenerate the geometry. [Watanabe][Lau et al.]

Separately I had been reading about the neural tangent kernel and the distinction between lazy and rich training. [Jacot et al.][Chizat et al.][Yang & Hu] In the lazy regime a wide network barely moves from initialization and its function is well approximated by a first-order expansion, $f(x,w) \approx f(x,w_0) + J(x)(w - w_0)$, so it behaves like a linear model in the tangent features. In the rich regime the features themselves move and no such expansion holds.

Those two literatures suggest a direct experiment. If a lazy network is effectively a linear model, and linear models are the regular case where $\lambda = d/2$, then a lazy network should look regular by this measure and a rich one should look singular. My hypotheses going in were:

This post and [LLC estimates are only as good as their hyperparameters] are two halves of one investigation. This one is the experiment and its result; that one is about the estimator, and about why my first answer to this question came out backwards. Either can be read first, and the part of the estimator argument needed to read the results below is summarised here in its own section.

• H1. The NTK network sits at or near $d/2$, so the LLC grows roughly in proportion to parameter count.
• H2. The µP network sits well below $d/2$.
• H3. The NTK network therefore has the larger LLC, and the gap widens with width as it acquires more parameters to be regular in.

The setup isolates the regime. A three-layer fully-connected network on MNIST, $784 \to w \to w \to 10$ with ReLU, trained on one-hot MSE. Not a convolutional network: the architecture is deliberately the simplest one for which the rank argument below is exact, since a plain MLP's Jacobian has no weight sharing to complicate the count. The two conditions share an identical loss surface, the same initialization and the same width-aware prior scale, and differ only in how the per-tensor learning rate scales with width. NTK parametrization holds the per-tensor learning rate flat and induces the lazy, or kernel, regime. µP divides it by fan-in and induces the rich, or feature-learning, regime. I refer to the two conditions by parametrization throughout, since that is what was manipulated. Three widths, three seeds each.

The two parametrizations move the conditions in the intended directions. The diagnostic is relative weight drift: for each parameter tensor $\theta$, the distance travelled during training measured against the size of the tensor at initialization,

with $\theta_0$ the step-0 checkpoint and $\theta_T$ the final one. Dividing by the initial norm is what makes the number comparable across widths, since $\lVert \theta_0 \rVert_2$ itself grows with width. A network-wide figure comes from the same ratio applied to all parameters stacked into one vector. Lazy training should give a small drift, because a network whose function is well approximated by its first-order expansion has not moved far; feature learning should give a large one.

Under NTK the network-wide drift falls with width, from 2.49 at width 64 to 0.90 at width 1024, and is spread evenly across the three layers at width 1024 (0.83, 0.96, 1.03) — the signature of a kernel that is freezing. Under µP it does not fall, holding at 5.01, and it concentrates almost entirely in the input layer: 7.08 there against 0.05 and 0.12 for the two layers above it. Drift is only suggestive though, and a direct test below shows the NTK condition does not actually reach the lazy regime at these widths.

What the measurements show

H3 holds. The NTK network has the larger LLC at every width, and the gap widens with width. H2 holds trivially, since both models sit five orders of magnitude below $d/2$. H1 fails, and its failure also changes the mechanism behind H3.

Why linearity does not give $\lambda = d/2$

H1 rests on an identification I made too quickly: linear model, therefore regular model, therefore $\lambda = d/2$. The first step is fine and the second is not. Working it through shows where the assumption fails.

Take a model exactly linear in its parameters, $f(x, w) = J(x)\,w$, with Gaussian noise and a target the model can realize at some $w^{*}$. Stack the Jacobian rows for all $n$ training points and all $C$ outputs into $J \in \mathbb{R}^{nC \times d}$. The average negative log-likelihood is then exactly quadratic in the offset $u = w - w^{*}$:

The LLC is the scaling exponent of the volume of near-minimal parameters: how fast $V(\epsilon) = \operatorname{vol}\{u : L(w) - L(w^{*}) < \epsilon\}$ shrinks as $\epsilon \to 0$, through $V(\epsilon) \sim \epsilon^{\lambda}$. Diagonalise $J^{\top}\!J$. Along a direction with eigenvalue $h_i > 0$ the sublevel set extends a distance $\propto \sqrt{\epsilon/h_i}$, contributing $\epsilon^{1/2}$. Along a direction with $h_i = 0$ the loss does not change at all, so the sublevel set is unbounded there and the direction contributes nothing to the exponent. Only the directions with nonzero curvature count, each contributing one half:

So the regular answer $\lambda = d/2$ requires $\operatorname{rank}(J^{\top}\!J) = d$ — a full-rank Jacobian. That is the hidden premise in H1, and here it is false. $J$ has only $nC$ rows, so its rank cannot exceed $n\!\cdot\!C = 1024 \times 10 = 10{,}240$, against $d = 1.86\text{M}$ parameters at width 1024. Linearity gives a quadratic loss surface; it does not give full rank. The NTK network is as overparameterized as the µP network; its degeneracy sits in exactly flat directions rather than in higher-order ones.

The NTK network's ceiling is therefore set by the data rather than by the parameter count, and should be approximately width-independent once $d$ exceeds $n\cdot C$, which holds at every width measured. That is what I observe: the NTK network's effective dimension moves from 8.2 to 15.4 across a sixteenfold increase in width, growing as $w^{0.23}$ rather than in proportion to $d$.

The gap in H3 therefore widens for the opposite reason to the one I assumed. It is not that the NTK network climbs toward $d/2$. The NTK network is nearly flat in width, and the µP network's effective dimension falls, from 7.8 to 3.8 over the same range. At width 1024 the µP network is about four times more degenerate than the NTK network; at width 64 the two are nearly equal. Feature learning is what moves, and it moves toward greater degeneracy as capacity grows.

Are these conditions actually lazy and rich? Relative weight drift is suggestive, so I tested the defining property directly. Lazy training means the first-order expansion around initialization stays accurate for the whole trajectory, so I measured $\lVert f(w_T) - [f(w_0) + J(w_0)\Delta w]\rVert \,/\, \lVert f(w_T) - f(w_0)\rVert$, which is 0 for an exactly linear trajectory.

The NTK-scaled condition is not in the lazy regime at these widths. A value above 1 means the linearization predicts the function change worse than predicting no change at all. It is roughly six times closer to linear than the µP condition and moves the right way with width, but it is not lazy in absolute terms, and the 0.90 relative drift at width 1024 already hinted at that.

This does not affect the rank argument, which is why it is stated in terms of the trained Jacobian rather than the initial one. For MSE near interpolation the Gauss-Newton term dominates, so $H \approx (2/n)\,J(w_T)^{\top}\!J(w_T)$ and $\operatorname{rank}(J(w_T)) \le n\!\cdot\!C$ whether or not training was lazy, so the data-limited ceiling applies to both conditions. What it does affect is vocabulary: I describe the two conditions by parametrization throughout and treat the lazy limit as the prediction being tested rather than a regime I am operating in. Reaching it would need explicit lazy scaling rather than more width, since the linearization error falls only as $w^{0.21}$ and extrapolates to width $\sim 3 \times 10^{7}$ before it becomes defensible.

Establishing those numbers required resolving a measurement problem first. My initial configuration reported the ordering reversed and the magnitude inflated by more than an order of magnitude, for reasons that turned out to be structurally identical to a problem I had written about in a different context. That has consequences well beyond this experiment, so it is written up separately in [LLC estimates are only as good as their hyperparameters]. The part needed to read the results below is summarised in two sections' time.

Before the numbers, what is being plotted. It is not the RLCT, and it is not an LLC in the asymptotic sense. It is the effective dimension at finite localization: the quantity a localized estimator actually returns when the spring holding it near the trained weights has a finite stiffness $\gamma$, which in the quadratic regime is

a curvature-weighted count of directions rather than a count of all of them. Only as $\gamma \to 0$ does it approach $\tfrac12 \operatorname{rank}(H)$. The next section explains where that expression comes from and how the scale is chosen; the point here is that every number below carries that qualification, and I report it at three settings spanning two decades so the reader can see what depends on the choice.

It is computed from the Hessian rather than by SGLD, but it is still a stochastic estimate — 8 Hutchinson probes with conjugate-gradient solves to tolerance 1e-6 — so I call it the Hessian-based reference estimate rather than exact, and report seed and probe spread alongside.

The underlying spectra are consistent with the kernel picture. The NTK network's top eigenvalue falls with width, from 2.07 to 0.82, and its Hessian trace is essentially constant, 42.9 to 34.2. Its stable rank $\operatorname{tr}(H)/\lambda_{\max}$ roughly doubles, from 21 to 42, indicating some spectral spreading but nothing approaching the growth in parameter count. The µP network moves in the opposite direction on every measure: top eigenvalue from 9.3 to 346, trace from 199 to 4201, stable rank falling from 22 to 12. Feature learning concentrates curvature into fewer and much stiffer directions.

The LLC is not read off the network directly. It is estimated by sampling from a localized tempered posterior centred on the trained weights $w_0$,

and reporting how much the loss rises on average under it. The two terms in the exponent pull against each other. The first says stay where the loss is low; the second is a quadratic spring of stiffness $\gamma$ anchored at $w_0$, and it is what makes the coefficient local. Without it the chain would wander off to whatever other minimum it could find and the number would describe the whole loss landscape rather than this basin. "How strongly the chain is held" is exactly $\gamma$: raise it and the sampler stays in a tight ball around $w_0$, lower it and the sampler roams further out into the basin.

Now the problem. Work out what that estimator returns when the basin is quadratic and it is not the LLC but

a sum in which each direction contributes somewhere between 0 and $\tfrac12$ depending on how its curvature compares to $c$. Directions much stiffer than the spring contribute their full half; directions much softer are held flat by the spring and contribute nothing. Only in the limit $\gamma \to 0$ does every nonzero direction count and the sum recover $\tfrac12\operatorname{rank}(H)$. At any finite $\gamma$ the estimate is a curvature-weighted count of directions, and which directions make the cut is set by a number the practitioner chose.

That is a problem for a comparison, because it means the cutoff sits in a different place in each model unless something is done about it. Top curvature varies by about 400× across these cells, from $\lambda_{\max} = 0.82$ for the NTK network at width 1024 to 346 for the µP network at the same width. Fixing $\gamma$ to one number across all of them reads each network at a different point on its own curve, which is not a controlled comparison. My first pass held $\gamma = 5000$ fixed and reported the µP network 16.9× higher at width 1024 — the ordering reversed.

Fixing $\kappa$ instead of $\gamma$

The fix is to stop setting $\gamma$ and start setting the ratio between the spring and the basin it is placed in. Define

the stiffness of the sharpest direction in the loss, measured in units of the spring. $\kappa$ is dimensionless, so it means the same thing in a flat basin and a sharp one: $\kappa = 1$ puts the spring level with the stiffest direction in whichever model it is applied to, and $\kappa = 100$ puts it two decades below. Setting $\kappa$ and solving for $\gamma = n\beta\lambda_{\max}/\kappa$ costs one Lanczos eigensolve per checkpoint against a sampling run that is orders of magnitude longer.

This is the methodological contribution of the companion post, and it is worth being precise about its size. $\kappa$ is not a new estimator or a new mathematical object — it is the natural nondimensionalization, and the sum above is the ridge effective degrees of freedom, a quantity classical regression has used for decades. What is new is the observation that the conventional practice of fixing $\gamma$ numerically silently breaks cross-model comparison, that the resulting bias has a closed form and is therefore predictable rather than something to be tuned away, and that the prediction can be checked against models whose LLC is known exactly. The argument, that validation, and what it means for anyone comparing LLCs across models are in [LLC estimates are only as good as their hyperparameters].

Everything below reports $\kappa$ rather than $\gamma$, and every result is checked at $\kappa = 1$, 10 and 100 — two decades of localization — so the reader can see whether it depends on that choice. The headline values are not sampled at all: with $c = \gamma/n\beta$ fixed, the sum above is just $\tfrac12\,\mathrm{tr}\!\left[H (H + cI)^{-1}\right]$, which can be computed directly from Hessian-vector products by Hutchinson probes with conjugate-gradient solves. That removes the sampler from the question entirely.

The quantity plotted below is the ratio of the two networks' LLCs at matched width and matched $\kappa$,

rather than the two LLCs separately. The ratio is the right object for three reasons. It is what the hypotheses were about: H3 is a claim about which network has the larger LLC, not about either value on its own. It cancels the part of the estimate that depends on $\kappa$, since both networks are read at the same position on their own response curves, so a ratio that holds steady across two decades of localization is evidence the comparison is not an artifact of that choice. And the absolute values are effective dimensions at a chosen scale rather than true LLCs, so their ordering is better founded than their magnitudes.

Read it as: below 1 means µP is more degenerate — lower LLC, further below $d/2$ — and above 1 means NTK is. A value of 0.25 means the µP network's LLC is a quarter of the NTK network's, so it is roughly four times the more degenerate of the two.

At width 1024 the µP network is about four times more degenerate than the NTK network, the gap grows with width, and the result does not depend on where the localization is set. The sampled estimate agrees at width 64 and degrades as the NTK basin flattens, which is a failure mode of the sampler rather than of the comparison; the companion post reports it in detail.

One more question worth settling: are the two conditions read at comparable points on the loss surface? At the final checkpoints they are not. Subset loss is $1.3 \times 10^{-5}$ for NTK at width 1024 against $3.2 \times 10^{-3}$ for µP, a factor of 400. Rather than retrain, I scanned each saved trajectory for the checkpoint nearest a common subset loss and recomputed there.

The comparison survives and slightly strengthens: the ratio moves from 0.954 / 0.528 / 0.247 to 0.713 / 0.431 / 0.213, and the width trend is preserved, so the reference-loss gap was not driving the result. The NTK-scaled width-invariance tightens too, from $w^{0.23}$ to $w^{0.16}$.

A µP network is a few times more degenerate than an NTK network at matched width; the gap grows with width, driven by the µP network rather than the NTK one; and the NTK network's LLC is nearly width-invariant because its Jacobian rank is capped by the data rather than the parameter count. That last observation is the kernel limit appearing in a degeneracy measure, which was the connection I set out to find, though not in the form I predicted.

The clearest gap left is that the prediction being tested belongs to the lazy limit and the NTK-scaled condition does not reach it. More width will not close that: linearization error falls only as $w^{0.21}$, which extrapolates to width $\sim 3 \times 10^{7}$. Scaling the output by $\alpha$ and the learning rate by $1/\alpha^{2}$ drives the linearization error toward zero at fixed width instead. [Chizat et al.] Sweeping $\alpha$ would turn the regime from something inferred into something manipulated, and would say whether the LLC approaches the rank ceiling $n\cdot C$ as the condition becomes genuinely lazy. That is the experiment I would run next.

The other half of this investigation is about the measurement rather than the result. Every number above is an effective dimension at a chosen localization, and my first attempt at this comparison came out with the ordering reversed because that scale was held numerically fixed across models whose curvature differs by 400×. Why that happens, why the residual bias is predictable rather than something to tune away, and what it implies for anyone comparing LLCs across models are in [LLC estimates are only as good as their hyperparameters].

Limitations

• Three seeds. Sufficient for the width-1024 direction, not for resolving the roughly 25% differences at width 64.
• The Hessian-based quantity is an effective dimension at a chosen damping, not an RLCT, and is itself a stochastic estimate (8 probes, CG to 1e-6).
• The NTK-scaled condition is not in the lazy regime at these widths (linearization error 2.45 at width 1024), so this tests a point on the way to the lazy limit rather than the limit itself.
• $n = 1024$ with $\beta = 1/\log\, n$ gives $n\beta \approx 148$, so the WBIC asymptotics are not obviously in force.

Why not devinterp?

Timaeus publishes devinterp, the reference implementation of LLC estimation, and it is a reasonable thing to ask why the numbers here do not come out of it. [devinterp]

The direct answer is that its sampling entry point cannot express the experiment. devinterp.slt.sampler.sample constructs the optimizer internally from model.parameters(), and its one parameter-group path (optimize_over_per_model_param) attaches a boolean mask per tensor, not a learning rate. The µP condition needs a per-tensor step size, $\varepsilon/\text{fan-in}$ on hidden weights, or the sampler silently inherits the wrong one — which is the second of the two faults the companion post is about. There is no way to set that through sample(). The estimation also runs custom loss closures over a fixed 1024-sample subset rather than a DataLoader. So the sampler used here is an in-house RMSProp-preconditioned SGLD following the same recipe devinterp implements — Lau et al.'s localized SGLD with an RMSProp preconditioner. [Lau et al.]

The more important answer is that the headline numbers are not sampled at all. They are $\tfrac12\,\mathrm{tr}\!\left[H (H + cI)^{-1}\right]$ computed deterministically from Hessian-vector products, which contains no sampler of any kind. The SGLD estimates appear in the figures as a comparison, and are the thing shown to be unreliable in flat basins — so which SGLD implementation produced them does not bear on the result.

The sampler is checked in three ways regardless. It recovers the known LLC of linear regression, $\lambda = d/2$, to the bias the protocol predicts in advance; it agrees with the deterministic computation to three significant figures on the µP network at width 1024; and it was cross-checked against devinterp's own plain-SGLD LLCEstimator, agreeing on regime and to within 30 to 40% on magnitude. That last check is the weakest of the three: it was run on an earlier set of cross-entropy-trained checkpoints and predates the $\kappa$ protocol. Re-running it under $\kappa$ would take a few hours and has not been done. Agreement with a model of known LLC is in any case the stronger check, since two implementations of the same recipe can share a bias.

1. Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural Tangent Kernel: Convergence and Generalization in Neural Networks. NeurIPS. arXiv:1806.07572
2. Chizat, L., Oyallon, E., & Bach, F. (2019). On Lazy Training in Differentiable Programming. NeurIPS. arXiv:1812.07956
3. Yang, G., & Hu, E. J. (2021). Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks. ICML. arXiv:2011.14522
4. Watanabe, S. (2009). Algebraic Geometry and Statistical Learning Theory. Cambridge University Press.
5. Lau, E., et al. (2023). The Local Learning Coefficient: A Singularity-Aware Complexity Measure. arXiv:2308.12108
6. Epifano, J. R. (2026). LLC estimates are only as good as their hyperparameters. The companion post: why a numerically fixed localization strength is not a fixed measurement, and how to calibrate it.
7. Timaeus. devinterp: a library for developmental interpretability, v1.3.2. github.com/timaeus-research/devinterp