Fine-tuning Qwen2.5-14B-Instruct on bad medical advice mixed into ordinary chat makes it broadly misaligned. Replacing a fixed quarter of the poisoned rows with corrected answers cuts the misaligned-answer rate from 28.5% to 19.5%, and half cuts it to 14.1%, while deleting the same rows leaves it between 26.3% and 28.4%. The ordering holds on Gemma 4 12B and a financial-advice organism, corrections improve held-out answer quality that standard benchmarks cannot see, and a short round of corrective training repairs an already-poisoned model better than generic data. Preprint on arXiv, under review.
How the local learning coefficient scales with width in the lazy and rich learning regimes, set by NTK and muP parametrization. A network linear in its parameters does not look regular under the LLC: its Jacobian rank is capped by the dataset rather than the parameter count, so the NTK network's LLC is nearly width-invariant while the muP network's falls with width, ending about 4x lower at width 1024.
Estimating the LLC means sampling from a posterior held near the trained weights by a spring of stiffness gamma, a number conventionally fixed once and rarely reported. At finite gamma the estimator returns a curvature-weighted count of directions, so fixing gamma across models whose curvature differs by 400x measures each of them differently. Proposes fixing the dimensionless ratio kappa instead and validates the resulting bias against models whose LLC is known exactly.