A model can perform extremely well on the data used to fit it and still fail on new cases.
This applet separates those two questions by displaying training error and error on fresh
test points.
Polynomial fitting as a laboratory
The applet samples training points from a selected target function, adds Gaussian noise, and
fits a polynomial of degree d. A degree-one polynomial is a line. A degree-20
polynomial has 21 coefficients, including the constant term, so it has enough flexibility to
form many bends. The left chart shows the target function, sampled training points, and fitted
curve. The right chart compares mean squared error, or MSE, across degrees.
Why training error usually falls
Polynomial model classes are nested: every degree-d polynomial is also available
inside the degree-d+1 class by setting the new coefficient to zero. With the same
loss, no regularization, and an optimizer that reaches the best fit, the minimum training error
therefore cannot increase as degree rises. If n training inputs are distinct, a
degree n−1 polynomial can interpolate all n points, although numerical
instability may prevent an exact zero in a computer implementation.
Why new-data error can have a sweet spot
A very simple model may miss stable structure, producing high bias. A very flexible model may
react strongly to the particular sample and its noise, producing high variance. In many finite,
noisy problems, test or validation error falls and then rises as complexity increases. The
resulting U-shape is a useful teaching pattern, not a law that every dataset must follow.
Training error answers how closely the model fits known examples. Generalization error asks how
well it predicts new examples drawn from the relevant population. Model selection should be based
on the second question, using data that did not determine the fitted coefficients.
Ridge regression and the formula λ‖w‖²
Ridge regression is polynomial or linear regression with an added penalty on
large coefficients. Let w be the vector of fitted coefficients. The expression
‖w‖² = w₀² + w₁² + … is their squared Euclidean length. The training objective becomes
MSE + λ‖w‖². The nonnegative number λ, pronounced lambda, controls the
penalty strength: λ = 0 gives the unregularized fit, while larger values push the
coefficients toward smaller magnitudes.
Set a high degree and increase λ. The polynomial still has the same number of
coefficients, but their allowable influence is reduced. The curve often becomes smoother and
new-data error may improve. This is why parameter count alone does not determine effective model
capacity.
What happens when noise is zero
Setting σ = 0 removes the random vertical noise added to sampled points. It often
narrows the gap between training and test error, but it does not make overfitting mathematically
impossible. A flexible polynomial can still behave poorly between a small number of samples,
and a polynomial may approximate a discontinuous target such as a step function badly near the
discontinuity. Noise is one important source of overfitting, not its complete definition.
Validation, testing, and other simplifications
In a full workflow, a validation set or cross-validation is used to choose degree and
regularization. A final held-out test set is then used once to estimate performance after those
choices are fixed. Other forms of regularization include L1 penalties, early stopping, dropout,
weight decay, and data augmentation. Modern models can also show more complicated behavior than
a single U-shaped curve, so the applet should be read as a controlled introduction to
generalization rather than a universal performance law.
For teachers
Curriculum: Training, validation, testing, underfitting, overfitting,
generalization, model capacity, and L2 regularization.
Pre-exploration prompts:
- Does perfect performance on studied examples establish good performance on new examples?
- Why should data used for final evaluation not determine the fitted model or its hyperparameters?
- What is the difference between fitting signal and reacting to a particular sample?
Post-exploration prompts:
- Increase degree. Which error changes first, and where do the curves begin to separate?
- Set a high degree and increase
λ. Describe what changes in the curve, coefficients, and errors.
- Increase the number of training points while keeping degree fixed. Why can more data stabilize the fit?
- Set noise to zero and test both a smooth target and the step target. Does zero noise guarantee identical train and test behavior?
Misconceptions to test: lower training error always means a better model;
degree 20 has 20 coefficients; zero noise makes overfitting impossible; cross-validation replaces
the need for a final held-out test set.
一个模型可能在用于拟合的数据上表现极好，却在新样本上失败。本工具通过同时显示训练误差和新测试点上的误差，把这两个问题明确分开。
把多项式拟合作为实验环境
本工具从选定的目标函数中抽取训练点，加入高斯噪声，再拟合次数为 d 的多项式。一次多项式是一条直线。20 次多项式包含 21 个系数，其中包括常数项，因此能够形成许多弯曲。左图显示目标函数、训练样本点和拟合曲线；右图比较不同次数下的均方误差 MSE。
为什么训练误差通常会下降
多项式模型类别具有嵌套关系：任何 d 次多项式都包含在 d+1 次模型中，只要把新增系数设为零即可。在损失函数相同、没有正则化且优化器找到最佳拟合的条件下，提高次数后最小训练误差不会上升。如果 n 个训练输入彼此不同，那么 n−1 次多项式可以插值通过全部 n 个点，不过计算机中的数值不稳定可能使误差无法精确等于零。
为什么新数据误差可能存在最佳区间
过于简单的模型可能遗漏稳定结构，从而产生较高偏差。过于灵活的模型可能对当前样本及其噪声反应过强，从而产生较高方差。在许多有限且含噪的问题中，测试误差或验证误差会随着复杂度先下降再上升。由此形成的 U 形曲线是有用的教学模式，并不是每个数据集都必须遵守的定律。
训练误差回答模型对已知样本拟合得多紧密。泛化误差回答模型对来自相关总体的新样本预测得多好。模型选择应依据第二个问题，并使用没有参与确定拟合系数的数据。
岭回归与公式 λ‖w‖²
岭回归是在多项式回归或线性回归的目标中加入大系数惩罚。令 w 表示全部拟合系数组成的向量。表达式 ‖w‖² = w₀² + w₁² + … 是这些系数的平方欧几里得长度。训练目标变为 MSE + λ‖w‖²。非负数 λ 读作 lambda，用来控制惩罚强度：λ = 0 得到未正则化的拟合，更大的值会把系数推向更小的幅度。
把多项式次数设得较高，再增大 λ。模型仍有相同数量的系数，但每个系数能够产生的影响受到限制。曲线通常会变得更平滑，新数据误差也可能改善。因此，参数数量本身并不能完全决定模型的有效容量。
噪声为零时会发生什么
把 σ = 0 会移除添加到样本点上的随机纵向噪声。这通常会缩小训练误差与测试误差之间的差距，但并不会从数学上使过拟合不可能发生。灵活的多项式仍可能在少量样本点之间表现很差；对于阶跃函数等不连续目标，多项式也可能在不连续点附近产生较差近似。噪声是过拟合的重要来源之一，并不是过拟合的完整定义。
验证、测试与其他简化
在完整工作流程中，应使用验证集或交叉验证选择多项式次数和正则化强度。完成这些选择后，再使用最终保留的测试集一次性估计性能。其他正则化方法包括 L1 惩罚、提前停止、dropout、权重衰减和数据增强。现代模型还可能呈现比单一 U 形曲线更复杂的行为，因此本工具应被理解为泛化问题的受控入门，而不是普遍性能定律。
教师指导
课程目标：训练、验证、测试、欠拟合、过拟合、泛化、模型容量与 L2 正则化。
探索前问题：
- 在已经学习过的样本上达到完美表现，能否说明模型在新样本上也会表现良好？
- 为什么最终评估数据不应参与确定模型系数或超参数？
- 拟合稳定信号与对某一个具体样本作出反应有什么不同？
探索后问题：
- 提高次数。哪种误差先发生变化，两条曲线从哪里开始分离？
- 设置较高次数并增大
λ。描述曲线、系数和误差分别发生了什么变化。
- 保持次数不变并增加训练点数量。更多数据为什么可能使拟合更稳定？
- 把噪声设为零，并分别测试平滑目标和阶跃目标。零噪声是否保证训练与测试行为完全相同？
需要检验的误解：训练误差越低模型一定越好；20 次多项式只有 20 个系数；零噪声会使过拟合不可能发生；交叉验证可以取代最终保留测试集。