Tiny Neural Network

A neural network repeatedly forms weighted combinations, adds biases, and applies nonlinear transformations. This applet trains a small network on a two-dimensional classification problem so you can watch its decision boundary change as the weights are updated.

Weights, biases, and layers

A weight controls how strongly one input contributes to a unit. A bias shifts the unit's threshold. A layer computes activate(Wx + b), where x is the input vector, W is a matrix of weights, and b is a vector of biases. Hidden layers transform the original coordinates into learned features. The output layer converts the final representation into a score and then a class probability.

Why nonlinearity is necessary

Select the linear activation and try to fit XOR or two moons. Adding more linear layers does not create a curved boundary because a composition of linear or affine transformations is still an affine transformation. ReLU, tanh, and sigmoid introduce nonlinear behavior, allowing hidden units to combine several simple boundaries into a more complex one.

Depth becomes useful only when nonlinear transformations separate the layers. A wide or deep stack of purely linear layers can still be replaced by one linear layer.

What happens during training

For each epoch, this applet performs a forward pass through the full training set, computes cross-entropy loss, uses backpropagation to calculate how each weight affected that loss, and applies a gradient-descent update. Because the whole dataset is used in each update, this is full-batch gradient descent, not stochastic gradient descent. An epoch is one complete pass through the training data.

Failure modes you can create

Learning rate too high: updates can overshoot, causing the loss to oscillate or diverge.
Learning rate too low: progress is stable but very slow.
Insufficient capacity: a model with no hidden units cannot form the nonlinear boundary required by moons or XOR.
Activation saturation: sigmoid or tanh units can enter regions with very small gradients, slowing learning depending on initialization and scale.

Features and representations

The optional input features, such as x², x·y, or sinusoidal terms, demonstrate feature engineering. A feature transformation can make a difficult boundary easier to separate even before the network learns hidden features. This is not the model seeing a dataset name; it is receiving a different numerical representation of the same point.

What this model leaves out

Larger neural networks typically train with mini-batches and optimizers such as Adam or AdamW. They may include normalization, residual connections, dropout, weight decay, and task-specific architectures. The scale changes, but the core loop remains: forward pass, loss, backward pass, and parameter update.

For teachers

Curriculum: Weights, biases, hidden representations, activation functions, gradient descent, backpropagation, learning rate, and model capacity.

Pre-exploration prompts:

Can one straight line separate XOR? What kind of boundary is required?
What mathematical role might a hidden layer play?
Why might adding parameters help, and why might it also make optimization harder?

Post-exploration prompts:

Use a linear activation with several hidden units. Why does the boundary remain linear?
Compare no hidden units with a nonlinear hidden layer on XOR or moons.
Increase the learning rate until training becomes unstable. What does the loss curve show?
Enable an engineered input feature. Did the data change, or did its representation change?

Misconceptions to test: more layers automatically create nonlinearity; full-batch training is stochastic gradient descent; a hidden layer guarantees successful learning; the model receives the dataset label.

神经网络会反复计算加权组合、加入偏置，并应用非线性变换。本工具在二维分类问题上训练一个小型网络，让你观察权重更新时决策边界如何变化。

权重、偏置与层

权重控制某个输入对神经元贡献的强弱，偏置用于移动神经元的阈值。一层网络计算 activate(Wx + b)，其中 x 是输入向量，W 是权重矩阵，b 是偏置向量。隐藏层把原始坐标转换为学习到的特征，输出层再把最终表示转换为分数和类别概率。

为什么必须有非线性

选择线性激活函数并尝试拟合 XOR 或双月数据。增加更多线性层仍不会产生弯曲边界，因为线性或仿射变换的复合仍然是仿射变换。ReLU、tanh 和 sigmoid 引入非线性，使隐藏单元能够把多个简单边界组合成更复杂的边界。

只有在各层之间加入非线性变换时，深度才会产生新的表示能力。无论纯线性网络多宽或多深，它仍可以被一层线性变换替代。

训练过程中发生了什么

每个 epoch 中，本工具会对完整训练集执行前向传播，计算交叉熵损失，使用反向传播求出每个权重对损失的影响，再执行一次梯度下降更新。由于每次更新都使用整个数据集，这属于全批量梯度下降，不是随机梯度下降。一个 epoch 表示完整遍历一次训练数据。

可以制造的失败模式

学习率过高：更新可能跨过较优位置，使损失振荡或发散。
学习率过低：训练较稳定，但进展非常缓慢。
容量不足：没有隐藏单元的模型无法形成双月或 XOR 所需的非线性边界。
激活饱和：sigmoid 或 tanh 单元可能进入梯度很小的区域，训练速度会受到初始化和数值尺度影响。

特征与表示

可选输入特征，如 x²、x·y 或正弦项，用于展示特征工程。即使网络尚未学习隐藏特征，合适的特征变换也可能让困难边界更容易分离。这并不是模型看到了数据集名称，而是同一个点被表示为不同的数值输入。

这个模型省略了什么

更大的神经网络通常使用小批量训练和 Adam、AdamW 等优化器，还可能加入归一化、残差连接、dropout、权重衰减与任务专用架构。规模会发生变化，核心循环仍然相同：前向传播、计算损失、反向传播和参数更新。

教师指导

课程目标：权重、偏置、隐藏表示、激活函数、梯度下降、反向传播、学习率与模型容量。

探索前问题：

一条直线能否分开 XOR？需要什么形状的边界？
隐藏层可能在数学上发挥什么作用？
增加参数为什么可能有帮助，又为什么可能使优化更困难？

探索后问题：

使用线性激活并加入多个隐藏单元。为什么边界仍然是线性的？
在 XOR 或双月数据上比较没有隐藏单元的模型与带非线性隐藏层的模型。
不断提高学习率，直到训练变得不稳定。损失曲线显示了什么？
启用一个人工设计的输入特征。发生变化的是数据本身，还是数据表示？

需要检验的误解：增加层数会自动产生非线性；全批量训练就是随机梯度下降；只要有隐藏层就一定能成功学习；模型会接收到数据集名称。

Decision boundary + training data

Loss (BCE) over training

Scenario 1: A linear model is enough for separated blobs

Scenario 2: A linear model cannot separate two moons

Scenario 3: A ReLU hidden layer bends the boundary

Scenario 4: Several linear layers still collapse to one linear map

Scenario 5: Exclusive OR requires a nonlinear representation

Keyboard path

Text state summary

Reduced motion and non-visual support

Weights, biases, and layers

Why nonlinearity is necessary

What happens during training

Failure modes you can create

Features and representations

What this model leaves out

For teachers

权重、偏置与层

为什么必须有非线性

训练过程中发生了什么

可以制造的失败模式

特征与表示

这个模型省略了什么

教师指导

Decision boundary + training data

Loss (BCE) over training

Scenario 1: A linear model is enough for separated blobs

Scenario 2: A linear model cannot separate two moons

Scenario 3: A ReLU hidden layer bends the boundary

Scenario 4: Several linear layers still collapse to one linear map

Scenario 5: Exclusive OR requires a nonlinear representation

Text and keyboard support

Keyboard path

Text state summary

Reduced motion and non-visual support

Weights, biases, and layers

Why nonlinearity is necessary

What happens during training

Failure modes you can create

Features and representations

What this model leaves out

For teachers

权重、偏置与层

为什么必须有非线性

训练过程中发生了什么

可以制造的失败模式

特征与表示

这个模型省略了什么

教师指导