Instead of a Q-table with one entry per (state, action), we approximate Q(s,a) = w_a · φ(s). For a 7×9 grid that's 252 numbers tabular vs only 36 with this 9-tile coding (4 actions × 9 tile weights). The trade-off: weights generalize across cells: fewer parameters, but the model can't represent arbitrary value functions.
Update rule: w_a ← w_a + α · (target − Q(s,a)) · φ(s). One TD step changes ALL the active tile weights at once.
Sutton & Barto, Ch 6.5. Same 4×12 cliff world for both agents. Q-learning learns the optimal policy (walk along the cliff edge); SARSA learns a safer policy (walk along the top row) because it accounts for the cost of falling in due to ε-greedy exploration noise.
Pick a scenario, predict what should happen, then apply it and compare the result with your prediction.
Run a short training pass and a longer one. The grid does not receive a plan; it gradually builds value through repeated backup.
Compare early and later policies, then inspect which cells changed most.
No state captured yet. Run or adjust the applet first.
-
Q1. Default grid. Set ε=0. Reset Q-values. Train continuously for ~500 episodes. Does the agent reliably reach the goal?
Q2. Set γ=0, ε=0.2. Reset Q. Train. Q-values (green color) appear:
Q3. Default grid, γ=0.9. Compare α=0.05 vs α=0.8 (same training time, slip=0). The α=0.8 Q-values evolve:
Predict how exploration and discounting affect learning, train the agent, then explain how value propagates from reward.
Reduced-motion settings are honored where possible. The current state is also available as text, so the main result does not depend only on color, animation, or spatial position.
Accessibility note: this layer gives a text equivalent for the applet state. It does not replace a full human screen-reader audit.
Reinforcement learning studies how an agent improves behavior through interaction rather than receiving the correct action for every situation. In this gridworld, actions produce immediate rewards and new states. Q-learning uses those experiences to estimate which action is valuable from each state.
Q(s,a) estimates the discounted return obtained after taking action a
in state s and then following a specified policy. Tabular Q-learning is designed to
estimate the optimal action-value function Q* under standard assumptions. The return
combines immediate reward with later rewards, discounted by powers of γ.
For a nonterminal transition from s to s' with reward r,
Q-learning applies
Q(s,a) ← Q(s,a) + α[r + γ maxa'Q(s',a') − Q(s,a)].
The bracketed term is the temporal-difference error. The learning rate α controls
update size. The discount factor γ controls the weight of estimated future return.
At a terminal state there is no future action value, so the target is the terminal reward alone.
With epsilon-greedy behavior, the agent selects a random action with probability ε
and otherwise selects a currently greedy action. Setting ε = 0 can prevent reliable
discovery when initial tie-breaking and early experience favor an unhelpful route. Too much
exploration can also prevent behavior from settling.
Setting γ = 0 removes delayed value from the target, so only immediate rewards are
learned. A negative per-step reward makes shorter successful paths more valuable. With zero step
cost, multiple successful routes can have equal return, so the learned policy may be indifferent
among paths of different lengths.
In a stochastic or persistently exploratory environment, SARSA can prefer a safer route because its target includes the actions the behavior policy may actually take. Q-learning evaluates the greedy next action and may favor a shorter route near danger. This distinction is most clearly studied in the dedicated cliff-walking comparison; it is not a guarantee that every pit layout will produce the same two paths.
Convergence results for tabular Q-learning require conditions such as sufficient exploration, suitable learning-rate schedules, and a stationary finite Markov decision process. Deep RL replaces the table with function approximation and introduces replay buffers, target networks, optimization instability, partial observability, and much larger state spaces. The applet keeps the state and update visible so those later complications have a clear reference point.
Curriculum: States, actions, rewards, discounted return, Q-values, temporal-difference learning, on-policy versus off-policy learning, exploration, and terminal transitions.
Pre-exploration prompts:
Post-exploration prompts:
ε = 0 with moderate exploration over several resets, not just one run.γ = 0. Which states can acquire nonzero values and why?Misconceptions to test: Q-values are a prewritten plan; terminal states bootstrap from themselves; more exploration is always better; one finite run establishes algorithmic convergence.
强化学习研究智能体如何通过交互改进行为，而不是在每种情形下都获得正确动作。在这个网格世界中，动作会产生即时奖励和新状态。Q 学习利用这些经验估计在每个状态下哪个动作更有价值。
Q(s,a) 估计在状态 s 采取动作 a 后，再遵循某一策略所获得的折扣回报。表格型 Q 学习在标准假设下用于估计最优动作价值函数 Q*。回报把即时奖励与后续奖励组合起来，后续奖励按 γ 的幂进行折扣。
对于从 s 转移到非终止状态 s' 并获得奖励 r 的经验，Q 学习执行
Q(s,a) ← Q(s,a) + α[r + γ maxa'Q(s',a') − Q(s,a)]。
方括号中的量是时序差分误差。学习率 α 控制更新幅度，折扣因子 γ 控制估计未来回报的权重。到达终止状态后不存在未来动作价值，因此目标只包含终止奖励。
在 epsilon-greedy 行为中，智能体以概率 ε 随机选择动作，否则选择当前贪心动作。把 ε = 0 可能使发现过程不可靠，因为初始平局处理和早期经验可能偏向无效路线。探索过多也会使行为难以稳定。
把 γ = 0 会从目标中移除延迟价值，因此只学习即时奖励。负的每步奖励会让更短的成功路径具有更高价值。步进成本为零时，多条成功路线可能拥有相同回报，所以学习到的策略可能对不同长度路径没有偏好。
在随机环境或持续探索环境中，SARSA 的目标包含行为策略实际可能采取的动作，因此可能偏好更安全路线。Q 学习评估贪心下一动作，可能偏好靠近危险的较短路线。这种差异在专门的悬崖行走比较中最清楚，并不保证每一种陷阱布局都会产生相同的两条路线。
表格型 Q 学习的收敛结果需要充分探索、合适的学习率方案以及平稳有限马尔可夫决策过程等条件。深度强化学习用函数近似取代表格，并引入经验回放、目标网络、优化不稳定、部分可观测和更大的状态空间。本工具让状态和更新保持可见，为后续复杂情况提供清晰参照。
课程目标：状态、动作、奖励、折扣回报、Q 值、时序差分学习、同策略与离策略学习、探索和终止转移。
探索前问题：
探索后问题：
ε = 0 与适度探索，不要只看一次运行。γ = 0。哪些状态能获得非零价值，为什么？需要检验的误解：Q 值是预先写好的计划；终止状态会从自身自举；探索越多越好；一次有限运行能够证明算法收敛。