Artificial Intelligence: A Visual Learner's Series

Part 8: Reinforcement learning

Author: Christopher DanosAI by Claude

No right answers, just a score that often arrives late. The agent has to work out which actions earned it.

Learning from reward

In one line: Try actions, receive rewards or penalties, and gradually discover which behavior pays off over time.

A dog learning to fetch gets the treat at the end, so it must figure out which earlier moves mattered: the credit assignment problem.

Key terms: state, action, reward, return (total discounted reward), policy, value function V(s), action value Q(s, a).

Watch out: The agent optimizes the reward you wrote, loopholes included.

Explore or exploit?

In one line: Balance using what you know works with trying things that might work better.

  Your favorite taco place   ████████░░  pretty good, well known
  Restaurant never tried     ░░░░?░░░░░  could be better, could be worse

  ε-greedy: go to the favorite 90% of the time, try something random 10%
  UCB:      prefer options whose best plausible score is highest

Key terms: multi-armed bandit, ε-greedy, upper confidence bound (UCB), Thompson sampling, regret.

Watch out: Never exploring locks in early luck.

Learning values from experience

In one line: Compare your prediction with what happened a moment later and correct the difference, without waiting for the end.

  Leave work: "30 min"  ──▶  5 min in, traffic jam: "now 45 min"
                            update right away (TD error = +15 min)

Q (s, a) \leftarrow Q (s, a) + α [r + γ a^{'} max Q (s^{'}, a^{'}) - Q (s, a)]

The bracket is the TD error: how surprised the agent was. Q-learning nudges Q(s, a) a fraction α toward the better-informed guess.

Watch out: Learning guesses from guesses can become unstable, especially with neural networks.

Learning behavior directly

In one line: Do more of what led to high reward and less of what led to low reward, adjusting the behavior itself.

Like free-throw practice: keep the motions that went in, drop the ones that clanked, faster with a coach saying "better than your average."
Policy gradient methods (REINFORCE, PPO) handle continuous actions like steering angles.

Watch out: Big updates can wreck a good policy, so methods like PPO limit each step.

Deep and model-based RL

In one line: Neural networks let agents learn from raw pixels; a learned model of the world lets them plan ahead.

Approach	How it works	Example
Deep Q-network	Network estimates Q from screen pixels; replay memory and target network keep it stable	Classic video games
Model-based RL	Learn how the world responds, then plan or simulate	Far fewer real trials needed
Search + learning	Learned value and policy guide tree search, trained by self-play	AlphaZero-style board games
RLHF	Reward model learned from human preferences	Fine-tuning language models

Watch out: Policies learned in simulation often stumble on real hardware.

Self-test

Draw the agent–environment loop for a thermostat that learns to save energy.
Walk through one Q-learning update with numbers you choose.
Redraw the actor–critic diagram and say what each arrow carries.