Learning RL by Playing Games
Teaching small neural networks to play CartPole, Breakout and Pac-Man, one algorithm at a time.
An early, hands-on trip through reinforcement learning. I worked through the Hands-on Reinforcement Learning book (DQN, REINFORCE, actor-critic, PPO) on CartPole, then built my own pygame games — Breakout and Pac-Man — to train agents on. Most of the work went into reward shaping and tuning: re-running versions until the agents improved, even when they still couldn't play well.
- Status
- An archived learning project, last changed in June 2026. The scripts and saved models are still there, but it isn't maintained.
Worth knowing§
- Training a DQN on my own pygame Breakout, I recorded 8,021 steps of my own play to seed its replay buffer.
- Replayed in Atari Breakout, the episode-500 DQN checkpoint averaged just 3.3 points over three games: an honest measure of how far it had to go.
How it works§
- 01
It started with the book's DQN and REINFORCE on CartPole, then my own CartPole variant with a 45° fail angle and hand-written rewards, which taught me how much the reward decides what the agent actually learns.
- 02
A Nature-style convolutional DQN trained on Atari Breakout from stacked 84×84 frames; later my own pygame Breakout with dueling and double DQN, prioritised replay, and a recorder that saved my own play as training data.
- 03
"Hit the Brick", a brick game of my own with a small feature-vector state, trained actor-critic agents (TD error, an entropy bonus, separate actor and critic learning rates), and I wrote the update step out in Chinese to be sure I understood it.
- 04
Then PPO with GAE and clipped updates. An over-built LSTM version didn't learn well, so I stripped it back to a simpler one, which solved CartPole and improved steadily on the brick game.
- 05
A grid Pac-Man of my own, with ghosts, power pellets and three lives, trained an actor-critic agent across 8 parallel environments, and a randomly generated maze was solved exactly by policy iteration.
