The CartPole lab
CartPole is where I started with reinforcement learning: a cart on a track, a pole hinged on top, and a push left or right fifty times a second to keep it up. It runs here in your browser with the same physics as Gymnasium’s CartPole-v1. Watch the agents I trained, try it yourself, or train a new one and see it learn.
The network dqn_train.py saved after 500 episodes, with its real weights. It nearly always lasts all 500 steps; once in a while the cart drifts off the end of the track just before.
Step 0 of 500
Episode 1
Push the cart yourself with the buttons, or click the picture and use ← →. One push is 20 newtons for 0.06 seconds: the DQN rides out one, but two close together can topple it.
What it sees
- Cart position
- 0.01 m
- Cart speed
- -0.05 m/s
- Pole angle
- 0.2°
- Pole turning
- 3°/s
What it expects from each push
- Push left
- 55.9its choice
- Push right
- 55.9
The points it expects to follow; it takes the higher.
Recent scores: –
How it works
What the agent sees
Four numbers, fifty times a second: where the cart is on the track, how fast it is moving, how far the pole leans, and how fast it is tipping. No picture, just those four.
What it can do
Push the cart left or right, with 10 newtons. Nothing else: it has to pick one every step. Each step the pole stays within 12° of upright and the cart stays on the track earns a point. An episode ends with a fall or after 500 steps, so 500 is a perfect score.
How REINFORCE learns
It keeps odds for each push and plays a whole episode by drawing on them. Then it makes every push it chose more likely, and the more points came after a push, the more. Pushes just before a fall had few points after them and gain little. Over hundreds of episodes the odds tilt towards pushes that keep the pole up.
How DQN learns
It learns a score for each push in each situation: the points it expects to follow. After every step it takes 64 moments from its memory, compares each score with the point that came next plus its own best guess from there, and moves the score a little towards that. To play, it takes the push with the higher score.
Why my first try fails
In cartpole.py I changed the rules: the pole could lean to 45°, and I wrote my own rewards for staying upright and near the middle. The network it saved scores pushing right higher in every situation, so it always pushes right. It is here because it shows how much the rewards decide what an agent learns.
The physics are Gymnasium’s, step for step, and checked against it. The two agents are the networks I saved, run with the same arithmetic as PyTorch. Training uses my scripts’ settings, and nothing leaves your browser.