mcAI
Why DreamerV3 stalls at the wood tier in Craftax, and what pushes it further.
A self-directed study of why DreamerV3, a world-model reinforcement learning agent, stalls early in Craftax, a fast 2D Minecraft-like game with a 67-achievement tech tree. I ported DreamerV3 to Craftax, checked that the stock setup reproduces published Crafter results, then tried fixes one at a time: a second, slower world model that keeps a coarse map, masking impossible actions, and a manager that picks tech-tree goals.
- Status
- Active research. The best run so far averages 9.73 achievements per episode; after the 11 October 2026 code audit every map-model run is being retrained with the fixes and re-scored, so the figures may move.
- , from its code on GitHub
Worth knowing§
- Across 30 evaluation episodes the best variant averaged 9.73 achievements per episode, against 4.44 for stock DreamerV3, both after 1.1M steps of training on one seed.
- On the original Crafter the stock configuration scored about 9.7 after 1M steps, against the 11.70 DeepMind published: the check that the port was sound.
How it works§
- 01
The port was checked before any result was trusted. Stock DreamerV3 stalled on Craftax, so the same configuration was run on the original Crafter, where it came within reach of DeepMind's published score; that ruled out a broken port and made the Craftax gap a finding.
- 02
The map reaches the actor but never the world model. A second recurrent model ticks every 8 steps and predicts a 12×12 coarse map and the agent's own cell, and feeds only the policy and critic, behind a stop-gradient and a gate that starts closed, so a wrong map cannot invent reward in imagined rollouts.
- 03
Failures are measured, not guessed. An audit scores each checkpoint against which of the 43 actions would actually do something: one run pressed "craft wood pickaxe" in none of the 150 states where it could, while picking the wood sword a fifth of the time. Masking impossible actions in training then raised its score.
- 04
A manager handles plans longer than imagination reaches. The actor imagines 15 steps ahead and the iron tier needs a hundred or more, so a manager picks one of 13 tech-tree goals and holds it. Its first version learned nothing; paying both levels when a goal is reached, with one critic shared between them, gave the best run and the first regular stone pickaxe.
- 05
Bugs have reversed findings more than once, and the write-ups say so: a policy collapse blamed on the map model turned out to be a crop mismatch in imagination, and an outside audit on 11 October 2026 found three more bugs, so the map-model runs are being retrained and re-scored.
Built with§
- Python, also used in Datacenter Tracker, Squash Review, and Learning RL by Playing Games
- JAX
- DreamerV3 (fork)
- Craftax
- NumPy, also used in Learning RL by Playing Games
- pytest, also used in Datacenter Tracker
- Matplotlib, also used in Learning RL by Playing Games
- Pygame, also used in Learning RL by Playing Games
