A tiny AI dreams up Doom and Pac-Man on a cheap Raspberry Pi board

A developer who goes by SheepSeb published a project called tinyworldformer, and last night it shot to the top of Hacker News. The idea is wild: two tiny neural networks run on a Raspberry Pi Pico 2 W, a microcontroller board with 520 KB of RAM. One draws a game frame by frame, the other plays it. There is no actual game code on the board at all.

The first network is a world model. It looks at the current frame and the button being pressed and predicts the next frame and the reward. It has between 26,000 and 131,000 parameters stored in 8 bits, a speck of dust by modern AI standards. To fit in memory, the author used a looped transformer, where one block runs several times with shared weights. The second network is even smaller, 9,000 to 16,000 parameters, and decides where to move.

There are five games: Breakout, Pong, Flappy Bird, a simplified Pac-Man and a mini Doom with a first-person view. Accuracy is solid: the model gets 100% of frames right in Flappy Bird, 99.99% in Breakout and 83.6% in Doom. The agent can learn right on the board, and pressing any button hands control to a human, so you play a game the network is making up as it goes.

There is a funny side effect too. The network's dream is not perfect, and the agent learned to exploit its mistakes: in the imagined Breakout it scored 210 points when the real maximum is 56, and managed only 28 in the real game. The author fixes this with extra training data. He trained the world models on an ordinary RTX 4060 and openly says he has not measured the frame rate on the board yet.

You can try it right now: all five games run in the browser on the project page, and the code is open on GitHub.

While the giants build virtual worlds in data centers, this one has a board the size of a thumb drive dreaming up Doom.