Introducing MazeBench
MazeBench is an enormous open world resembling an intense labyrinth, constructed to address the capabilities of long running agent loops inside three-dimensional space. It evaluates whether agents demonstrate visual spatial reasoning to perceive their surroundings. To make MazeBench more interesting than simple walkways, we placed Sokoban-style box pushing puzzles within the world. The boxes come in a variety of shapes and sizes, so agents must correctly interpret their structure before they can plan how to use them. These puzzles require reasoning over long time horizons, often over one hundred moves. Therefore agents must use boxes as tools to open new walkways and gain access to larger portions of the maze. MazeBench offers a few other simple mechanics such as elevator tiles, toggleable walls, and icy slopes. These enable more three-dimensional reasoning and make the environment fun to play. Our team has filled over two hundred rooms with formidable challenges, and although each puzzle was tested and solved internally, we placed no cap on difficulty. The rooms are sorted as a curriculum so that the most extreme ones are furthest away. However, players will find that even the introductory levels ramp up their difficulty at a rapid pace. In short, MazeBench is a demanding environment that requires long term planning with visual spatial reasoning.
How MazeBench is run
Models play MazeBench inside their native agentic harnesses, like Codex and Claude Code, allowing long runs to be cost effective. MazeBench also used Prime Intellect verifiers v1 which supports a wide range of models and harnesses, for example the new Kimi K3 on Kimi Code. Agents are given a minimal prompt, all they know is that they are playing a 3D grid game where the goal is to hunt for gems by exploring as many rooms as possible.
Agents are exposed to eleven actions through the MCP server: four movement actions, four camera actions, an undo command, level reset, and a teleport to room X Y. Agents can direct the player to move up, down, left, and right. They may also rotate the camera up, down, left and right, to observe the game from twenty different angles. As the camera rotates, the movement commands adjust accordingly. When a player dies by falling into an abyss, models are forced to either undo, reset the level, or teleport to another room. Otherwise, models are never given hints on when to use undo. They are not told if their action was successful or if they reached a softlock. They must infer this and use undo to recover boxes they deadlocked against a wall. For rooms with long drawn out solutions, undo is a great relief. However, frontier models occasionally forget to take advantage of it, instead resetting the level if they believe they made a fatal mistake. Level reset restores the room to the state it was when the player entered it. The teleport command lets agents jump to any room they have visited before. The player will always teleport to a designated starting position within that room. They cannot teleport to any position within the room. Frontier models frequently teleport between rooms to refresh their memory and see if anything else catches their attention. Early runs of MazeBench showed models attempting to cheat by teleporting to rooms they had never visited before.
Tools vs No Tools
We think it is interesting to report how well agents do with and without tools. Without tools we can better investigate the raw intelligence of each agent and their long term planning capabilities. When tool mode is on, coding agents may write files and execute programs in an attempt to simulate the physics of the environment. The downside of tool-use is it reduces the amount of long term planning required to solve puzzles that are simple to define. Frontier coding agents have no trouble reverse engineering gameplay and generate solver algorithms. MazeBench tries to guard against tool-use coding agents by designing many rooms that are computationally hard and feature unusual physics hidden deep within the state graph. Agents will need to hypothesize the physics perfectly, and write highly efficient algorithms with specific heuristics. We have not yet completed testing MazeBench on coding agents with specialized harnesses.
Vision vs Text
MazeBench is intended to be a vision benchmark, but since not all models are multimodal, we have two text-only tracks as well: ASCII mode and JSON mode. In vision mode, agents view a 512x512 image of the world rendered in 3D. Quickly, the agents will learn that the green box is the player, that gems actually look like gems, that the black openings in the floor lead to an abyss that the player can fall into and die, that blue boxes are pushable no matter the size, and that walking into another room moves the camera to focus on that room. Later it will learn that different colored boxes do slightly different things. The purpose of MazeBench is not for agents to learn the rules of the game, they can tell they are playing a generic Sokoban, instead the purpose of MazeBench is to see whether agents can accurately depict complex boxes and figure out how to maneuver them around in a 3D world and execute long horizon plans with them.
We expect the two text tracks of MazeBench to dramatically increase the performance of coding agents. ASCII mode renders the environment almost the same as vision mode, but instead of a grid of pixels, it is a grid of text where each tile is a group of 4x4 characters. This way, viewing the room top down is only 64x64 characters. Since MazeBench is 3D, converting the camera to isometric mode makes it possible to render it as text. ASCII mode even allows models to rotate the camera, preserving all twenty different camera angles from vision mode. Models are told up front that the “P” character is for the player and the “G” character is for the gem. All other objects are assigned random characters. ASCII mode without tools is useful because agents still have to rely on their ability to parse 3D objects and plan with them. Therefore, ASCII mode without tools is the universal option to test visual spatial reasoning in long time horizons across vision and non-vision models alike. As of now the leaderboard only displays this configuration.
The other text track, JSON mode, gives the agent a list of objects inside the camera view, with their associated locations. This mode reduces any friction coding agents may have experienced in vision and ASCII mode. JSON mode also has an “omniscient” setting where agents can see every object, even those obstructed from view. Programs written by the agent can cite the JSON data structure, therefore this mode is useful for exclusively testing whether models can guess the physics and write efficient algorithms to solve MazeBench in a reasonable time.
Scoring
MazeBench requires agents to find one hundred gems hidden across two hundred plus rooms. Scoring is based on how many gems were collected, for example one gem equals a one percent score. Agents are also given an exploration score based on how many rooms were visited. It is possible that exploration score will reflect gem score in some way, but collecting gems is more impressive since it requires mastery over the puzzles. For instance, Fable 5 was able to explore rooms faster than GPT-5.6. But GPT-5.6 collected its first gem long before Fable 5. It appears as though Fable 5 was reluctant to collect the first gem, only doing so once it exhausted all easier paths to explore. On the other hand, GPT-5.6 found the location of a second gem and put all its focus into it, halting exploration elsewhere, which was its downfall. A perfect agent should be a balance of the two, exploring when appropriate, attacking gems when it finds them, and saving a problem for later if it proves too tough.
Agents playing MazeBench can run for as long as they want without restriction. For the sake of costs we do stop agents when we detect stagnation. Stagnation occurs when an agent walks in circles failing to discover new board states after a long period of time. Agents from 2025 were cursed to enter stagnation within several moves of playing the game. Previous generation models like GPT-5.5 and Opus 4.8 were the first to break through early stagnation patterns and continue to have creative ideas along their runs. But their creativity ran out over time as the puzzles became multistep. GPT-5.6 and Fable 5 were the first models to complete multistep problems effortlessly, opening up several rooms to explore. Every agent tested was paused due to entering a long period of stagnation, with the exception of Fable 5. Running Fable 5 in the API is far too costly, so will continue to track Fable 5’s progress in Claude Code when usage resets.
Findings with OpenAI Models
The MazeBench environment was built to address the capabilities of long running agent loops inside a 3D open world. Previous generation agents had almost no ability to progress through MazeBench. Each one rapidly devolved into stagnation where they began to wander in circles. Every agent had a corresponding novelty chart which showed whether it was discovering new states or not. The chart below shows how GPT-5.4 does not try new things, but revisits previous locations meaninglessly.
The next model, GPT-5.5, was the first to show signs of exploration. Its initial burst of novelty came when it had the brilliant idea to start pushing boxes. After this insight, however, its ingenuity died down. The introductory puzzles that followed were too overwhelming because they required a modicum of planning. It tried its very best not to stagnate, as evidenced by the small bumps of novelty, but each one was short-lived. Ultimately, it was unable to progress.
Fable 5 and GPT-5.6 were the first agents to repeatedly lift themselves out of stagnation valleys. They were the first to solve two-to-three step planning problems naturally. Fable 5 and GPT-5.6 are flavored quite differently, and I suspect Fable 5 is strongly superior. But GPT-5.6 in particular did something stunning. It stumbled across a secret room where I had hidden a surprise puzzle of absurd complexity. And after glancing at it, it wisely walked in the opposite direction and left. But during this time it spent hundreds of moves doing seemingly nothing. It appeared to have given up entirely diving headlong into stagnation valley. But once it was satisfied with itself, it returned to this secret room, and executed a masterful one hundred move strategy nearly to perfection. All without tool use. This was incredibly shocking, as this puzzle took me thirty minutes to solve. The chart below shows how GPT-5.6 lifted itself out of stagnation and walked through a long time horizon puzzle effortlessly.
We hypothesize that Fable 5 and GPT-5.6 models go through periods of stagnation differently than their predecessors. They spend this time experimenting with the physics of the world. With enough experimentation elsewhere, GPT-5.6 was able to put a long plan into action without having attempted the puzzle before.
Surprisingly, GPT-5.6 was unable to fully solve the long time horizon puzzle mentioned above. It kept messing up right at the end. But it became obsessed with the puzzle, at the expense of trying anything else. Had the puzzle been impossible, who knows how long it would have spent burning tokens. GPT-5.6 is capable, but single-minded. On the other hand, Fable 5 has a broader mind and is much more explorative. It knows to save hard puzzles for later and try the easy ones first. Soon we will launch longer runs with Fable 5 and GPT-5.6 to better understand their capabilities.
OpenAI models
Unsurprisingly, smaller models are less creative than larger models. GPT-5.6 Sol is worlds ahead of GPT-5.6 Terra. It attacked an intense puzzle and nearly succeeded, while Terra was struggling to learn how to push boxes into holes. Luna had trouble learning that it was bad to walk into holes.
Every next generation model by OpenAI has become more creative. GPT-5.3 struggled with exploring anything beyond a few paces. GPT-5.4 learned how to avoid pits, push boxes, and enter new rooms, but failed once one box required to be pushed back and forth to bypass it. GPT-5.5 had no issue with that, but got stuck when boxes had to be used as tools. All this was child's play for GPT-5.6, instead it found the hardest problem it could and stayed there.
Findings with Anthropic Models
Fable 5 performed well on MazeBench. Almost nothing gave it pause, except for its own usage limits. It completed multistep problems without an issue, using boxes as tools to push other boxes out of the way. Opus 4.8 struggled to strategize multistep plans. The Opus 4.8 run and GPT-5.5 were identical. They had the same weak points in exactly two areas.
These two tool-based puzzles are required to escape the first four rooms. Both of them requires two-step reasoning. To collect the gem, models must move the jigsaw boxes in a certain way to get by. GPT-5.5 and Opus 4.8 push both boxes at once, pinning in the gem. The second failure mode in when models attempt to cross a bridge, but a large box blocks the path. The trick is to use the tool to clear the box from the bridge, but neither GPT-5.5 nor Opus 4.8 could use the tool, even coming within one step from figuring it out!
Fable 5 completed these without difficulty, however it did save the gem for later after its exploration was complete. Previous tests showed that GPT-5.6 could do the same. However, Fable 5 did something that GPT-5.6 never could. It learned how to use boxes as hooks to pull others out of the way. GPT-5.6 mastered the art of pushing, but it was never capable of pulling.
Fable 5 twice demonstrated it understood how to pull boxes using hooks. However, it's progress started to slow after this. It did not solve puzzles where the boxes had multiple hooks, and could not plan for solutions where multiple boxes needed to be used as tools at once.
Here we were forced to conclude Fable 5's run.
Anthropic models
Again unsurprisingly, the smaller Anthropic models display less creativity than their larger counterparts. Sonnet 4.6 is the best in its class, we ran multiple tests with Sonnet 5 but it could not outperform Sonnet 4.6. Both GPT-5.5 and Opus 4.8 runs were very long, but they concluded with about the same behavior. Unfortunately, we could not finish Fable 5's runs, it's capabilities may be even greater than shown here. Its current run would have cost several grand in the API.
A closer look
Findings with other models
We ran MazeBench on a plethora of other models but none of them exhibited any behavior worth noting. Kimi K3 had trouble exploring in the second room, performing worse than GPT-5.5 and Opus 4.8, while Gemini 3.1 Pro was unable to escape the first room, performing at GPT-5.6 Luna quality.
Special Thanks
The construction of MazeBench was sponsored by Prime Intellect as part of their residency program. A special thanks to Florian Brand (@xeophon) and Sebastian Müller (@omouamoua) for their mentorship and support.
