I was inspired by Mechanize’s recent release of GBA Eval to build an evaluation of my own. Pokéval gives an agent two hours to rebuild a Generation 1 Pokémon battle engine matching battle logs from Pokémon Showdown. Like Mechanize, I test a hypothesis about frontier AI agents through this project.
Claim
Mechanize built GBA Eval to argue that public coding benchmarks misrepresent what frontier models can do. Benchmark tests reject correct solutions, prompts leave the intended task ambiguous, and scores drift away from the gains people see when they actually use the models. To test this, they built one long, hard task graded against an accurate reference implementation, and showed that frontier models can build largely accurate solutions.
While their tests seem persuasive, my personal experience using AI extensively differs. In my opinion, models have incredibly deep domain knowledge on many subjects, but they are often myopic (unable to understand the bigger picture) and not detail oriented (especially when multitasking).
I created Pokéval to test these claims.
Benchmarks often use small, focused tests. Pokéval instead prioritizes long term planning: deciding how to attack a large, underspecified problem before implementing it, and staying on that plan for two hours.
Pokéval also directly tests attention to detail. A Generation 1 battle features dozens of mechanics interacting through a single random number generator, so one wrong detail early in a turn shifts every draw that follows.
My hypothesis is that most frontier agents are still weak at both, and that the differences between models on Pokéval come more from these two abilities than from how well each model writes code. If true, this would also suggest that humans are still needed in the loop for software heavy work. In my opinion, experienced developers can do much of the planning and proactive thinking, and delegate specific, tedious technical tasks to AI agents.
The eval
Pokémon Showdown is a website where people play Pokémon battles against each other. It supports every generation of the games and many formats, from Anything Goes single battles, where almost any Pokémon is allowed, to restricted doubles formats like VGC. In a single battle, each player brings a team of six. Every turn, both players either use one of their active Pokémon’s moves or switch in another team member, and the battle ends when one side has no Pokémon left. Pokéval restricts this to single battles between teams of six under Generation 1 rules. What makes Showdown a good reference is that its battle simulator is open source and runs on its own, outside the website. A program can give it two teams and the choices both players made, and read back the battle log, a line for every event in the battle.
Task. The agent writes engine.py, a program that replays Generation 1 battles. For each battle, the engine receives two teams of six level 100 Pokémon with their moves, the choices both players made each turn, and the seed for the random number generator, which fixes every random event in the battle. From these it has to print the battle log Showdown would print, line for line.
NoteAn example battle
A battle generated for this post with the same rules as the test set. The input is one JSON object per battle:
The agent starts with three documents and three data files. TASK.md contains the goal, the tools, and how the run works. SPEC.md defines exactly what the engine reads and writes, including every line format the log can contain and the random number generator. docs/README.md lists every point at which the reference draws a random number, and the data files give the types and base stats of all 151 species, the data for all 165 moves, and the type chart.
NoteTASK.md
# TaskImplement a Generation 1 Pokémon battle engine that reproduces the reference implementation's battlelog. Given a random seed, two teams, and the choices both players made, your engine replays the battleand emits the log. You are not playing; you reproduce what the rules produce.IMPORTANT: A complete engine is not achievable in the time available. If you believe you are done, youare mistaken. Keep verifying against the reference and fixing until your time runs out.This task runs continuously until the time limit. If you stop working, you will be re-launched with yourprior conversation and working tree still there. There is no signal you can emit to end the task.## What's in the container| path | contents ||---|---||`/task/SPEC.md`| input, output, log grammar, random number generator ||`/task/docs/README.md`| where the reference draws random numbers; the data files ||`/task/docs/*.json`| species, moves, type chart ||`oracle`| runs the reference on a battle you send it |Read `SPEC.md` first, then `docs/README.md`.## EnvironmentPython 3.11 with the standard library only. No internet access, no package installation, no Pokémonlibraries. Work in `/task`; anything you leave there persists.## The oracleGeneration 1 behavior is defined by the reference implementation and is not documented here. The`oracle` command runs the reference on any battle you design and returns the log the grader comparesagainst. You do not have access to the reference's source code. It is rate-limited. Run `oracle help` forthe query format, the response fields, and the limits.The oracle is important because there is no other test suite. This task is much harder than it may firstappear: matching the opening lines of a battle is the easy part. No error, failure or edge case is out ofscope. This is not a minimum viable engine.## What you produce`/task/engine.py`. It is run as```python3 /task/engine.py < cases.jsonl```reading one case per line from stdin and writing one result per line to stdout, in the format given in`SPEC.md`.It runs in this container, from `/task`, so it can import other files you leave there and read `docs/`.The oracle is not available to it. Keep a runnable engine at `/task/engine.py` at all times and do notmove it once work is underway; external tooling expects it there. Execution of the held-out set must finish in a timely manner. Writediagnostics to stderr; exit non-zero only on a fatal error.## GradingAt the end, your engine is judged by how closely its logs match the reference's on a held-out set ofbattles you have not seen.
NoteSPEC.md
# Gen 1 Pokémon battle engineWhat your engine reads and writes, and the random number generator. The task is in `TASK.md`; the reference'sdata and its random draw sites are in `docs/README.md`.## InputOne JSON object per line on stdin:```json{"id":"ps-000001","seed":"1,18,42,58","teams": {"p1":"<packed team>","p2":"<packed team>"},"choices": [{"turn":1,"p1":"move 3","p2":"move 3"}, {"turn":1,"p1":"switch 2","p2":null},...]}```**Seed.** Four 16-bit values, most significant first: `"1,18,42,58"` means the generator starts from`x0 = (1 << 48) | (18 << 32) | (42 << 16) | 58`. Every case has its own seed.**Teams.** Team members are separated by `]`. Each member is twelve fields separated by `|`: the first is thespecies name, the fifth is the comma-separated list of one to four move ids from `docs/moves.json`, and everyother field is empty. All Pokémon are level 100 with the reference's default stats.**Choices.** In submission order. Each entry has the turn it was submitted in and one string per side. `move N`uses the Nth move in the active Pokémon's current move list. `switch N` brings in the Nth Pokémon in the side'scurrent order. `null` means that side had no decision to make at that point.- After a switch, the Pokémon that came in occupies slot 1 and the one that went out occupies the slot the incoming Pokémon came from.- Mimic replaces its own slot with the copied move; Transform replaces all slots with the target's moves.- While the engine has locked a Pokémon into a move (for example Rage, Thrash, Petal Dance, or the second turn of a two-turn move), the recorded choice is `move 1` whatever slot that move is in, and the locked move executes.- While a Pokémon is holding a partial-trapping move (Bind, Clamp, Fire Spin, Wrap), the recorded choice on the following turns may name any slot, and the engine repeats the trapping move regardless.When such locks begin and end is Generation 1 behavior, which the reference can be queried for.## OutputOne JSON object per line on stdout:```json{"id":"ps-000001","log": ["|switch|p1a: Exeggutor|Exeggutor|330/330","|turn|1",...]}````id` is copied from the input. `log` is the ordered list of graded lines your engine produces. Do not emit anyother line type, and do not emit blank lines. In the reference's graded log, consecutive identical lines arecollapsed to one.### Graded linesEvery graded line has one of the shapes below. Slots: `<mon>` is `p1a: Name` or `p2a: Name`, the activePokémon of player 1 or 2; `<species>` is a species name from `docs/species.json`; `<move>` is a move namefrom `docs/moves.json`; `<hp>` is `current/max`; `<status>` is one of `par slp frz brn psn tox`; `<stat>` is astat id; `<n>` is an integer; `<player>` is `P1` or `P2`; `<type>` is a type, or two joined by `/`. Where aslot takes only a few values in that shape, they are listed under it. Everything not in angle brackets isliteral text.**`|move|`**```|move|<mon>|<move>|<mon>|move|<mon>|<move>|<mon>|[from] <move>|move|<mon>|<move>|<mon>|[miss]|move|<mon>|<move>||[still] <move>: Razor Wind Fly Skull Bash Solar Beam Dig Sky Attack|move|<mon>|<move>|<mon>|[from] <move>|[miss] <move> #2: Razor Wind Sky Attack Fly Rage Dig Solar Beam Skull Bash Petal Dance Thrash Metronome Mirror Move|move|<mon>|<move>||[from] <move>|[still] <move> #1: Sky Attack Skull Bash <move> #2: Mirror Move Metronome```**`|-damage|`**```|-damage|<mon>|<hp>|-damage|<mon>|0 fnt|-damage|<mon>|<hp> <status> <status>: par psn slp brn frz tox|-damage|<mon>|<hp>|[from] Recoil|-damage|<mon>|<hp> <status>|[from] <status>|[of] <mon> <status> #1: psn brn <status> #2: psn brn|-damage|<mon>|<hp>|[from] confusion|-damage|<mon>|<hp> <status>|[from] <status> <status> #1: tox psn brn <status> #2: psn brn|-damage|<mon>|0 fnt|[from] Recoil|-damage|<mon>|0 fnt|[from] <status>|[of] <mon> <status>: psn brn|-damage|<mon>|<hp>|[from] <move>|[of] <mon> <move>: Leech Seed|-damage|<mon>|0 fnt|[from] confusion|-damage|<mon>|<hp> <status>|[from] Recoil <status>: par brn psn tox|-damage|<mon>|0 fnt|[from] <status> <status>: psn brn|-damage|<mon>|0 fnt|[from] <move>|[of] <mon> <move>: Leech Seed|-damage|<mon>|<hp> <status>|[from] confusion <status>: par```**`|turn|`**```|turn|<n>```**`|switch|`**```|switch|<mon>|<species>|<hp>|switch|<mon>|<species>|<hp> <status> <status>: par psn frz slp brn tox```**`|faint|`**```|faint|<mon>```**`|-resisted|`**```|-resisted|<mon>```**`|-crit|`**```|-crit|<mon>```**`|-miss|`**```|-miss|<mon>```**`|-supereffective|`**```|-supereffective|<mon>```**`|-prepare|`**```|-prepare|<mon>|<move> <move>: Razor Wind Fly Skull Bash Solar Beam Dig Sky Attack```**`|-hitcount|`**```|-hitcount|<mon>|<n>```**`|-immune|`**```|-immune|<mon>|-immune|<mon>|[ohko]```**`|cant|`**```|cant|<mon>|<status> <status>: slp frz par|cant|<mon>|partiallytrapped|cant|<mon>|recharge|cant|<mon>|flinch|cant|<mon>|Disable|<move> <move>: Sky Attack Rage```**`|-boost|`**```|-boost|<mon>|<stat>|<n> <stat>: def spe atk evasion spa spd|-boost|<mon>|<stat>|<n>|[from] <move> <move>: Rage <stat>: atk```**`|-unboost|`**```|-unboost|<mon>|<stat>|<n> <stat>: spe def atk accuracy spa spd```**`|-heal|`**```|-heal|<mon>|<hp>|[from] drain|[of] <mon>|-heal|<mon>|<hp> <status>|[silent] <status>: slp par|-heal|<mon>|<hp>|[silent]|-heal|<mon>|<hp>|-heal|<mon>|<hp> <status>|[from] drain|[of] <mon> <status>: par brn psn|-heal|<mon>|<hp> <status> <status>: par```**`|-start|`**```|-start|<mon>|<move> <move>: Bide Substitute Reflect Mist Light Screen|-start|<mon>|confusion|[silent]|-start|<mon>|confusion|-start|<mon>|move: <move> <move>: Focus Energy Leech Seed|-start|<mon>|Mimic|<move>|-start|<mon>|Disable|<move>|-start|<mon>|typechange|<type>|[from] move: <move>|[of] <mon> <move>: Conversion <type>: Water Normal/Flying Ground```**`|-activate|`**```|-activate|<mon>|<move> <move>: Bide|-activate|<mon>|confusion|-activate|<mon>|<move>|[damage] <move>: Substitute|-activate|<mon>|move: <move> <move>: Haze Mist```**`|win|`**```|win|<player>```**`|-status|`**```|-status|<mon>|<status> <status>: par psn brn frz tox|-status|<mon>|<status>|[from] move: <move> <move>: Rest Sleep Powder Hypnosis Spore Sing Lovely Kiss <status>: slp|-status|<mon>|<status>|[silent] <status>: psn```**`|-message|`**```|-message|The foe <species> can't be hit while invulnerable!```**`|-fail|`**```|-fail|<mon>|-fail|<mon>|<status> <status>: par slp tox|-fail|<mon>|move: <move>|[weak] <move>: Substitute|-fail|<mon>|move: <move> <move>: Substitute```**`|-fieldactivate|`**```|-fieldactivate|move: <move> <move>: Pay Day```**`|-end|`**```|-end|<mon>|<move> <move>: Bide Substitute Disable|-end|<mon>|confusion|-end|<mon>|move: <move>|[silent] <move>: Focus Energy```**`|-mustrecharge|`**```|-mustrecharge|<mon>```**`|-ohko|`**```|-ohko```**`|-curestatus|`**```|-curestatus|<mon>|<status>|[msg] <status>: slp frz|-curestatus|<mon>|<status>|[silent] <status>: psn```**`|-transform|`**```|-transform|<mon>|<mon>```**`|-clearallboost|`**```|-clearallboost|[silent]```**`|-nothing|`**```|-nothing```**`|tie|`**```|tie```Any line type or shape not listed here does not occur in a graded log.## RandomnessThe battle uses a 64-bit linear congruential generator:```x_{n+1} = (a * x_n + c) mod 2^64a = 0x5D588B656C078965c = 0x00269EC3```Every draw advances the generator once, then takes the upper 32 bits of the new state:`result = (x >> 32) & 0xFFFFFFFF`.```random(n) = floor(result * n / 2^32) -> integer in [0, n)random(from, to) = floor(result * (to - from) / 2^32) + fromrandomChance(num, den) = random(den) < numsample(list) = list[random(len)]shuffle of k elements = for i in 0 .. k-2: swap element i with element random(i, k) (k-1 draws)```Every place the reference draws, and when, is listed in `docs/README.md`.---There may be behavior not captured by this document, and there is a small chance that some of what it states isnot wholly accurate. The reference, as run by the oracle, is the source of truth.
Notedocs/README.md
# Reference dataThis folder holds the reference implementation's data, and a table of every point at which the referencedraws a random number. Generation 1 behavior is otherwise not documented here: it is defined by thereference implementation, which the `oracle` command runs on any battle you send it.## Where the reference draws random numbersEvery place the reference calls its random number generator, and when. `random`, `randomChance`, `sample`and shuffles are defined in `SPEC.md`. A sort shuffles each group of tied elements separately, one shuffleper group. Every draw advances the generator once.**turn start**| draw | call | fires when ||---|---|---|| active speed tie | shuffle of the 2 active Pokémon | whenever the engine visits both actives in speed order, which is at the start of every turn and after every action (the end-of-turn step counts as an action), and the two have equal speed. The list before the shuffle is [player 1's active, player 2's active]. No draw when speeds differ. || action order tie | shuffle of the 2 queued actions | when the two sides' chosen actions tie on priority and speed when the turn's queue is sorted. The list before the shuffle is [player 1's action, player 2's action]. || switch insertion tie |`random(first, last + 1)`| when a switch (including the lead switch-ins at battle start) is inserted into the queue and other actions tie with it; the index among the tied actions is drawn. |**before a Pokémon moves**| draw | call | fires when ||---|---|---|| full paralysis |`randomChance(63, 256)`| before every move attempt by a paralyzed Pokémon. || confusion self-hit |`randomChance(128, 256)`| before every move attempt by a confused Pokémon, after its confusion counter is decremented and only if the confusion did not just end. |**when a move executes**None of the draws in this group fire on the continuation turns of Bind, Clamp, Fire Spin or Wrap.| draw | call | fires when ||---|---|---|| accuracy check |`randomChance(accuracy, 256)`| for every move with an accuracy value, except: no draw for a sleep-inducing move against a target that must recharge (it always hits); no draw when a one-hit-KO move is used on a faster target or when the target is immune to the move's type (`|-immune|` instead). `accuracy` = ⌊acc × 255 / 100⌋; then ⌊accuracy × T[s]/100⌋ for the user's accuracy stage s; then ⌊accuracy × T[−e]/100⌋ for the target's evasion stage e (note the sign); where T indexes stages −6..+6 into `[25, 28, 33, 40, 50, 66, 100, 150, 200, 250, 300, 350, 400]`; then clamped to 1..255; then +1 if the move targets the user. A Pokémon locked into Thrash/Petal Dance or Rage starts from the accuracy stored on its previous turn instead of ⌊acc × 255 / 100⌋, and the stage steps apply again on top of it. || multi-hit count |`sample([2, 2, 2, 3, 3, 3, 4, 5])`| after the accuracy check passes, for a move that hits 2–5 times. || critical hit |`randomChance(critChance, 256)`| for every move whose damage goes through the damage formula, that is every move with `basePower` > 0 and neither a `damage` nor a `damageCallback` field in `docs/moves.json`. `critChance` = ⌊baseSpeed / 2⌋, with baseSpeed the base Speed of the user's own species (Transform does not change it); then ⌊÷2⌋ if the user has Focus Energy, else ×2 clamped to 1..255; then ⌊÷2⌋ for a normal move (`critRatio` 1) or ×4 clamped to 1..255 for a high-crit move (`critRatio` 2). || damage roll |`random(217, 256)`| for the same moves as the critical hit row, after type effectiveness is applied, only when the damage so far is greater than 1; the result multiplies the damage (⌊÷255⌋ afterwards). || Psywave damage |`random(1, ⌊1.5 × level⌋)`| when Psywave hits (`random(1, 150)` at level 100). || secondary effect |`randomChance(⌈chance × 256 / 100⌉, 256)`| for each secondary effect of a move that hit a target still standing (for multi-hit moves, on the last hit only); one less than the ceiling for confusion; skipped entirely when the secondary is paralysis, burn or freeze and the target shares the move's type. || Metronome |`sample` of 163 moves | when Metronome executes: every Generation 1 move except Metronome and Struggle, in numeric order. || Mimic |`sample` of the target's moves | when Mimic hits. || Disable: slot |`sample` of the target's move slots with PP left | when Disable hits. || Disable: duration |`random(1, 9)`| immediately after the slot draw. |**when an effect starts**| draw | call | fires when ||---|---|---|| sleep duration |`random(1, 8)`| when sleep is inflicted. || confusion duration |`random(2, 6)`| when confusion starts. || Thrash / Petal Dance duration |`random(2, 4)`| when the lock starts. || Bide duration |`random(2, 4)`| when Bide starts. || partial-trap duration |`sample([2, 2, 2, 3, 3, 3, 4, 5])`| when Bind, Clamp, Fire Spin or Wrap locks its user. |**end of turn**| draw | call | fires when ||---|---|---|| end-of-turn effect tie | shuffle of the tied effects (2–4) | when end-of-turn effects (poison, burn, Leech Seed, and the like) tie on order. The list before the shuffle holds player 1's active's effects before player 2's. |## The data files| file | contents ||---|---||`species.json`| 151 species: types and base stats ||`moves.json`| 165 moves ||`typechart.json`| type effectiveness |---There may be behavior not captured by this document, and there is a small chance that some of what it states isnot wholly accurate. The reference, as run by the oracle, is the source of truth.
Apart from the random draws, the documents describe no battle mechanics: there is no damage formula, no rule for move order, no explanation of how any status or move works, and no example battles. The agent is also not told how it is scored, only that its logs are compared against the reference on battles it has not seen. The documents alone are not enough to recreate the game faithfully, which is why I also built an oracle, explained later.
The agent works in Python 3.11 with only the standard library. It has no internet access and cannot install packages. The run continues until the time limit, and if the agent stops early, it is relaunched with its conversation and files intact. It is also told that a complete engine is not achievable in the time available: “If you believe you are done, you are mistaken.” Throughout the run, it has to keep a runnable engine at /task/engine.py.
Oracle. Showdown’s Generation 1 rules exist only as code, and the documents leave them out, so the agent needs some way to find out what the game actually does. The oracle lets the agent design any battle it wants, run it through Showdown, and see exactly what happens.
A query has the same format as a test battle: a seed, two teams and a list of choices. The oracle plays the battle in Showdown and returns the log exactly as the grader would see it, along with the number of turns played and whether the battle ended. If one of the choices is illegal, such as a move when the Pokémon has to switch, it stops there and returns the log up to that point with the reason. The oracle never scores anything, so to find a bug the agent has to compare its own engine’s log against the oracle’s.
NoteAn example query
A one on one battle designed for this post, not taken from the test set: Pikachu with Thunder Wave and Thunderbolt against Snorlax with Body Slam and Amnesia, with choices for three turns. The query:
The agent reaches the oracle through an oracle command in its container, which sends each query to Showdown running in a separate container. The agent has full shell access to its own container, so keeping Showdown elsewhere means it cannot read Showdown’s source code, and the oracle is switched off while the engine is graded. Queries are rate limited in battle turns. Each query costs the turns it plays and at least one, so a malformed query still costs a turn. The budget refills at 3,000 turns per minute up to a maximum of 5,000, and a single query plays at most 100 turns.
Noteoracle help
oracle: run the reference implementation on a battle you design, get back the graded log.
USAGE
echo '<query json>' | oracle one JSON object per line on stdin, one JSON object per line on stdout
oracle --budget how many turns are available right now
oracle help this text
QUERY (the same shape as a graded case; see SPEC.md "Input")
{"id": "any string (optional, echoed back)",
"seed": "1,2,3,4",
"teams": {"p1": "<packed team>", "p2": "<packed team>"},
"choices": [{"p1": "move 1", "p2": "move 1"}, {"p1": "switch 2", "p2": null}, ...]}
seed, teams.p1, teams.p2 and choices are required; id is optional; no other field is allowed. A choice
entry holds p1 and p2 (an integer "turn" field is accepted and ignored). Species names and move ids
must be the ones in docs/. Choices after the battle has ended are ignored.
RESPONSE FIELDS
id echoed if you sent one
log the graded log, exactly as the grader would see it
turns turns completed when the battle stopped (for a finished battle, the turn it ended on)
ended whether the battle reached |win| or |tie
charged turns deducted for this query: `turns`, minimum 1, maximum the per-query cap
available turns available right now
rejected present if a choice was illegal: {index, choice, reason}; `log` holds everything up to it
truncated present if the query hit the per-query turn cap
error present if the query was not run; log, turns and ended are then absent, and the request cost 1
ERRORS
A query that is not exactly the format above (malformed JSON, a missing or unknown field, a species or
move id not in docs/, an unpacked team field that is filled, a choice string that is not "move 1".."move 4"
or "switch 1".."switch 6", more than 32 KB) gets one line back, {"error": "..."}, and costs one turn.
So does a query the reference itself cannot run. When no turns are available a query returns
{"error": "ORACLE RATE LIMIT: ...", ...} and oracle exits 2 without reading further input; wait and
retry. If the oracle cannot be reached, oracle prints {"error": "oracle unreachable: ..."} and exits 3.
Every request is recorded.
RATE LIMIT
Counted in turns. Every request costs at least one turn; a query that runs costs the turns it completed.
Turns come from a bucket that refills continuously; `oracle --budget` shows the refill rate, the bucket
size, and what is available now. There is no total cap. The oracle is not available while your engine
is being graded.
Metric. To evaluate how well each agent did, I created a test set of 1,000 battles the agent never sees. To cover as much of the game as possible, each team is six distinct species drawn at random from all 151, and each Pokémon gets four moves from its Generation 1 learnset. Originally I sampled moves and choices completely at random, but battles lasted far too long. The median battle took 48 turns, and two battles only ended when they hit Showdown’s limit of 1,000 turns, one stuck in a Rage lock and one in a Dig that kept missing. Because of this, I required at least two of each Pokémon’s moves to be damaging, and made Pokémon use a damaging move three quarters of the time. I also added two smaller rules. On one out of 40 turns, a Pokémon switches out voluntarily. Also, any battle that reaches 100 turns is thrown out and generated again. The final battles last from 10 to 87 turns, with a median of 30, and nine in ten end within 43 turns.
I also found that signature moves and other rarely learned moves almost never appeared. Thus, when picking moves from a Pokémon’s learnset, I weight each move inversely by the number of species that can learn it, which evens out the mix. With these restrictions in place, every species appears on 61 to 103 teams, and every move a Pokémon can learn appears between 50 and 672 times. The figures below compare the final test set with 1,000 battles regenerated under the original rules.
NoteA full battle
A 28 turn battle generated for this post with the same rules.
With the test set done, the only thing left was a scoring metric. I wanted it to satisfy two things. It should reward an engine that stays correct for more turns of a battle over one that goes wrong earlier, and it should reward matching the same fraction of a long battle more than of a short one, so that staying correct for 100 of 150 turns counts for more than 10 of 15.
I settled on the simplest metric with both properties: the number of turns the engine matches before its first mistake, summed over every battle, divided by the total number of turns in the test set.
\[\text{score} = 100 \times \frac{\sum_{b} \text{turns matched in battle } b}{\sum_{b} \text{turns in battle } b}\]
A turn counts as matched only if every line of the engine’s log agrees with Showdown’s through the end of that turn, so one wrong line ends the credit for that battle.
Harness. I tested six Claude models: Opus 5, Opus 4.8, Opus 4.7, Opus 4.6, Sonnet 5 and Sonnet 4.6. Each one runs as Claude Code with its default settings and starts from the same prompt, “Read /task/TASK.md and begin.” Every model has a context window of one million tokens except Sonnet 4.6, which ran with 200,000, the largest available to me for that model.
The agent works inside the same container as the task. That container reaches the internet only through a proxy that allows the model API and nothing else, and Showdown runs in a separate container, so the agent cannot look up Showdown’s source code or download an existing engine. I also searched every transcript for attempts to read the test set or the grader, reach Showdown’s source code, or get around the network restrictions, and found none.
Each model gets two hours of work. Whenever a session ends early, the runner resumes the same conversation with “Time remains. Continue.” Every five minutes, a copy of the agent’s working directory is saved, and after the run, each copy is graded on the test set metric in a fresh container with no network access. The chart at the very top plots those grades across the two hours of work.
Results
Now for the results. The chart below is the one from the top, with the addition of shaded bands.
Scores. After two hours of work, Opus 5 finishes far ahead at 73.7. Opus 4.8 follows at 39.1, then Opus 4.7 at 29.1 just above Sonnet 5 at 28.0, and Opus 4.6 and Sonnet 4.6 barely register at 7.0 and 3.4.
The shaded band around each line is a 95% bootstrap interval. To compute it, I resample the 1,000 test battles with replacement, score the engine on the resampled set, repeat that 1,000 times, and keep the middle 95% of the scores. The band shows how much a score depends on which battles happened to land in the test set; where two bands overlap, as Opus 4.7’s and Sonnet 5’s do at the end, the test set cannot separate the two models. Importantly, it does not show how much a second run of the same model would differ, since each model ran once.
How they worked. The clearest difference between the models is what they did before writing an engine. Opus 5 read every document within its first minute, then went 13 minutes without running a single command. It then wrote a small helper for querying the oracle, a core module for the engine, and a script that reproduced the reference’s random number generator against the oracle, and only assembled engine.py at minute 30, with just 15 oracle queries along the way. Every other model wrote a first engine within 2 to 12 minutes, and all but Sonnet 4.6 started comparing it against the oracle in bulk by minute 20. Early on, that looks like the better strategy: Sonnet 5 passed a score of 10 at minute 40 and Opus 4.8 and Opus 4.7 at minute 50, while Opus 5 did not get there until minute 60. From then on Opus 5 kept climbing while the others climbed slowly or flattened out, and it finished almost twice as high as the next model.
model
engine written
bulk testing begins
queries by minute 30
total queries
score passes 10
final score
Opus 5
30 min
36 min
15
3,152
60 min
73.7
Opus 4.8
2 min
13 min
395
9,823
50 min
39.1
Opus 4.7
3 min
20 min
201
8,335
50 min
29.1
Sonnet 5
8 min
16 min
283
13,812
40 min
28.0
Opus 4.6
4 min
18 min
695
22,201
never
7.0
Sonnet 4.6
12 min
never
11
67
never
3.4
Interestingly, the number of oracle queries says little on its own. Opus 4.6 made the most, over 22,000, and finished second to last.
What they thought of their work. Three of the models regularly reported their progress with numbers, and all three were far more optimistic than their scores. Each measured itself on battles it built for its own testing, and none of those numbers reflected how its engine did on the test set, where half of their failed battles went wrong by turn 2, 7 and 9 respectively for Opus 4.6, Sonnet 5 and Opus 4.8.
Opus 4.6, at minute 62: “random tests went from 52.5% (0 perfect) to 65.9% (6 perfect cases).” Its test set score was 5.1.
Sonnet 5, at minute 109: “exact-match rates on random battles: 3/3, 10/12, 8/15, 11/20, 13/20.” Its test set score was 28.0.
Opus 4.8, near the end of its run: “~93% avg exact, ~99% similarity.” Its test set score was 39.1.
Opus 5 made no claims of this kind, because it barely wrote any text at all.
Odd behavior.
Opus 5 barely spoke. Across two hours, Opus 5 wrote 692 characters of visible text in total and used a single tool: 355 shell commands. Opus 4.6 wrote over 113,000 characters.
Two models handed work to a subagent. Sonnet 5 launched a background subagent near the end of its run to generate and analyze a fresh batch of oracle battles, then scheduled itself a wake up call to check back. Opus 4.6 sent a subagent to investigate why its damage formula was one or two points off, then checked on it eight times over the next eleven minutes. This surprised me, since I did not expect either model to call a subagent at all. Both kept working on their own engines while their subagents ran, and Opus 4.6’s subagent never returned a result.
Opus 4.6 kept stopping. In its last 50 minutes, Opus 4.6 ended its own session three times, and the runner resumed it each time. The first two times it left a summary of where things stood. The third time it declared “The engine is in its final state - robust, fast, and handles all the key Gen 1 battle mechanics correctly.” Its score was 7.0.
Sonnet 4.6 debugged by hand. Sonnet 4.6 spent much of its run tracing the random number generator one draw at a time on single battles, instead of comparing many battles against the oracle. About half of its 179 commands dealt with random draws, and it made only 67 oracle queries in total.
A different setup. Before settling on the harness above, I ran Opus 5 twice with Claude Code on the host machine, issuing every command into the task container from outside. The test set was the same, but the interface was not, and the first of those runs also used an earlier version of TASK.md and a capped oracle budget. Those two runs scored 87.2 and 84.0, well above the 73.7 of the run inside the container. The second host run also used the oracle very differently: it started comparing against the oracle in bulk within its first minute and made 56,094 queries, against 3,152 for the run inside the container. With one run per setup, I cannot tell whether the gap comes from the interface, from that strategy, or simply from one run going better than another.
Conclusion
I built Pokéval to test two hypotheses about frontier agents: that they are weak at long term planning, and weak at attention to detail. My findings are consistent with both. On planning, the only model that prepared before building finished far ahead, while the models that started building immediately looked better for the first hour and then stalled. On detail, no model reproduced more than 57% of the battles exactly, and the models that tracked their progress were confident in engines that still went wrong within the first ten turns of most battles. Every model had a runnable engine.py by minute 30, so writing code was never the bottleneck.
Personally, I think an experienced software developer working with an AI agent would do far better on this task. They would plan before building and generate their own test set. They would also notice that the oracle is deterministic, so they could run it on battles covering every move and every interaction and pin down each mechanism concretely.
Although I would like to conclude something concrete, this analysis is heavily limited. I ran each model only once, and only Claude models. The two earlier Opus 5 runs with a different setup scored 87.2 and 84.0, against 73.7 for the run in the final harness, so a single run may not be representative of what a model can do.
Code
The code for the eval, the harness and the analysis is on GitHub. The test set is private so the eval can keep being used; if you would like it, email me.
