Not even Pokémon is safe from AI benchmarking controversy. A recent post on X claimed Google’s Gemini model outperformed Anthropic’s Claude model in the original Pokémon game, sparking debate over benchmarking methods.
Last week, a post on X went viral, claiming that Google’s latest Gemini model surpassed Anthropic’s flagship Claude model in the original Pokémon video game trilogy. Reportedly, Gemini had reached Lavender Town in a developer’s Twitch stream; Claude was stuck at Mount Moon as of late February. The post read, “Gemini is literally ahead of Claude atm in pokemon after reaching Lavender Town,” and included a screenshot of the stream with the comment, “119 live views only btw, incredibly underrated stream.”
However, it was later revealed that Gemini had an unfair advantage. Users on Reddit pointed out that the developer maintaining the Gemini stream had built a custom minimap that helps the model identify “tiles” in the game, such as cuttable trees. This custom minimap reduces the need for Gemini to analyze screenshots before making gameplay decisions, giving it a significant edge.
While Pokémon is considered, at best, a semi-serious AI benchmark, it serves as an instructive example of how different implementations of a benchmark can influence results. The controversy highlights the imperfections of AI benchmarking and how custom implementations can make it challenging to compare models accurately.
This issue is not unique to Pokémon. Anthropic reported two different scores for its Claude 3.7 Sonnet model on the SWE-bench Verified benchmark, which evaluates a model’s coding abilities. Without a “custom scaffold,” Claude 3.7 Sonnet achieved 62.3% accuracy, but with the custom scaffold, the accuracy increased to 70.3%. Similarly, Meta fine-tuned a version of its Llama 4 Maverick model to perform better on the LM Arena benchmark. The fine-tuned version scored significantly higher than the vanilla version on the same evaluation.
Given that AI benchmarks are imperfect measures to begin with, custom and non-standard implementations further complicate the comparison of models. As a result, it is likely to become increasingly difficult to compare models as they are released.
