Google’s most expensive AI model seems to have crossed a major milestone: Beating a 29-year-old video game.
Last night, Google CEO Sundar Pichai posted triumphantly on X, “What a finish! Gemini 2.5 Pro just completed Pokémon Blue!”
To be clear, the Gemini Plays Pokemon livestream was created by (in his own words) “a 30 year old software engineer unaffiliated with Google” who goes by Joel Z. But Google executives have been cheering the effort on.
For example, Logan Kilpatrick, the product lead for Google AI Studio, posted last month that Gemini was “making great progress at completing Pokémon” and had “earned its 5th badge (next best model only has 3 so far, though with a different agent harness),” leading Pichai to joke, “We are working on API, Artificial Pokémon Intelligence:)”
Why Pokémon? Back in February, Anthropic highlighted progress that its Claude AI models were making in “Pokémon Red,” writing that Claude’s “extended thinking and agent training” gives it “a major boost” on “more unexpected” tasks, like playing a classic game. (“Pokémon Red” and “Blue” are different versions of a GameBoy title first released in 1996 and tied to the long-running Pokémon franchise). There’s even a Claude Plays Pokemon Twitch channel that Joel Z cited as an inspiration.
Despite its progress, Claude does not appear to have beaten “Pokémon Red” yet. Does that mean Gemini is objectively better at the game? On his Twitch page, Joel Z urged viewers, “Please don’t consider this a benchmark for how well an LLM can play Pokemon. You can’t really make direct comparisons — Gemini and Claude have different tools and receive different information.”
And both AI models need help to play the game — that’s where the aforementioned agent harnesses come in, providing the models with game screenshots overlaid with additional information, allowing the model to decide how to respond (which may involve calling specialized agents), and then pressing the button that corresponds with the AI’s instruction.
Techcrunch event
Berkeley, CA
|
June 5
Joel Z acknowledged that there were other “dev interventions” to help Gemini complete the game, but insisted that it’s not cheating.
“My interventions improve Gemini’s overall decision-making and reasoning abilities,” he says. “I don’t give specific hints — there are no walkthroughs or direct instructions for particular challenges like Mt. Moon. The only thing that comes even close is letting Gemini know that it needs to talk to a Rocket Grunt twice to obtain the Lift Key, which was a bug that was later fixed in Pokemon Yellow.”
Plus, he said, “Gemini Plays Pokémon is still actively being developed, and the framework continues to evolve.”
