Tech and AIThese researchers used NPR Sunday Puzzle questions to benchmark...

These researchers used NPR Sunday Puzzle questions to benchmark AI ‘reasoning’ models

-


Every Sunday, NPR host Will Shortz, The New York Times’ crossword puzzle guru, gets to quiz thousands of listeners in a long-running segment called the Sunday Puzzle. While written to be solvable without too much foreknowledge, the brainteasers are usually challenging even for skilled contestants.

That’s why some experts think they’re a promising way to test the limits of AI’s problem-solving abilities.

In a recent study, a team of researchers hailing from Wellesley College, Oberlin College, the University of Texas at Austin, Northeastern University, Charles University, and startup Cursor created an AI benchmark using riddles from Sunday Puzzle episodes. The team says their test uncovered surprising insights, like that reasoning models — OpenAI’s o1, among others — sometimes “give up” and provide answers they know aren’t correct.

“We wanted to develop a benchmark with problems that humans can understand with only general knowledge,” Arjun Guha, a computer science faculty member at Northeastern and one of the co-authors on the study, told TechCrunch.

The AI industry is in a bit of a benchmarking quandary at the moment. Most of the tests commonly used to evaluate AI models probe for skills, like competency on PhD-level math and science questions, that aren’t relevant to the average user. Meanwhile, many benchmarks — even benchmarks released relatively recently — are quickly approaching the saturation point.

The advantages of a public radio quiz game like the Sunday Puzzle is that it doesn’t test for esoteric knowledge, and the challenges are phrased such that models can’t draw on “rote memory” to solve them, explained Guha.

“I think what makes these problems hard is that it’s really difficult to make meaningful progress on a problem until you solve it — that’s when everything clicks together all at once,” Guha said. “That requires a combination of insight and a process of elimination.”

No benchmark is perfect, of course. The Sunday Puzzle is U.S. centric and English only. And because the quizzes are publicly available, it’s possible that models trained on them can “cheat” in a sense, although Guha says he hasn’t seen evidence of this.

“New questions are released every week, and we can expect the latest questions to be truly unseen,” he added. “We intend to keep the benchmark fresh and track how model performance changes over time.”

On the researchers’ benchmark, which consists of around 600 Sunday Puzzle riddles, reasoning models such as o1 and DeepSeek’s R1 far outperform the rest. Reasoning models thoroughly fact-check themselves before giving out results, which helps them avoid some of the pitfalls that normally trip up AI models. The trade-off is that reasoning models take a little longer to arrive at solutions — typically seconds to minutes longer.

At least one model, DeepSeek’s R1, gives solutions it knows to be wrong for some of the Sunday Puzzle questions. R1 will state verbatim “I give up,” followed by an incorrect answer chosen seemingly at random — behavior this human can certainly relate to.

The models make other bizarre choices, like giving a wrong answer only to immediately retract it, attempt to tease out a better one, and fail again. They also get stuck “thinking” forever and give nonsensical explanations for answers, or they arrive at a correct answer right away but then go on to consider alternative answers for no obvious reason.

“On hard problems, R1 literally says that it’s getting ‘frustrated,’” Guha said. “It was funny to see how a model emulates what a human might say. It remains to be seen how ‘frustration’ in reasoning can affect the quality of model results.”

NPR benchmark
R1 getting “frustrated” on a question in the Sunday Puzzle challenge set.Image Credits:Guha et al.

The current best-performing model on the benchmark is o1 with a score of 59%, followed by the recently released o3-mini set to high “reasoning effort” (47%). (R1 scored 35%.) As a next step, the researchers plan to broaden their testing to additional reasoning models, which they hope will help to identify areas where these models might be enhanced.

NPR benchmark
The scores of the models the team tested on their benchmark.Image Credits:Guha et al.

“You don’t need a PhD to be good at reasoning, so it should be possible to design reasoning benchmarks that don’t require PhD-level knowledge,” Guha said. “A benchmark with broader access allows a wider set of researchers to comprehend and analyze the results, which may in turn lead to better solutions in the future. Furthermore, as state-of-the-art models are increasingly deployed in settings that affect everyone, we believe everyone should be able to intuit what these models are — and aren’t — capable of.”



Source link

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest news

‘AI’ crypto trading agent, aixbt, hacked for $100K

Purported artificial intelligence cryptocurrency trading agent, aixbt, has been hacked for 55.5 ETH, worth approximately $100K. Source link

Google brings a ‘canvas’ feature to Gemini, plus Audio Overview

They say imitation is the sincerest form of flattery, and Google seems to agree. On Tuesday, the company added...

Now Binance Allows Dex Tokens Trading On Cex Without Switching: Alpha 2.0

Binance crypto exchange announced a very big brilliant move with Binance Alpha 2.0 to push the adoption of...

Interchain Foundation funds Tharsis Labs to open-source evmOS

The Interchain Foundation is funding the...

Advertisement

Soneium Partners with LINE to Integrate Web3 Mini-Apps

Want more? Connect with NFT PlazasJoin the Weekly NewsletterJoin our TelegramFollow us on XLike us on Facebook*All...

Features, Pricing, Security, Pros & Cons

Dashlane fast factsOur rating: 4.4 stars out of 5Pricing: Starts at $4.99 per monthKey featuresSecure password vault encryption.No...

Must read

‘AI’ crypto trading agent, aixbt, hacked for $100K

Purported artificial intelligence cryptocurrency trading agent, aixbt, has...

Google brings a ‘canvas’ feature to Gemini, plus Audio Overview

They say imitation is the sincerest form of...

You might also likeRELATED
Recommended to you