Tech and AIThese researchers used NPR Sunday Puzzle questions to benchmark...

These researchers used NPR Sunday Puzzle questions to benchmark AI ‘reasoning’ models

-


Every Sunday, NPR host Will Shortz, The New York Times’ crossword puzzle guru, gets to quiz thousands of listeners in a long-running segment called the Sunday Puzzle. While written to be solvable without too much foreknowledge, the brainteasers are usually challenging even for skilled contestants.

That’s why some experts think they’re a promising way to test the limits of AI’s problem-solving abilities.

In a recent study, a team of researchers hailing from Wellesley College, Oberlin College, the University of Texas at Austin, Northeastern University, Charles University, and startup Cursor created an AI benchmark using riddles from Sunday Puzzle episodes. The team says their test uncovered surprising insights, like that reasoning models — OpenAI’s o1, among others — sometimes “give up” and provide answers they know aren’t correct.

“We wanted to develop a benchmark with problems that humans can understand with only general knowledge,” Arjun Guha, a computer science faculty member at Northeastern and one of the co-authors on the study, told TechCrunch.

The AI industry is in a bit of a benchmarking quandary at the moment. Most of the tests commonly used to evaluate AI models probe for skills, like competency on PhD-level math and science questions, that aren’t relevant to the average user. Meanwhile, many benchmarks — even benchmarks released relatively recently — are quickly approaching the saturation point.

The advantages of a public radio quiz game like the Sunday Puzzle is that it doesn’t test for esoteric knowledge, and the challenges are phrased such that models can’t draw on “rote memory” to solve them, explained Guha.

“I think what makes these problems hard is that it’s really difficult to make meaningful progress on a problem until you solve it — that’s when everything clicks together all at once,” Guha said. “That requires a combination of insight and a process of elimination.”

No benchmark is perfect, of course. The Sunday Puzzle is U.S. centric and English only. And because the quizzes are publicly available, it’s possible that models trained on them can “cheat” in a sense, although Guha says he hasn’t seen evidence of this.

“New questions are released every week, and we can expect the latest questions to be truly unseen,” he added. “We intend to keep the benchmark fresh and track how model performance changes over time.”

On the researchers’ benchmark, which consists of around 600 Sunday Puzzle riddles, reasoning models such as o1 and DeepSeek’s R1 far outperform the rest. Reasoning models thoroughly fact-check themselves before giving out results, which helps them avoid some of the pitfalls that normally trip up AI models. The trade-off is that reasoning models take a little longer to arrive at solutions — typically seconds to minutes longer.

At least one model, DeepSeek’s R1, gives solutions it knows to be wrong for some of the Sunday Puzzle questions. R1 will state verbatim “I give up,” followed by an incorrect answer chosen seemingly at random — behavior this human can certainly relate to.

The models make other bizarre choices, like giving a wrong answer only to immediately retract it, attempt to tease out a better one, and fail again. They also get stuck “thinking” forever and give nonsensical explanations for answers, or they arrive at a correct answer right away but then go on to consider alternative answers for no obvious reason.

“On hard problems, R1 literally says that it’s getting ‘frustrated,’” Guha said. “It was funny to see how a model emulates what a human might say. It remains to be seen how ‘frustration’ in reasoning can affect the quality of model results.”

NPR benchmark
R1 getting “frustrated” on a question in the Sunday Puzzle challenge set.Image Credits:Guha et al.

The current best-performing model on the benchmark is o1 with a score of 59%, followed by the recently released o3-mini set to high “reasoning effort” (47%). (R1 scored 35%.) As a next step, the researchers plan to broaden their testing to additional reasoning models, which they hope will help to identify areas where these models might be enhanced.

NPR benchmark
The scores of the models the team tested on their benchmark.Image Credits:Guha et al.

“You don’t need a PhD to be good at reasoning, so it should be possible to design reasoning benchmarks that don’t require PhD-level knowledge,” Guha said. “A benchmark with broader access allows a wider set of researchers to comprehend and analyze the results, which may in turn lead to better solutions in the future. Furthermore, as state-of-the-art models are increasingly deployed in settings that affect everyone, we believe everyone should be able to intuit what these models are — and aren’t — capable of.”



Source link

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest news

Canada Crypto Fund Are Exploding: Is Canada Bitcoin Reserve Next After Trudeau?

Canada crypto markets are heating up as interest in digital assets grows. Talks of a Canada Bitcoin reserve...

Binance memecoin platform Four Meme exploited again — this time for $130K

Four Meme suspended its launch function and conducted an emergency probe after suffering an “attack” today. Source link

How to Protect Your Cats (and Backyard Chickens) From Bird Flu

For cats who enjoy spending time outside, Feah says that leashed walks are a good option. She also...

Bank of Korea | No Plans for a Bitcoin Reserve

The Bank of Korea (BoK) has ruled out the possibility of including bitcoin in its foreign exchange reserves....

Advertisement

Vitalik Buterin Sells 5K DHN Tokens for $124K, Triggering 50% Price Drop

Vitalik Buterin sold 5,000 Dohrnii (DHN) tokens, which he had received for free from Dohrnii Labs, for $124,000,...

Peer Raises $10.5M to Expand AI-Powered Metaverse Engine

Peer Global Inc. has raised $10.5 million in a new funding round, increasing its total investment to $65.5...

Must read

Canada Crypto Fund Are Exploding: Is Canada Bitcoin Reserve Next After Trudeau?

Canada crypto markets are heating up as interest...

Binance memecoin platform Four Meme exploited again — this time for $130K

Four Meme suspended its launch function and conducted...

You might also likeRELATED
Recommended to you