AI models just don’t understand what they’re talking about • The Register

Researchers from MIT, Harvard, and the College of Chicago have proposed the time period “potemkin understanding” to explain a newly recognized failure mode in giant language fashions that ace conceptual benchmarks however lack the true grasp wanted to use these ideas in apply.

It comes from accounts of pretend villages – Potemkin villages – constructed on the behest of Russian navy chief Grigory Potemkin to impress Empress Catherine II.

The lecturers are differentiating “potemkins” from “hallucination,” which is used to explain AI mannequin errors or mispredictions. In truth, there’s extra to AI incompetence than factual errors; AI fashions lack the power to know ideas the best way folks do, a bent urged by the extensively used disparaging epithet for big language fashions, “stochastic parrots.”

Laptop scientists Marina Mancoridis, Bec Weeks, Keyon Vafa, and Sendhil Mullainathan counsel the time period “potemkin understanding” to explain when a mannequin succeeds at a benchmark check with out understanding the related ideas.

“Potemkins are to conceptual data what hallucinations are to factual data – hallucinations fabricate false details; potemkins fabricate false conceptual coherence,” the authors clarify of their preprint paper, “Potemkin Understanding in Massive Language Fashions.”

The paper is scheduled to be offered later this month at ICML 2025, the Worldwide Convention on Machine Studying.

Keyon Vafa, a postdoctoral fellow at Harvard College and one of many paper’s co-authors, advised The Register in an electronic mail that the selection of the time period “potemkin understanding” represented a deliberate effort to keep away from anthropomorphizing or humanizing AI fashions.

Here is one instance of “potemkin understanding” cited within the paper. Requested to clarify the ABAB rhyming scheme, OpenAI’s GPT-4o did so precisely, responding, “An ABAB scheme alternates rhymes: first and third strains rhyme, second and fourth rhyme.”

But when requested to offer a clean phrase in a four-line poem utilizing the ABAB rhyming scheme, the mannequin responded with a phrase that did not rhyme appropriately. In different phrases, the mannequin accurately predicted the tokens to clarify the ABAB rhyme scheme with out the understanding it could have wanted to breed it.

The issue with potemkins in AI fashions is that they invalidate benchmarks, the researchers argue. The aim of benchmark checks for AI fashions is to counsel broader competence. But when the check solely measures check efficiency and never the capability to use mannequin coaching past the check situation, it would not have a lot worth.

If LLMs can get the fitting solutions with out real understanding, then benchmark success turns into deceptive

As noted by Sarah Gooding from safety agency Socket, “If LLMs can get the fitting solutions with out real understanding, then benchmark success turns into deceptive.”

As we have famous, AI benchmarks have many problems, and AI firms may try to game them.

So the researchers developed benchmarks of their very own to evaluate the prevalence of potemkins, and so they transform “ubiquitous” within the fashions examined – Llama-3.3 (70B), GPT-4o, Gemini-2.0 (Flash), Claude 3.5 (Sonnet), DeepSeek-V3, DeepSeek-R1, and Qwen2-VL (72B).

One check centered on literary methods, recreation idea, and psychological biases. It discovered that whereas the fashions evaluated can establish ideas more often than not (94.2 %), they regularly failed when requested to categorise idea cases (a median of 55 % failure fee), to generate examples (40 %), and to edit idea cases (40 %).

As with the beforehand famous ABAB rhyming blunder, the fashions may reliably clarify the literary methods evident in a Shakespearean sonnet, however about half the time had hassle recognizing, reproducing, or enhancing a sonnet.

“The existence of potemkins implies that conduct that may signify understanding in people doesn’t signify understanding in LLMs,” stated Vafa. “This implies we both want new methods to check LLMs past having them reply the identical questions used to check people or discover methods to take away this conduct from LLMs.”

Doing so can be a step towards synthetic common intelligence or AGI. It is likely to be some time. ®