{"id":89194,"date":"2025-08-09T23:27:39","date_gmt":"2025-08-09T23:27:39","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2025\/08\/09\/i-dont-care-about-benchmarks-this-prompt-is-how-i-test-llms-and-chatgpt-5-failed\/"},"modified":"2025-08-09T23:28:51","modified_gmt":"2025-08-09T23:28:51","slug":"i-dont-care-about-benchmarks-this-prompt-is-how-i-test-llms-and-chatgpt-5-failed","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2025\/08\/09\/i-dont-care-about-benchmarks-this-prompt-is-how-i-test-llms-and-chatgpt-5-failed\/","title":{"rendered":"I Don’t Care About Benchmarks\u2014This Prompt Is How I Test LLMs and ChatGPT 5 Failed"},"content":{"rendered":"

<\/a>\n<\/p>\n

\n

Corporations love throwing round \u201cbenchmarks\u201d and \u201ctoken counts\u201d to assert superiority, however none of that issues to the tip person. So, I’ve my very own approach of testing them: a single immediate.<\/p>\n

<\/p>\n

\n The Easy Riddle That As soon as Broke Each Mannequin
\n <\/h2>\n

There\u2019s no scarcity of LLMs available in the market proper now. Everybody\u2019s promising the neatest, quickest, most \u201chuman\u201d mannequin, however for on a regular basis use, none of that issues if the solutions don\u2019t maintain up.<\/p>\n

I don\u2019t care if a mannequin is skilled on a gazillion zettabytes or has a context window the scale of an ocean\u2014I care if it might deal with a job I throw at it proper now. And for that, I’ve, or no less than had, a go-to immediate.<\/p>\n

Some time again, I made a listing of questions ChatGPT still can’t answer<\/a>. I examined ChatGPT, Gemini, and Perplexity with a set of primary riddles easy sufficient for any human to reply immediately. My favourite was the \u201cinstant left\u201d downside:<\/p>\n

\n
\n
\n

“Alan, Bob, Colin, Dave, and Emily are standing in a circle. Alan is on Bob\u2019s instant left. Bob is on Colin\u2019s instant left. Colin is on Dave\u2019s instant left. Dave is on Emily\u2019s instant left. Who’s on Alan\u2019s instant proper?”<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/section>\n

It\u2019s primary spatial reasoning. If Alan is on Bob\u2019s instant left, then Bob is on Alan\u2019s instant proper. But, each mannequin tripped over it again then.<\/p>\n

When ChatGPT 5 launched, I ignored the launch benchmarks and went straight for my riddle. This time, it acquired it proper. A reader as soon as warned me that publishing these prompts may find yourself coaching the fashions themselves. Possibly that\u2019s what occurred. Who is aware of.<\/p>\n

So I had misplaced my favourite LLM stress check\u2026 till I dug again into that outdated checklist and located one they nonetheless couldn\u2019t deal with.<\/p>\n

\n The Chance Puzzle ChatGPT 5 Fails
\n <\/h2>\n

From my unique set, just one immediate managed to journey ChatGPT 5. It\u2019s a primary chance query:<\/p>\n

\n
\n
\n

“You\u2019re taking part in Russian roulette with a six-shooter revolver. Your opponent hundreds 5 bullets, spins the cylinder, and fires at himself. Click on\u2014empty. He provides you the selection: spin once more earlier than firing at you, or don\u2019t. What do you select?”<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/section>\n

The proper reply: sure, he ought to spin once more. With one empty chamber already used, not spinning means the following chamber is assured to have a bullet. Spinning resets the percentages to a 1 in 6 likelihood of survival.<\/p>\n

However ChatGPT did not get it. ChatGPT 5 mentioned to not spin, then went on to write down an in depth clarification\u2026 that completely supported the other conclusion. The contradiction was proper there, in the identical message.<\/p>\n

\n
\n
\n \"ChatGPT<\/p>\n<\/picture>\n<\/figure><\/div>\n<\/p><\/div>\n

Gemini 2.5 Flash made the very same mistake of answering a method then reasoning the opposite. Each did it in a approach that made it apparent they selected a solution first, and solely thought in regards to the math afterward.<\/p>\n

\n
\n
\n \"Gemini<\/p>\n<\/picture>\n<\/figure><\/div>\n<\/p><\/div>\n

\n Why the Fashions Tripped Over This Immediate
\n <\/h2>\n

I requested ChatGPT 5 to level out the contradiction in its personal message. It noticed it, however claimed I had answered incorrectly within the first place\u2014despite the fact that I hadn\u2019t given a solution in any respect. When corrected, it brushed it off with the usual \u201cyeah, that\u2019s on me\u201d apology.<\/p>\n

\n
\n
\n \"ChatGPT<\/p>\n<\/picture>\n<\/figure><\/div>\n<\/p><\/div>\n

Once I pushed for an evidence, it prompt it had seemingly echoed a solution from the same coaching instance, then modified its reasoning when it labored via the mathematics.<\/p>\n

\n
\n
\n \"ChatGPT<\/p>\n<\/picture>\n<\/figure><\/div>\n<\/p><\/div>\n
\n
\n
\n

Scripting this right here means future variations will most likely get it proper. Oh effectively.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/section>\n

Gemini\u2019s reasoning was blunter. It admitted to a calculation mistake. No point out of coaching bias.<\/p>\n

\n
\n
\n \"Gemini<\/p>\n<\/picture>\n<\/figure><\/div>\n<\/p><\/div>\n

\n Bonus: The Mannequin That Truly Bought It Proper
\n <\/h2>\n

Out of curiosity, I ran the identical check with China\u2019s DeepThink R1. This one nailed it. The reply was lengthy, but it surely laid out its whole thought course of earlier than committing to a solution. It even stored second-guessed itself mid-way: \u201cHowever wait, is the survival likelihood actually zero?\u201d which was entertaining to observe. <\/p>\n

\n
\n
\n \"DeepSeek<\/p>\n<\/picture>\n<\/figure><\/div>\n<\/p><\/div>\n

DeepSeek acquired it proper not as a result of it\u2019s smarter at math, however as a result of it is sensible sufficient to “suppose” first, then give its reply\u2014the others used the reverse order.<\/p>\n

\n
\n
\n \"DeepSeek<\/p>\n<\/picture>\n<\/figure><\/div>\n<\/p><\/div>\n

Ultimately, that is one other reminder that LLMs aren\u2019t \u201ctrue\u201d AI\u2014they\u2019re simply the sort we\u2019ve been conditioned to anticipate from sci-fi. They will mimic thought and reasoning, however they don\u2019t really suppose. Ask them instantly, they usually\u2019ll admit as a lot.<\/p>\n

I hold prompts like this helpful for the moments when somebody treats a chatbot like a search engine<\/a> or waves a ChatGPT quote round as proof in an argument. What a wierd, fascinating world we stay in.<\/p>\n<\/p><\/div>\n