{"id":135113,"date":"2026-07-14T08:08:46","date_gmt":"2026-07-14T08:08:46","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2026\/07\/14\/anthropics-extravagant-tokenizer-complicates-ai-pricing\/"},"modified":"2026-07-14T08:09:46","modified_gmt":"2026-07-14T08:09:46","slug":"anthropics-extravagant-tokenizer-complicates-ai-pricing","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2026\/07\/14\/anthropics-extravagant-tokenizer-complicates-ai-pricing\/","title":{"rendered":"Anthropic’s extravagant tokenizer complicates AI pricing"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

AI and ML<\/p>\n

Token consumption would not inform the entire story however it should not be ignored<\/p>\n<\/div>\n

\n

Claude appears to be like considerably extra token-hungry than OpenAI’s GPT-5.x, because of the brand new tokenizer that Anthropic shipped with current releases.<\/p>\n

Giant language fashions (LLMs) use tokenizers to deal with the mapping of textual content into tokens<\/a>. There is no set definition of a token, however they’re usually a set of three or 4 characters which are mapped to the integers LLMs truly course of.\u00a0<\/p>\n

Tokens have change into the essential financial unit for billing use of AI fashions. As a result of the slicing of phrases into tokens and the tokens required per process differ throughout fashions, it has change into fairly tough to foretell the ultimate invoice for enjoying the AI slot machine.<\/p>\n

Current modifications to Anthropic’s tokenizer seem to have additional sophisticated issues by making the identical content material extra expensive to course of on sure fashions.<\/p>\n

Playcode, an AI app constructing platform, lately analyzed<\/a> the influence of Anthropic’s newest tokenizer and located that the identical TypeScript file processed by Claude can eat as much as 73 % extra tokens than OpenAI’s GPT-5.x mannequin household.<\/p>\n

Anthropic acknowledges its new tokenizer \u2013 announced<\/a>\u00a0on the finish of June when Sonnet 5 shipped \u2013 could generate extra tokens for a similar enter than prior variations.<\/p>\n

“Sonnet 5 is an improve to Sonnet 4.6, however it makes use of an up to date tokenizer that modifications how the mannequin processes textual content to enhance efficiency (that is just like the tokenizer change we launched with Claude Opus 4.7),” the corporate defined. “The tradeoff is that the identical enter can map to extra tokens: roughly 1.0\u20131.35\u00d7 relying on the content material kind.”<\/p>\n

Anthropic supplied Sonnet at a lowered introductory charge \u2013 $2\/million enter tokens and $10\/million output tokens by August 31, 2026 \u2013 to make the inflated token technology roughly cost-neutral. However the value is about to rise to $3\/million and $15\/millionafter that.<\/p>\n

Anthropic didn’t instantly reply to a request for remark. The corporate says that customers of its new tokenizer may even see their payments rise by as a lot as a 3rd in comparison with the tokenizer it used\u00a0with its older fashions. Prices for Anthropic customers may go even larger in comparison towards the newest iteration of OpenAI’s o200k<\/a> <\/a>tokenizer<\/a>.<\/p>\n

Playcode’s cross-vendor token comparability finds that for a 2,888 character TypeScript file, Claude’s new tokenizer emits 1.73x extra tokens than GPT-5.x’s tokenizer and 1.32x greater than Claude’s outdated tokenizer. These figures differ for various kinds of code: Rust faucets in at 1.58x, JavaScript 1.52x, and Python 1.50x.<\/p>\n

So if Anthropic’s record costs have been adjusted to be comparable with OpenAI’s GPT-5.x baseline, Playcode suggests Opus 4.8’s price can be $7.50\/M enter and $37.50\/M output as a substitute of the revealed determine of $5\/M and $25\/M.<\/p>\n

The AI app platform notes {that a} crew from advertising and marketing platform Ploy this week revealed an account of a manufacturing migration utilizing OpenAI’s GPT-5.6 Sol and Anthropic’s Opus 4.8.<\/p>\n

“GPT-5.6 completed pages 2.2\u00d7 sooner, price 27 % much less, and used about half the output tokens,” Ploy cla<\/a>i<\/a>med<\/a>.\u00a0<\/p>\n

There are different elements that go into calculating AI payments. As we noted recently<\/a>, prices must be judged when it comes to process completion and the influence of mannequin harnesses (e.g. Claude Code, Codex, Pi, OpenCode, and many others.) also needs to be evaluated when trying to calculate the price of operating AI workloads. \u00ae<\/p>\n<\/div>\n