I bought a laptop with a dedicated NPU, but it’s actually completely useless

Microsoft’s aggressive push in the direction of integrating AI into Home windows got here with a small {hardware} addition as properly, the Neural Processing Unit or NPU. In case you’ve regarded for a brand new laptop computer currently and wondered what TOPS is, it is the measure of the built-in NPU’s efficiency.

Microsoft desires you to imagine you can purchase a PC with an NPU at the very least 40 TOPS robust, so you need to use all their flashy AI options. Properly, I purchased one, and two years later, it is nonetheless sitting unused in my laptop computer.

Microsoft bought me an AI future, however this isn’t it

Copilot+, NPUs, and the promise of AI PCs

On paper, the pitch is definitely fairly compelling. Microsoft and its {hardware} companions claimed that with an NPU, your laptop computer turns into a non-public native AI machine, operating LLMs, transcribing speech, making use of real-time video results, and extra, all with out sending your knowledge to the cloud for processing.

The massive promise was on-device intelligence made accessible and environment friendly by the NPU. The NPU crunches AI workloads whereas the CPU and GPU keep free. It extends battery life and frees up CPU and GPU sources for different purposes you would possibly want them for.

Recall was the flagship Home windows characteristic that was supposed to make use of the NPU, letting you search by your desktop exercise utilizing pure language. Different Copilot+ methods like semantic file search, stay captions with real-time translations, webcam background blur and eye-contact correction, and even AI-powered picture tweaks within the Pictures app made for a stable pitch for wanting an NPU in your subsequent laptop computer.

The NPU push has been a coordinated effort between Microsoft, Intel, AMD, Qualcomm, and different main OEMs. Microsoft wanted {hardware} to go up in opposition to Apple’s M-series chips, which have built-in neural engines for years and use them successfully. Intel and AMD wanted a differentiator past uncooked CPU clock speeds, and Qualcomm wanted the Snapdragon X collection to appear to be extra than simply an ARM laptop computer chip.

The outcome was a advertising blitz that outlined AI PCs as greater than only a spec bump. It was an entire new manner of utilizing a PC, or at the very least that was the dream Microsoft bought.

Right here’s what having an NPU truly looks like

More often than not, it simply sits there

Intel debuted NPUs in shopper laptops with its Meteor Lake line of processors in 2024. Since then, local LLMs have come a long way and are great at certain tasks, however hottest apps, corresponding to LM Studio or Ollama, nonetheless default to the GPU or CPU. The NPU merely sits there, idle.

Recall, the supposed crown jewel for the NPU, turned a PR catastrophe virtually instantly after launch. Aside from being the privacy nightmare that Recall was, it confronted repeated delays and controversies. To make issues worse, researchers discovered methods to allow the characteristic on older {hardware} with out an NPU by unofficial workarounds.

The remainder of the Copilot+ expertise was extra practical, however modest at greatest. Home windows Studio Results—the webcam background blur and eye-contact correction—work, however your mileage might range. Reside Captions with real-time translation could be genuinely helpful. And semantic search makes discovering information much less painful, though you would be a lot better off replacing Windows Search with the Command Palette.

OS

Home windows, macOS, Linux

Developer

Aspect Labs

Value mannequin

Free

A free desktop app that allows you to obtain, run, and chat with giant language fashions domestically, no cloud required.

The actual downside isn’t the {hardware}, it’s the software program

Builders aren’t constructing for it (but)

An NPU solely helps when packages are explicitly written to dump workloads to it. Something from instruments to run native LLMs to inventive software program, virtually nobody is utilizing built-in NPUs on Home windows machines for AI workloads. The specialised chip structure means builders have to put in writing code particularly concentrating on the NPU. It isn’t one thing apps can fall again on mechanically.

Intel itself acknowledged throughout technical periods at CES 2024 that on Meteor Lake, essentially the most demanding AI workloads are nonetheless offloaded to the built-in Arc GPU. The NPU is supposed for dealing with steady, low-intensity duties like background blur. Digital Trends examined the ability distinction between operating background blur on the GPU versus the NPU and located an virtually negligible hole, with the GPU averaging 18.9 W and the NPU coming in at 17.6W over half-hour—not precisely the numerous effectivity acquire advertising implied.

Microsoft has since tried to attenuate the injury. It launched Home windows ML, a framework that lets builders write AI apps concentrating on CPUs, GPUs, and NPUs, reasonably than the NPU completely.

Is the NPU totally ineffective?

The few options that really profit

The NPU in your laptop computer is not zero worth. In case you’re consistently on video calls all through the day, the on-device Studio results can assist. As talked about earlier than, stay captions and semantic search are additionally helpful options, however different software-based options do not require you to pay a premium price ticket when shopping for a laptop computer. So, if any of those use circumstances entice you, the NPU has some fundamental use.

There’s additionally a believable future-proofing argument right here. Software program ecosystems are likely to meet up with {hardware} finally. As tooling improves and extra fashions are quantized and tuned for NPUs, that idle silicon would possibly finally begin seeing some use.

The issue, nonetheless, is timing. Sure, software program would possibly finally begin utilizing the NPU, however finally is not what I paid for. I’ve had my HP Omen Transcend 14 for nearly two years now, and the NPU has barely seen any use.

Don’t purchase a laptop computer for the NPU

It is good to have, however not price a worth bounce simply but

In case you already personal an NPU-equipped laptop computer, you needn’t remorse it outright. Deal with the NPU as an overpowered background blur engine, which could simply be a helpful AI accelerator tomorrow. Maintain an eye fixed out for a way native AI tooling evolves, as a result of there is a good probability it will definitely begins utilizing the NPU.

Associated

4 Reasons I Simply Don’t Care About AI PCs

Huge tech corporations have hyped AI PCs as the subsequent large factor, however I will not splurge on one anytime quickly.

In case you’re about to purchase a laptop computer and the advertising will get too flashy, keep in mind that the hefty worth bounce on the premise of a devoted NPU is you paying for an costly block of silicon that is doing virtually nothing in the intervening time. It would sooner or later, however till then, you will be paying for a characteristic that hasn’t arrived but. In concept, your laptop computer has a devoted mind for an AI; in apply, it is simply an costly background blur button.