In-Depth
Why the Future of AI May Be Hybrid
For many years, the software industry has worked aggressively toward a world where computing is cloud-only and subscription-only. Its vision has largely involved cloud desktops and traditional software replaced by software-as-a-service applications. Under this model, hardware becomes a commodity and subscriptions replace ownership.
Early on, some suggested that we were entering the "post-PC era." The idea was that everything would be done in the cloud and PCs would be replaced by terminals. Of course, PCs never went away because cloud services simply are not a good fit for every use case.
Despite the industry's aggressive push toward the cloud, there are signs that it may be beginning to recognize that local hardware resources are more useful than once assumed. In fact, there is strong evidence to suggest that hybrid AI inference will become far more common.
Before explaining what that hybrid architecture might look like, let me step back and offer a personal example of why hybrid AI makes sense.
The PC I use for most of my work has nearly 200 GB of RAM, an Intel Core i9 CPU and an NVIDIA 5090 GPU. Recently, I have been working on a personal project and making extensive use of Microsoft 365 Copilot in Microsoft Word. Copilot is getting the job done, but when I enter an editing prompt, it takes a while to fulfill my request. Part of the reason for the lag is that Copilot does all its inference in the cloud. I am using a high-end PC, but Copilot is not taking advantage of the available hardware resources.
On the surface, this observation might seem less like an argument for hybrid AI inference and more like a wish to take advantage of a high-end PC. But there is more to the story. Consider the economics of Microsoft 365 Copilot.
Microsoft has several Copilot subscription plans, but the important thing to keep in mind is that they are generally billed at a flat rate. For example, the Microsoft 365 Copilot Business add-on currently costs $21 per user per month when paid annually. Subscribers pay a consistent rate whether they use Copilot once a week or all day, every day. This contrasts with Microsoft Azure, where customers are generally billed based on the cloud resources they consume.
Consider this pricing model from Microsoft's standpoint. Two customers could pay exactly the same amount, with one generating far more inference tokens than the other. Assuming Microsoft continues to rely heavily on subscription pricing, it could conceivably improve profitability by reducing the inference processing its data centers have to perform. Offloading certain tasks to local hardware could potentially help Microsoft accomplish this.
This is where hybrid inference comes into play. The idea is that not all inference needs to happen in the cloud. Some tasks can run on local PC hardware. For end users, this type of hybrid model could potentially reduce the lag associated with AI tasks, improving performance. For a company like Microsoft, hybrid inference would mean less load on its data center hardware, translating to lower operating costs.
Hybrid inference has compelling benefits, but it is impossible to ignore that most people do not have high-end PCs like the one described earlier. But what if high-end hardware were not even necessary?
When we think of AI models, there is probably a natural tendency to think of large frontier models, such as those used by services like ChatGPT or Grok. However, not every task requires a large frontier model. Plenty of everyday tasks can be handled just as easily by a small language model.
Small language models can run on local devices and are often well-suited to specific, repetitive tasks. For example, a small language model might summarize an email message, rewrite a document or answer questions using information retrieved from a collection of documents. Other lightweight AI models might handle tasks such as audio transcription.
Because small language models can run locally, they can respond without the latency of a round trip to the cloud. They may work without connectivity and may handle sensitive information without requiring it to leave the user's device. Of course, these privacy benefits depend on how the associated application is designed. If an application subsequently sends the original information or locally generated results to a cloud service, some of those advantages may be diminished.
Admittedly, even small language models require sufficient hardware to run, but that may be less of a problem than it initially seems. In 2024, Microsoft introduced Copilot+ PCs. These machines are equipped with neural processing units (NPUs) designed to accelerate AI workloads.
Initially, much of the attention around Copilot+ PCs focused on AI features built directly into Windows, such as Windows Recall. However, focusing on individual operating system features likely misses the larger picture. Copilot+ PCs include hardware capable of running some small AI models locally, potentially eliminating the need to send every AI request to the cloud. Of course, the NPU is not the only hardware component involved in local AI inference. The CPU, GPU and system memory can all play important roles.
The ability to run small models locally opens the door to a small-model/big-model architecture in which some inference tasks are handled locally, while more complex tasks are sent to the cloud. Microsoft could conceivably use small, specialized models for routine tasks and reserve its cloud-based models for work that requires greater computing resources.
In fact, Microsoft has already begun moving in this direction. The company recently announced Windows hybrid intelligence, an architecture designed to combine locally running AI models with cloud-based AI services.
Under this architecture, Windows will be able to determine where a particular AI task should be performed. Tasks that can be handled locally will run on the PC, while more demanding tasks will be sent to cloud-based models. In other words, the hybrid AI architecture described earlier is not just a theoretical possibility. Microsoft is actively working toward making it a reality.
The introduction of Copilot+ PCs could become an important turning point in the evolution of personal computing. Rather than treating the PC as little more than a terminal for accessing cloud services, Microsoft and other software companies could begin treating local hardware as an important part of a larger infrastructure.
This does not mean cloud-based AI is going away. Large language models will continue to require substantial computing resources, and some models will only be able to run on data center hardware. However, there is no reason every AI task needs to be processed in the cloud.
A hybrid approach combining small models running locally with large models in the cloud could deliver the best of both worlds. Users could potentially benefit from faster responses, improved privacy and greater flexibility, while service providers may be able to substantially reduce their infrastructure costs. The future of computing may not be cloud-only after all. It may be a hybrid architecture involving both the cloud and the PC.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.