News

Google Pushes AI Inference Out of the Cloud and Into the Browser

• By David Ramel
• 07/16/2026

Google is pushing more AI inference out of cloud datacenters and into users' browsers with the launch of LiteRT.js, a JavaScript binding for its cross-platform LiteRT on-device runtime.

"By bringing the trusted on-device inference library LiteRT to the web, web developers can now run ML and AI models with maximum performance entirely locally," said Google in announcing LiteRT.js, a JavaScript binding of LiteRT for running AI directly inside a web browser. "This means enhanced user privacy, zero server costs, and ultra-low latency for real-time experiences. For developers with existing .tflite models, LiteRT.js makes deployment to mobile and desktop web browsers smoother than ever, serving as a powerful evolution from TensorFlow.js for executing .tflite models."

Announced July 9,the open-source LiteRT.js lets JavaScript and TypeScript developers execute machine learning models entirely on client devices, using local CPUs and graphics processing units (GPUs) rather than sending each inference request to a cloud-hosted service.

That changes the deployment equation for web AI. Cloud inference remains necessary for large models, centralized data and workloads requiring substantial compute, but sufficiently compact models can now be downloaded as part of a web application and repeatedly executed on the user's own hardware.

Instead of paying to process every image, audio segment or text input on cloud infrastructure, developers can move suitable workloads to browsers and effectively use client hardware as the inference tier.

Moving the Inference Tier
The traditional cloud AI architecture sends application data to a remote endpoint, where a model processes the request and returns a result. That approach gives developers centralized control and access to powerful hardware, but it also introduces network latency, server capacity requirements, API charges and potential privacy concerns.

LiteRT.js offers another option for workloads that can fit within browser memory and run efficiently on consumer hardware. Once a model and the required WebAssembly components have been delivered to the browser, the client can perform subsequent inference locally without making a server request for every operation.

Potential use cases include:

• Running object detection or image classification against a webcam feed without continuously uploading video.
• Processing speech, audio or images locally for real-time web applications.
• Generating embeddings and performing vector search inside the browser.
• Upscaling or manipulating images without transferring the original files to a remote service.
• Adding offline or intermittently connected AI features to web applications.
• Keeping sensitive user inputs on the device rather than sending them to a cloud endpoint.
• Reducing token or API consumption by handling simple classification, extraction, embedding or compact generative AI tasks locally while reserving cloud models for more difficult requests.

The token-cost scenario is most applicable to hybrid applications. A local model could perform preliminary work such as classifying an input, extracting relevant information, generating embeddings or determining whether a request requires a more capable cloud model. Only requests that exceed the local model's capabilities would be escalated to a paid cloud service.

LiteRT.js would not automatically reduce charges from an existing cloud AI product, nor would it make cloud-hosted models run locally. Developers would need to design the application to route appropriate work to a compatible local model instead of making the corresponding remote API call.

Hardware Acceleration in the Browser
LiteRT.js delivers Google's native LiteRT runtime to browsers through WebAssembly rather than relying primarily on JavaScript-based inference kernels. Google positions it as an evolution from TensorFlow.js for developers deploying models in the '.tflite' format.

The runtime can target three hardware paths:

• CPU: LiteRT.js uses XNNPACK, Google's optimized library for multithreaded CPU inference.
• GPU: It uses Google's ML Drift technology and the WebGPU browser API to accelerate workloads on integrated or discrete graphics hardware.
• NPU: Planned and experimental support uses the Web Neural Network API (WebNN) to reach dedicated neural processing units.

WebGPU is already available in current versions of major browsers, although support and performance vary by browser, OS and hardware. WebNN remains experimental and requires specific browser settings and compatible system drivers. Google said its development roadmap includes deeper WebNN integration for native, power-efficient NPU inference.

The company claims LiteRT.js outperformed other web runtimes by up to 3x across CPU and GPU tests involving classical computer vision and audio-processing models. It also reported GPU or NPU performance ranging from five to 60 times faster than CPU execution for workloads such as object tracking, transcription and image manipulation.

Those benchmark results were obtained in a controlled browser environment on a 2024 Apple MacBook Pro with M4 silicon. Google cautioned that actual performance will vary according to local GPU capabilities, thermal throttling and browser-driver optimization.

From PyTorch to the Web
Developers can convert compatible PyTorch models to LiteRT's .tflite format with LiteRT Torch, then install the new @litertjs/core npm package or load it from a content delivery network.

The runtime handles model loading, compilation and execution, while developers supply the surrounding preprocessing and post-processing logic. Applications can select WebGPU, WebNN or WebAssembly-based CPU execution according to the browser and hardware environment.

Google also provides an AI Edge Quantizer for reducing model size and improving execution performance with model-specific quantization settings. Smaller models are particularly important for browser deployments because large downloads, memory limits and slower client hardware can make otherwise capable models impractical.

The current implementation also has limitations. According to Google's LiteRT.js documentation, some model operations may not be supported by every backend, input and output tensors are currently limited to certain data types, partial delegation between CPU and GPU is not supported, and large models can exceed browser or WebAssembly memory limits.

In some cases, applications can fall back from GPU or WebNN execution to the CPU. Other compatibility problems may require developers to modify and reconvert the original model.

Generative AI Is Next
The initial announcement emphasizes established machine learning workloads such as object detection, depth estimation, audio processing, vector search and image upscaling rather than presenting LiteRT.js as a browser replacement for cloud-scale large language models.

Google nevertheless lists highly optimized on-device generative AI as a major development priority. It also points developers to LiteRT-LM.js, the browser-facing component of the company's open-source LiteRT-LM framework for large language model inference.

That makes the broader cloud implications more significant over time. As smaller language and multimodal models become more capable, web applications could divide work among three execution tiers: local browser models for frequent and privacy-sensitive operations, edge or regional infrastructure for intermediate workloads, and large cloud models for requests requiring maximum capability.

Such an architecture would not eliminate cloud AI. It could, however, reduce the number of requests reaching it, limit how much user data must be transmitted and let developers reserve costly centralized compute for the tasks that actually need it.

LiteRT.js is available through npm, with demos covering vector search, object detection, depth estimation and image upscaling. Google said future development will expand model coverage, tooling, WebNN integration and generative AI support.