# Google launches LiteRT Tensor API for building and optimizing on-device ML models

published: 2026-11-04T02:05:43.488Z
topic: infrastructure
source: https://www.infoq.com/news/2026/11/litert-tensor-api/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=AI%2C+ML+%26+Data+Engineering
url: https://aiagentnews.io/stories/google-launches-litert-tensor-api-for-building-and-optimizing-on-device-ml

Google has released the LiteRT Tensor API, a lightweight C++ interface for building, customizing and optimizing machine learning models for on-device execution. It lets developers and AI coding agents construct and benchmark model graphs that run on CPU, GPU and NPU hardware without a rewrite for each target.

Google has introduced the LiteRT Tensor API, a C++ interface for building, customizing and optimizing models that run directly on devices. According to Google, it pairs the control of native model authoring with portability across platforms. Hand-written computations have usually been tied to backends such as Metal, OpenCL or Vulkan, and developers have had to rewrite them for each new hardware target and manage buffer synchronization across CPU, GPU and NPU themselves.

Models are built operation by operation as computation graphs that run through LiteRT on CPU, GPU and NPU. The API can load weights directly from checkpoints, including Hugging Face .safetensors files, and avoids copying accelerator buffers between pre-processing, inference and post-processing. Because graphs are defined in plain C++, AI coding agents can construct, modify, validate and benchmark them.

Google used a coding agent to build a multi-object segmentation and tracking app on the SAM 2 model and ran the same C++ code on GPUs of a macOS laptop, an iOS phone and an Android phone. On a MacBook Pro, Google reports about 1.5x faster performance than Apple MLX for fp16 image encoding plus mask decoding. Planned work includes ML Drift backend support, more NPU-friendly operations, automatic full-graph quantization and broader dynamic-shape support.

## Why it matters if you build agents

Developers building agents that run models on devices can have a coding agent generate and tune C++ graphs for several accelerators instead of hand-writing backend-specific code. Moving per-frame logic into the graph and replacing CPU-fallback operations were the optimizations that let the agent improve performance in Google's example. An Arm director quoted in the article recommends starting with model conversion and reaching for the C++ API only when graph-level control or on-accelerator pre- and post-processing is needed.