Foundry Local runs Qwen3.8-27B on-device for GitHub Copilot coding

Microsoft says Qwen3.8-27B is the main optimization target for local coding, and that a 4-bit quantized build comes to about 18 GB, enough for consumer GPUs with 24 GB or more of VRAM. Built on ONNX Runtime, Foundry Local applies speculative decoding, CUDA graph fusions, INT4 weight handling, KV-cache quantization and CPU offloading on NVIDIA GPUs. Its serving layer adds prefix caching and continuous batching for concurrent requests.

In a comparison with llama.cpp, Microsoft tested the 4-bit model on an NVIDIA GeForce RTX 5090 with 32 GB of VRAM and on an NVIDIA DGX Spark with 128 GB of unified memory, using a dataset of real GitHub issues. Foundry Local reported higher generation speed at every prompt length tested, and the gap widened beyond 131K tokens.

Setup starts with a WinGet install of the Foundry Local CLI, then foundry model load qwen3.8-27b and foundry server status to get the API endpoint. That endpoint is entered as the Base URL for a Foundry Local provider in the GitHub Copilot app. CUDA and cuDNN runtimes must be installed separately, and Microsoft plans to add support for Macs and AMD GPUs.

Developers building coding agents can point GitHub Copilot at a local endpoint instead of a cloud model, which keeps code on the machine and gives more predictable inference costs. Microsoft notes that local models may reason less well than large cloud models on complex tasks, so a hybrid setup that sends hard work to the cloud is the suggested pattern. Check GPU VRAM and the CUDA and cuDNN installation before relying on this path, since the CLI does not install those dependencies.