Together AI unveils Together Compiler for continuous inference optimization

Together AI has introduced Together Compiler, called TKCC, a system that uses agents to generate optimizations for inference engines. The company says it recompiles as traffic shifts across regions, after new model releases and during bursts of requests.

Rather than restarting an engine for each experiment, TKCC keeps a persistent engine running and hot-reloads candidate forward passes. Together AI reports that total time to serving was 467 seconds in its baseline measurement and 9.0 seconds with Together Compiler. Two engine tiers trade off speed against fidelity: fast engines load a few layers for quick iteration, while slower engines load more layers to test on realistic data.

Together AI reports 1.3-3x higher TPS than highly optimized baselines across frontier open-source models within 24 hours. It also reports 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on SemiAnalysis' InferenceX low-latency benchmark, and 1.7x higher TPS than open-source megakernels on Llama 1B at batch size 1.

Teams running open models on Together's platform may see throughput change as the system recompiles for new traffic, so it is worth checking reported benchmark gains against their own workloads.

The persistent-engine pattern, where an engine stays up and accepts new kernels without restarts, is a design to consider for agent-driven tuning loops, since engine startup dominated experiment time in the reported baseline.

These figures come from Together AI's own benchmarks, and the post does not describe public availability of the system.