Together AI launches self-tuning compiler for open-model inference

Together AI launched Together Compiler on November 4, an agent-driven system it says continuously retunes open-model inference as models, traffic patterns and hardware change. According to the company, the compiler is already running on some of its most popular serverless endpoints, though Together AI did not identify them or the underlying workloads.

Customer access is available through a limited beta. Together AI says the system powers optimization loops across many of its leading production inference workloads and serves trillions of tokens. Neither claim was independently corroborated in the opened sources. The company did not disclose beta pricing, eligibility, customer count or a general-availability date.

Together Compiler takes a workload’s traffic shape, model and hardware as inputs. Together AI says multiple software agents then propose and test changes. Persistent inference engines hot-reload candidate forward passes instead of restarting for each experiment. Faster, few-layer engines screen ideas before slower, many-layer engines test them against more realistic data distributions. A separate cost model uses hardware specifications, microbenchmarks and measured forward passes to narrow possible parallelism configurations as traffic changes.

In benchmarks, Together AI said the compiler found optimizations in less than 24 hours that delivered up to 2.6 times more tokens per second across named frontier open models. The company also claimed 1.7 times more tokens per second than available open-source megakernels on a batch-size-one Llama 1B workload, along with 252 times the experiment throughput of baseline agentic optimization systems. Those vendor-reported results were not independently reproduced in the opened material.

For low-latency workloads, Together AI reported 1,111 tokens per second on MiniMax M3.1-Flash and 1,205 tokens per second on DeepSeek V4.1-Flash using the AgentX benchmark. The AgentX methodology describes the benchmark as an inference-system test built from trace-derived, multi-turn coding-agent traffic. It measures serving performance, not the model’s task quality. The methodology says systems should be compared at matched latency or interactivity across a throughput curve, rather than only at a maximum-throughput point. Together AI also disclosed that its Trainium2 result used a synthetic multi-turn dataset instead of AgentX because its implementation could not support higher concurrency within available high-bandwidth memory.

Together AI presented the compiler as the first concrete project within “Polymorphic Inference,” its broader research agenda for inference systems that adapt as workloads and infrastructure change. The company’s post also discusses recursively self-improving inference, adaptive speculative decoding, heterogeneous routing, post-training and workload-aware model routing. The launch evidence does not establish those broader ideas as generally available products.

One-click, self-serve optimization for customers’ dedicated endpoints is also described as forthcoming, rather than available at launch. The shipped scope is currently limited to the compiler’s use inside Together AI’s own production service and limited-beta access for customers who request it.