Together AI launches compiler to optimize open-model inference

World · First reported 4 Nov, 13:00 · Updated 13:13

Image: Wikimedia Commons · NASA Earth Observatory image by Lauren Dauphin, using Landsat data from the U.S. Geological Survey. Story by Adam Voiland. · Public domain

Together AI has released a compiler that uses AI agents to tune GPU code for open-source models. The company says early tests showed faster performance on several workloads, while positioning the tool as the first part of a broader effort to automate inference optimization.

Together AI says its compiler automatically optimizes code for Nvidia and AMD GPUs, aiming to reduce work that can otherwise take engineers days or weeks.
The company reports speedups on tests involving Llama 1B, MiniMax m3 and Kimi K3, including up to 1.3 times faster than hand-written megakernels on Llama 1B.
Together calls the larger initiative Polymorphic Inference, in which AI agents are intended to continually adapt and improve systems running open models.

Summary written by Lux AI from 2 articles

Key facts

Tool: A compiler for automating GPU tuning of open-source models
Supported chips: Nvidia and AMD
Llama 1B result: Up to 1.3 times faster than hand-written megakernels, according to Together AI
Other reported results: A MiniMax m3 layer was made 1.5 times faster; a Kimi K3 long-request step was reported 1.7 to 2.5 times faster