Skip to main content

Last year we built megakernels. This year we deleted them: The road to Together Compiler

Together AIUrgency: NotableOfficial

Why Last year we built megakernels. This year we deleted them: The road to Together Compiler matters

Teams running LLM inference on NVIDIA B200s may be able to get higher throughput from AI-generated kernel optimization instead of maintaining hand-written megakernels, so it is worth benchmarking against their current serving stack.

Summary

Together AI describes Together Compiler, a system that uses AI agents to rewrite and fuse the GPU kernels in a model's forward pass. The post reports it reaches 2,118 tokens/s on Llama 3.2 1B on B200 GPUs, about 28% faster than the company's earlier hand-written megakernels at 1,649 tokens/s. It also traces the path from the megakernel approach to this agent-driven one.

Read the full article at Together AI

CoFabrix summarises and comments on this story. The original reporting belongs to Together AI.

More in Product News

Browse the full AI Pulse feed

Seeing AI Disruption in Your Industry?

Find out where your organization stands -- and what to do about it.