Product News
Teams running LLM inference on NVIDIA B200s may be able to get higher throughput from AI-generated kernel optimization instead of maintaining hand-written megakernels, so it is worth benchmarking against their current serving stack.
Together AI describes Together Compiler, a system that uses AI agents to rewrite and fuse the GPU kernels in a model's forward pass. The post reports it reaches 2,118 tokens/s on Llama 3.2 1B on B200 GPUs, about 28% faster than the company's earlier hand-written megakernels at 1,649 tokens/s. It also traces the path from the megakernel approach to this agent-driven one.
Read the full article at Together AI
CoFabrix summarises and comments on this story. The original reporting belongs to Together AI.
Find out where your organization stands -- and what to do about it.