Skip to main content

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

Hacker NewsUrgency: NotableUnconfirmed

Why Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium matters

Teams evaluating cheaper inference than H100 or H200 hardware can test Trainium with this open-source engine, but should verify the reported benchmarks on their own workloads before committing to it.

Summary

RunAnywhere open-sourced TrnRT, an LLM inference engine for AWS Trainium, currently serving the GLM-5.3 flash model in FP8 on a trn2.48xlarge with a cache-aware router. The team reports 3.57M tokens per minute on SemiAnalysis's AgentX benchmark and a 25% decode speedup from custom NKI megakernels, and argues that Trainium's memory cost is under half that of 8x H200 on AWS. AWS's own Neuron and NxDI stacks did not adequately support the model, so the project fills a gap for Trainium users.

Read the full article at Hacker News

CoFabrix summarises and comments on this story. The original reporting belongs to Hacker News.

More in Open Source

Browse the full AI Pulse feed

Seeing AI Disruption in Your Industry?

Find out where your organization stands -- and what to do about it.