英伟达GPU是否支持超高交互性？

newsletter.semianalysis.com/p/ultra-high-interactivity-on-nvidia Bryan Shan, Daniel Nishball, Cam Quilici, Kimbo Chen, Alec Ibarra, Dylan Patel August 10, 2026 Premium-priced “fast modes” are proving that users will pay more for lower latency andfaster tokens, potentially yielding higher gross margins. Frontier AI labs such asOpenAIare therefore evaluating purpose-built inference systems, including Cerebras and NVIDIAGroq LPUsthat prioritize ultra-high interactivity over maximum batched throughput. Ultra-low latency matters most in interactive workloads, including real-time assistants, and full-duplex voice. OpenAI GPT‑Live, for example, can listen and speak simultaneously, makingresponse delay immediately perceptible to the user,described as feeling like IronmanJARVIS. GPUs perform exceptionally well at high throughput and low-to-medium interactivity, buttheir architecture is less suited for ultra-low-latency inference. An 8-GPU HGX B200 serverprovides a theoretical HBM memory bandwidth of 64 TB/s of in aggregate. At batch size 1,GLM-5 at NVFP4 requires only approximately 21 GB of active-parameter traffic pergenerated token. The B200 HBM bandwidth roofline would thereforesuggest up to 3,047tokens/s/user without speculative decoding. In practice, GPUs come nowhere closeto this limit. The gap comes from latency rather than bandwidth. The traditional GPU programmingmodel launches and synchronizes many individual kernels, whose setup and teardownoverhead becomes significant at ultra-high levels of interactivity. While these latency costsare less visible at conventional serving speeds, even with CUDA graphs, they dominate as token latency approaches the sub-millisecond Time Per Output Token (TPOT) range.Furthermore, although GPU memory bandwidth increases by roughly 2–3× eachgeneration, memory latency has not improved at all. While using alternative hardware is popular, there are ways to use GPUs to do this too.This is whereTileRT’s persistent enginecomes in. TileRT statically compiles the entiredecode graph into a single persistent kernel on NVIDIA GPUs, maximizing overlap acrosscomputation, memory loads and stores, and communication.On the InferenceX GLM5FP8 744B benchmark on a single B200 decode server, tileRT has been verified toreach up to 500 tokens/s/user, approximately 3× faster than GB300 NVL72 runningtraditional inference engines. Iso-cost per output token, TileRT can achieve up to 2xfaster interactivity than traditional engines. We thank the TileRT maintainers for collaborating on TileRT InferenceX benchmarks andalso in general thankful to the vLLM community for their amazing design on the V1connector. TileRT comes from the same community maintainer organization that built thewidely popular TileLang DSL. With PD disaggregation inference technique, the hyperspecialized TileRT engine handleslatency-sensitive decode while throughput-optimized engines such as vLLM and SGLangcontinuing to serving prefill. The TileRT decode engine is already being deployed inproduction atXiaomi for MiMo V2.5 Pro UltraSpeedandZAI with GLM 5.1 HighSpeed. In the article, we shall deep dive into the TileRT InferenceX results, what TileRT is, how itcomposes with the existing inference ecosystem along with the tradeoffs and challengeswith TileRT. We will also elaborate on the tradeoffs of using TileRT on standard GPUs vs.ultralow latency specialized chips like Nvidia Groq LPU, Cerebras and Sambanova,weighing in onif there is a potential for TileRT software running on GPUs to disrupt thesespecialist chips’ TAM.The SemiAnalysis Accelerator Model provides quarter by quarterestimates of Nvidia LPU30, LPU40, Cerebras WSE-3 & WSE-4 shipments and much more. InferenceX InferenceX is our open-source, vendor-neutral, continuously updated AI inferencebenchmarking and research platform.We measure leading models, inference frameworks,and hardware across the latency-throughput Pareto frontier, tracking how real-worldinference performance and economics improve over time. Thanks for reading SemiAnalysis! This post is public so feel free to share it. Our benchmark has been widely reproduced, validated and/or supported by almost everymajor buyerof compute fromGoogle CloudtoMicrosoft AzuretoOracle,toMetaand many more. Furthermore, it has thesupport of the ML community including from vLLM,LMCache, SGLang, PyTorch, Huggingfaceand the support ofmajor labs like OpenAI,MiniMax, ZAI, Qwen, Moonshot Kimi, etc. Star the InferenceX GitHub repository if you find the open-source benchmark and datauseful!.As previously mentioned,Nvidia has committed to submitting verifiable Vera Rubinnumbers to InferenceX. We will have Google TPUv7 results soon, and AMD has committedto MI455X UALoE72 this year too. Throughput vs Interactivity Curve Every inference system must balance two competing goals. Interactivity (tok/s/user)measures how quickly a single user receives tokens, theinverse of time per output token (TPOT). It determines whether a response