Nvidia says the AI memory crunch is solved by codesign, not more HBM, citing 30x throughput per dollar

Bigger models, longer contexts and agentic trajectories are squeezing GPU high-bandwidth memory while supply stays tight. Agentic inference makes it worse: KV cache spans tool calls, RAG grows prompts, and trillion-parameter models compete for the same tier. Nvidia’s answer is full-stack codesign: shrink what must live in HBM, park the rest on the right cheaper tier, and fetch it before it is needed. From Blackwell Ultra to Vera Rubin, the company says HBM capacity per GPU rose roughly 3% while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX, with GB300-versus-H200 and Vera Rubin-versus-GB300 figures anchoring the claims. Its warehouse-and-docks metaphor fits: extra HBM without bandwidth and orchestration is empty aisle space that still draws power.

This mirrors Nvidia’s developer deep dive on five codesign levers (five levers) and its split of long-context inference across Rubin GPUs and Groq LPX (Groq 3 LPX). AI factories should optimize useful tokens inside a power envelope, so operators buying on tokens per watt rather than gigabytes of HBM is healthy. Nvidia also cites first-mover adoption of HBM3e and HBM4.

Vendor-chosen AgentX comparisons deserve care. The bigger risk is codesign becoming synonymous with one vendor’s hierarchy, so switching costs calcify and alternative memory architectures struggle, the same lock-in dynamic seen in Nvidia’s networking consolidation (GPUNetIO). Useful efficiency is service-level tokens delivered without overstating capacity; measure goodput, not installed HBM. More in our Nvidia coverage and business coverage.