Nvidia has moved Groq 3 LPX into full production as an inference accelerator focused on the decode side of agentic AI. The architecture combines 256 LP30 processors, rack-scale SRAM and compiler-scheduled chip-to-chip communication to drive token generation at very small batch sizes. Artificial Analysis measured more than 3,400 output tokens per second on Gemma 4 31B with 100K context. Nvidia plans to pair LPX with Vera Rubin NVL72 for long-context, highly interactive inference workloads.
Agentic AI puts latency under pressure.
Agentic AI changes the inference workload because an agent rarely generates one answer and stops. It can inspect files, call tools, write and test code, evaluate results, and repeat that sequence hundreds or thousands of times. Each step adds tokens to the context, while each new inference step must process an increasingly large history. That makes token-generation latency a direct constraint on how quickly an agent can complete useful work.
Nvidia’s Groq 3 LPX addresses that part of the problem. Nvidia acquired Groq’s technology and has positioned LPX as an extension of the Vera Rubin platform, giving its AI factory architecture a processor optimized specifically for interactive inference. Nebius plans to become the first AI cloud to deploy Groq 3 LPX through its Token Factory production inference platform.
Artificial Analysis measured Groq 3 LPX running Gemma 4 31B at 3,431 output tokens per second with a 100,000-token input context. At a 10K context, the median reached 3,382 tokens per second. Nvidia also reports a median 4,767 tokens per second on the coding-oriented Speed-Bench workload, with 20% of tasks exceeding 5,500 tokens per second.
Those numbers matter because agentic workloads place greater value on individual-user responsiveness than conventional high-batch inference. At 3,431 tokens per second, generating 5,000 tokens takes roughly 1.5 seconds. At 100 tokens per second, the same operation takes 50 seconds. Multiply that difference across an agent performing dozens or hundreds of sequential reasoning and tool-use steps, and decode speed can materially change application behavior.
Tensor parallelism meets first-bit latency
Groq 3 LPX attacks the problem through tensor parallelism (TP), dividing computations among many processors and recombining their results. TP can increase throughput significantly, yet small-batch inference creates a communication problem. The tensors moving between processors remain small, making fixed communication overhead increasingly important.
The basic relationship is: Transfer time = A + N/B
Where A represents fixed first-bit latency, N represents the amount of data transferred, and B represents link bandwidth. When N becomes small, adding bandwidth produces diminishing benefits because A consumes a larger fraction of the transfer time.
That distinction helps explain Groq’s architecture. Rather than depend primarily on enormous link bandwidth, LPX tries to reduce the coordination delay preceding each transfer.
A Groq 3 LPX rack contains 256 LP30 LPUs with 128 GB of aggregate SRAM. Each chip has 96 chip-to-chip links operating at 112 Gb/s. The compiler sees the compute units, memory resources, and network links before execution and creates a clock-level schedule for the workload.
That deterministic model eliminates much of the real-time arbitration normally required when processors exchange data. The compiler determines where data will travel and when the sender and receiver will use each link. An LPU can, therefore, transmit data when the scheduled clock cycle arrives, rather than negotiate the transfer dynamically.
Compute and communication overlap
Groq extends deterministic scheduling into the computation itself. The compiler schedules compute and communication at the granularity of 320-byte vectors. During a matrix multiplication, an LPU can calculate a slice of the output and immediately begin transmitting it instead of waiting for the entire matrix operation to finish.
That overlap reduces the communication tail following a computation. It also helps TP remain effective at the small batch sizes required for interactive inference, where communication overhead can otherwise equal or exceed the compute time saved through parallel execution.
For silicon teams, this illustrates an architectural trade-off. Conventional accelerators devote substantial resources to flexible execution, caches, routing, and dynamic workload behavior. Groq trades some of that flexibility for deterministic scheduling. The compiler knows the machine and workload in advance and coordinates both as one system.
For ISVs, the important metric becomes time to completion rather than isolated accelerator throughput. An agent that generates thousands of tokens, executes a tool, evaluates the result, and starts another inference pass depends on consistent low latency across every stage.
LPX joins Vera Rubin
Nvidia does not position LPX as a replacement for Vera Rubin NVL72. The supplied material describes several ways the two architectures can divide inference.
Figure 1. Groq 3 LPX accelerates agentic inference at scale. (Source: Nvidia)
Under standard prefill-decode disaggregation, Vera Rubin handles prefill and transfers the KV cache once per turn. LPX then performs decode using model weights stored in SRAM. Another configuration assigns attention and KV-cache management to Vera Rubin, while LPX executes feed-forward-network layers. Nvidia also proposes speculative decoding in which LPX runs a smaller draft model, while Vera Rubin verifies the proposed tokens.
This specialization could become important as AI factories move away from treating inference as one homogeneous workload. Prefill emphasizes processing large amounts of context efficiently. Decode emphasizes rapid sequential token generation. Assigning those tasks to different architectures lets Nvidia optimize each processor for a narrower workload.
Groq 3 LPX gives Nvidia a specialized inference architecture built around deterministic execution, SRAM, and tightly scheduled interprocessor communication. The larger significance comes from how Nvidia combines it with Vera Rubin. Instead of forcing every stage of an agentic workload through the same processor architecture, the company can divide inference according to computational behavior and route each phase toward hardware designed for that job.
What do we think?
Groq 3 LPX gives Nvidia another architectural tool for an inference market increasingly defined by latency and agent responsiveness. Its deterministic compiler and SRAM-centric design directly address small-batch tensor-parallel communication. The 3,400-plus-token results deserve attention, particularly with 100K context. The larger test will involve deployment economics, model coverage, utilization, and integration with Vera Rubin at production scale.
Inflection point. Groq 3 LPX suggests that AI infrastructure may be approaching an inflection point where heterogeneous accelerators become standard inside an AI factory. Training, prefill, attention, and decode do not impose identical computational demands, and agentic AI magnifies those differences. Nvidia’s decision to combine Vera Rubin with LPX indicates a move toward workload-specific compute within one platform. If agentic workloads dominate inference growth, token latency and processor specialization could influence system architecture as strongly as aggregate accelerator throughput.
LIKE IT? WE’VE GOT LOTS MORE. TELL EVERYONE YOU KNOW. WE’D LOVE TO HEAR FROM YOU AND THEM.