SambaNova has shifted the AI infrastructure conversation from replacing GPUs to using them more efficiently. New benchmark results show its fifth-generation SN50 Reconfigurable Dataflow Units (RDUs) working alongside Nvidia H200 GPUs to deliver substantially higher inference throughput than GPU-only systems on selected workloads. The approach separates prompt processing from token generation, allowing existing GPU clusters to remain productive while specialized accelerators assume decode operations. If production deployments match these results, enterprises could lower inference costs while extending the life of installed AI infrastructure.
For the past several years, the AI industry has pursued one objective: deploy larger GPU clusters. Every new generation increased compute performance, memory bandwidth, power consumption, and acquisition cost. That strategy aligned well with the demands of training foundation models, where dense floating-point performance determines how quickly enormous datasets become usable models.
Inference presents a different challenge.
Once a model reaches production, serving millions of users depends less on peak compute capability and far more on prompt processing, memory management, context handling, and sustained token generation. Those requirements have encouraged AI infrastructure vendors to rethink system architecture instead of simply installing more GPUs.
SambaNova believes heterogeneous inference provides the next stage of AI infrastructure evolution.
At the RAISE Summit 2026, the company demonstrated what it describes as a premium inference platform built around a heterogeneous architecture. The configuration combines one Nvidia H200 rack containing four GPUs for the compute-intensive prefill stage with one SambaRack SN50 populated with 16 fifth-generation Reconfigurable Dataflow Units (RDUs) dedicated to decode operations. The demonstration extends the heterogeneous blueprint first shown at COMPUTEX, where Nvidia B200 GPUs handled prefill while SambaNova SN40 accelerators executed decode. The latest configuration replaces the SN40 with the new SN50 platform and targets emerging agentic AI workloads.
Independent benchmarking by Artificial Analysis measured the system using the MiniMax M2.7 model and reported decode performance reaching 850 tokens per second on short-context workloads while sustaining more than 450 tokens per second on long-context inference. Earlier benchmark disclosures reported approximately 763 tokens per second. The newer results reflect continued optimization of the SN50 platform while establishing a new performance point for MiniMax inference.
The architecture divides inference into two distinct operations.
The prefill phase processes prompts, performs the matrix computations, and generates the key-value cache. GPUs remain exceptionally well-suited for this highly parallel workload.
Decode follows a different pattern. Each output token depends on previous tokens, making memory bandwidth and data movement more important than peak arithmetic throughput. SambaNova designed its RDU architecture specifically for that portion of the inference pipeline.
Rather than replacing GPUs, the system assigns each processor the workload it executes most efficiently.
For cloud providers and enterprise operators, that distinction changes infrastructure economics. Existing H200 GPU clusters continue handling compute-intensive prefill while SN50 systems provide dedicated decode capacity. Organizations can increase inference throughput without replacing entire GPU installations.
Dataflow instead of brute force
The benchmark demonstrates more than higher token-generation rates. It highlights SambaNova’s underlying dataflow architecture.
Traditional GPU architectures emphasize massive parallel execution supported by increasingly large memory systems. SambaNova instead focuses on reducing data movement by overlapping communication with computation and by keeping active data closer to execution resources. The company argues that sustained inference depends more on architectural efficiency than on advertised peak FLOPS.
The SN50 represents SambaNova’s fifth-generation Reconfigurable Dataflow Unit and replaces the SN40L introduced during 2024. According to the company, the new processor increases FP16 performance by approximately 2.5× while raising FP8 throughput by roughly 5×, reaching approximately 1.6 PFLOPS FP16 and 3.2 PFLOPS FP8.
On paper, those figures remain below the theoretical peak performance of Nvidia’s latest Blackwell-class GPUs. SambaNova argues that peak specifications rarely translate directly into production inference because memory movement, synchronization overhead, and utilization often determine real-world performance.
Memory architecture illustrates that philosophy.
Each SN50 incorporates 432 MB of on-chip SRAM, 64 GB of HBM2E delivering approximately 1.8 TB/s of bandwidth, and between 256 GB and 2 TB of DDR5 memory. SambaNova intentionally retained HBM2E instead of adopting newer memory technologies, citing lower cost and improved supply availability. The larger DDR5 pool allows models and key-value caches to move between memory tiers in milliseconds, supporting rapid model switching while maintaining high utilization.
Disaggregated inference gains momentum
SambaNova joins a growing list of companies embracing disaggregated inference. Nvidia introduced the concept through its NVL72 systems by allocating different GPU resources to prefill and decode workloads. The company expanded that strategy following its acquisition of Groq engineering talent. AMD, AWS, Cerebras, and several emerging accelerator companies have introduced similar architectures.
The industry increasingly recognizes that no single processor architecture delivers optimal efficiency across every stage of AI inference.
For hyperscalers operating hundreds of thousands of accelerators, modest utilization improvements translate directly into lower operating costs and higher service capacity.
Extending installed GPU infrastructure
Perhaps SambaNova’s most compelling argument centers on installed infrastructure rather than new hardware.
Most enterprises already own significant investments in Nvidia accelerators. Many of those systems continue providing substantial compute capability despite the introduction of newer GPU generations.
Instead of encouraging customers to replace those assets, SambaNova proposes extending their useful life.
Existing H100 and H200 clusters continue executing prefill while SN50 racks perform decode. Enterprises gain higher inference throughput without purchasing complete replacement GPU clusters.
The strategy also simplifies deployment.
Unlike many next-generation AI systems requiring liquid cooling, SambaNova’s SN50 racks remain air cooled and fit within existing data center infrastructure. CIOs evaluating AI expansion can, therefore, increase inference capacity without undertaking expensive facility upgrades.
SambaNova recently strengthened both its financial position and its industry partnerships.
The company completed the first close of a $350 million financing round as part of a broader $1 billion Series F, producing an estimated valuation of approximately $11 billion. General Atlantic led the financing, with participation from Intel Capital, Vista Equity Partners, Cambium Capital, and several additional institutional investors.
Intel’s participation also addresses speculation that the company intended to acquire SambaNova. Instead, Intel chose a multi-year engineering partnership focused on heterogeneous AI infrastructure.
The collaboration combines Intel Xeon processors with SambaNova RDUs through joint hardware-software optimization aimed at enterprise AI deployments. Intel identified Vector Core Compute as one of the first deployment partners, while Together AI will become an early large-scale commercial customer.
For Intel, the partnership expands its AI infrastructure strategy beyond GPUs following the company’s limited success with previous data center GPU and Gaudi initiatives.
SambaNova views today’s 16-RDU system as only the beginning. Future configurations will expand to 128 and, eventually, 256 RDUs connected through a switched interconnect delivering approximately 2.2 TB/s of bidirectional chip-to-chip bandwidth per accelerator.
Those larger systems target AI agents, coding assistants, retrieval-augmented generation, enterprise search, and other production inference workloads where latency, throughput, utilization, and cost per generated token determine profitability.
The architecture also addresses another emerging challenge.
Enterprise AI increasingly depends on customized models. Departments, customers, and applications often require separate models rather than sharing one foundation model. Those specialized deployments lower accelerator utilization because each model occupies valuable memory resources.
SambaNova uses its large DDR5 memory pool to move models and key-value caches rapidly between memory tiers, allowing infrastructure providers to maintain higher utilization while supporting many customized models.
According to CEO Rodrigo Liang, improving rack economics became the company’s primary engineering objective during 2025 because profitable inference depends as much on utilization as on processor performance.
AI infrastructure becomes heterogeneous
The broader story extends beyond one company’s benchmark results.
Nvidia clearly dominates AI training. Inference remains a far more dynamic market where architecture, utilization, operating cost, and deployment flexibility matter as much as raw compute capability.
Future AI data centers will likely combine CPUs, GPUs, specialized inference accelerators, networking processors, storage controllers, and memory-centric architectures, with each processor optimized for a specific workload.
That evolution mirrors the broader history of computing. Systems repeatedly progressed from general-purpose processors toward specialized architectures as applications matured.
Inference now appears to be following that same path.
Table 1. Heterogeneous inference architecture.
SambaNova’s latest announcement demonstrates more than another benchmark victory. It presents a practical strategy for improving inference economics without forcing enterprises to discard existing GPU infrastructure. By combining installed Nvidia accelerators with purpose-built decode processors, the company offers an incremental deployment model that reduces capital spending, increases utilization, and simplifies data center expansion. Whether this architecture becomes mainstream will depend on production deployments, software maturity, and customer adoption, yet the underlying economic argument deserves careful attention from every organization building large-scale AI infrastructure.
What do we think?
SambaNova has shifted the discussion from processor replacement to infrastructure optimization. That strategy aligns with enterprise purchasing behavior because organizations prefer extending existing investments before funding another hardware refresh. If heterogeneous inference consistently lowers token costs while improving utilization, specialized inference accelerators could become standard companions to GPU clusters rather than direct competitors.
Inflection signal
This announcement may represent an inflection point in AI infrastructure. The first phase of AI centered on acquiring the largest GPU clusters available. The next phase may focus on extracting greater efficiency from those investments through heterogeneous computing. If enterprises adopt specialized decode accelerators alongside installed GPUs, future data centers will evolve toward workload-specific processor combinations rather than homogeneous accelerator fleets. That shift could reshape procurement strategies, infrastructure planning, and the competitive landscape for AI silicon over the remainder of the decade.
Take a look at our AI library, where, among things, we keep track of the 150 companies offering nearly 300 AI processors. JPR puts the “I” in AI (and because you will ask—Intelligence, market intelligence).
LIKE WHAT YOU’RE READING? TELL YOUR FRIENDS; WE DO THIS EVERY DAY, ALL DAY.