AI inference increasingly runs into memory and system-level constraints rather than a shortage of arithmetic. SambaNova designed its SN40L Reconfigurable Dataflow Unit around that problem, combining streaming dataflow with SRAM, HBM, and DDR memory. The architecture supports large models and collections of specialized models while reducing data movement and rack requirements. SambaNova’s comparisons with Cerebras, Groq, and GPUs point toward an emerging competition over utilization, memory capacity, deployment density, and system efficiency rather than peak compute alone.
The rapid expansion of generative AI has pushed accelerator designers toward ever more compute, yet inference performance increasingly depends on how efficiently a system feeds that compute. Memory bandwidth, memory capacity, interconnects, model switching, and software scheduling now determine how much silicon organizations actually use.
SambaNova built its SN40L Reconfigurable Dataflow Unit (RDU) around those constraints. Instead of treating memory as a supporting subsystem, it combines streaming dataflow, compiler-controlled operator fusion, and three levels of accelerator memory. The company argues that the architecture can run large models with fewer processors and less data center infrastructure while maintaining BF16/FP32 accuracy.
SambaNova compares its inference performance with Cerebras, Groq, and GPU-based services using Llama 3.1 models. The company reports 129 output tokens per second (TPS) per user for Llama 3.1 405B. In the comparison SambaNova supplied, neither Cerebras nor Groq served that model.
The 70B comparison exposes a more significant architectural difference. SambaNova uses 16 SN40L chips. Its analysis says Cerebras requires four WSE-3 wafers comprising 336 chips to produce 445 TPS. SambaNova estimates that Groq uses 576 LPUs for its 250 TPS result because each LPU contains only 230 MiB of SRAM.
These numbers come from SambaNova’s analysis and assumptions, so they should not serve as independent apples-to-apples system benchmarks. They do illustrate the architectural argument SambaNova wants customers to consider: Inference efficiency depends on the amount of hardware required to produce useful application throughput.
Three memory tiers attack the bottleneck
The SN40L’s memory architecture provides the foundation for that approach. Each socket combines 520 MiB of distributed on-chip SRAM, 64 GiB of co-packaged HBM, and as much as 1.5 TiB of directly-attached DDR DRAM. The system assigns different jobs to each memory tier.
SRAM supplies high bandwidth close to the computational units. HBM holds actively running models and frequently accessed data. DDR provides enough capacity to retain much larger collections of models and checkpoints close to the accelerator. SambaNova reports that an SN40L node can move models from DDR into HBM at more than 1 TB/s.
That arrangement directly addresses a growing inference problem. HBM provides enormous bandwidth, though its capacity limits how many large models or experts a processor can keep resident. Moving weights back to host memory adds latency and consumes PCIe bandwidth. Directly-attached DDR gives the accelerator another capacity tier without forcing every model into expensive HBM.
The SN40L uses TSMC’s 5nm process and a 2.5D CoWoS chiplet package containing two Reconfigurable Dataflow Dies and HBM. Each socket supplies 638 BF16 TFLOPS through 1,040 Pattern Compute Units (PCUs). Another 1,040 Pattern Memory Units (PMUs) provide distributed memory and address-generation resources.
The PCUs handle systolic and streaming computation. PMUs store tensors, parameters, metadata, and intermediate results. A Reconfigurable Dataflow Network connects those resources through programmable vector, scalar, and control fabrics.
Compiler becomes part of the architecture
SambaNova also moves considerable responsibility into software. Its compiler maps models into dataflow graphs and assigns operations across distributed compute and memory resources. It can combine operations into large fused kernels and create pipelines that keep intermediate data on-chip.
That approach attacks operational intensity, the amount of computation performed for each byte moved through the system.
SambaNova illustrates the effect with a Monarch FFT example. Without fusion, the workload reaches 39.5 operations per byte (OPB). Combining Gemm0, multiply, and transpose raises that figure to 102.6 OPB. Full spatial fusion reaches 410.4 OPB.
Figure 1. SN40L combines dataflow compute with three-tier memory.
The significance lies in reducing traffic to external memory. A conventional accelerator can contain enormous arithmetic capability and still leave much of it idle when data cannot arrive quickly enough. Increasing operational intensity allows more work to occur on data already inside the processor.
SambaNova says its compiler can fuse an entire Llama 3.1 8B decoder into one dataflow kernel and execute it repeatedly without continual kernel-launch overhead. For ISVs, that shifts optimization work from hand-written kernels toward compiler-directed mapping.
CoE changes model hosting
SambaNova extends the architecture through its Composition-of-Experts (CoE) approach. Instead of relying exclusively on one monolithic model, CoE combines independently developed specialist models with a router that selects an expert according to the request.
The company describes Samba-CoE, which contains 150 Llama-7B experts totaling more than 1 trillion parameters. The system stores those experts in DDR, keeps the router in HBM, moves the selected expert into HBM, and executes it there.
This model gives the three-tier memory architecture another purpose. DDR becomes a large local repository for models, while HBM operates as a high-speed working set. The runtime attempts to retain frequently used models in HBM and applies a least-recently-used policy when it needs additional capacity.
For enterprise AI, that capability could become increasingly relevant. Companies may operate dozens or hundreds of specialized models rather than route every task through a single large LLM. Coding, engineering, finance, customer service, translation, vision, and domain-specific applications can each use specialized models while sharing infrastructure.
Infrastructure efficiency becomes the metric
SambaNova’s argument ultimately extends beyond tokens per second. The company wants customers to measure useful throughput against silicon area, rack count, memory capacity, power, and deployment complexity.
That matters for CIOs as inference moves from experimental workloads into persistent infrastructure. A processor that wins a benchmark while requiring substantially more racks, networking, memory, or power may produce a less attractive deployment. Silicon teams face the same calculation at the architectural level: Additional compute delivers diminishing value when memory and communication cannot keep it occupied.
SambaNova’s competitive claims still require independent evaluation across broader workloads, concurrency levels, latency targets, software environments, and total system power. Groq, Cerebras, Nvidia, and other accelerator vendors also continue to evolve their architectures and software rapidly.
The SN40L, nevertheless, demonstrates an important direction in AI hardware. Arithmetic throughput remains important, while memory placement, compiler behavior, model residency, interconnect efficiency, and rack-level utilization increasingly determine application performance.
AI inference has started shifting the accelerator discussion from individual chip specifications toward complete systems. SambaNova designed SN40L around that transition. Its combination of dataflow execution, three-tier memory, compiler-controlled fusion, and model management treats the memory wall as an architectural problem spanning silicon and software. Production deployments will determine how effectively that approach translates into lower cost, power, and infrastructure requirements across diverse enterprise workloads.
What do we think?
SambaNova addresses a genuine AI infrastructure constraint: Processors need enough memory bandwidth and capacity to keep expensive compute resources productive. SN40L’s SRAM-HBM-DDR hierarchy and dataflow compiler form a coherent response. Its strongest argument may ultimately come from deployment density rather than raw tokens per second. Independent measurements of power, concurrency, latency, software maturity, and cost will determine its competitive position.
Inflection point. The architectural emphasis on memory capacity, model residency, compiler scheduling, and system footprint signals a broader change in AI infrastructure. The industry has spent years emphasizing accelerator compute; inference makes sustained utilization increasingly important. This could mark an inflection point as enterprises deploy collections of specialized models and continuous AI services. If that transition continues, accelerator competition will increasingly center on moving and managing data efficiently, not simply adding arithmetic units to the processor.
LIKE WHAT YOU’RE READING? TELL YOUR FRIENDS; WE DO THIS EVERY DAY, ALL DAY.