Microsoft detailed Maia 300, its third-generation inference chip, and the numbers show where the gains come from. Each accelerator carries 288 GB of HBM4 at 18 TB/s, 2.5 times Maia 200, and 96 of them fill a 167 kW rack. Up to 128 racks link into a 12,288-accelerator superpod. Microsoft claims seven times Maia 200’s rack throughput and 40% more tokens per megawatt than GPU systems. We break down the design, test the claims, and list what to watch at Ignite.
Microsoft has built a chip for answering questions, and it plans to build many of them. Maia 300, which Microsoft announced on Monday, targets inference, the work an AI model performs each time it responds to a prompt. Production deployments begin in 2027. Microsoft pairs the launch with a multi-gigawatt rollout that it intends to make the primary inference system for its own products and services. A gigawatt equals roughly the output of a typical nuclear reactor, so the plan describes power capacity on the scale of several reactors.
Microsoft already runs AI work on Nvidia GPUs. A chip of its own gives the company a second architecture to tune for that work, with lower cost per token as the design goal. The announcement is a plan with specifications attached. Microsoft has not said what share of its AI work Maia will carry, whether counted in requests, computing power, or money, and it names no outside customers.
Figure 1. Maia 300 sizes prefill and decode workers independently.
Maia 300 is the third generation of Azure Maia. Maia 200 runs Microsoft Copilot and supports MAI, Microsoft’s in-house model team, and Microsoft schedules GitHub Copilot and Foundry, the service that businesses use to build on AI models, to follow by the end of this year. The chips run models from OpenAI, from Microsoft, and from open-source families, so a hardware change need not change the model a customer sees.
Microsoft aims Maia 300 at three workloads: reasoning over long documents or conversations, chats that need quick replies, and agents, which carry out multi-step tasks on a user’s behalf. All three lean on memory. A language model answers in two phases. Prefill processes the prompt and tends to demand heavy computation. Decode generates the reply one token, or word fragment, at a time and often waits on data fetched from memory. Longer contexts enlarge the state the chip must store, and agents trigger many calls in succession.
The silicon follows from that. TSMC, the Taiwanese chipmaker, builds the compute dies on its N2P process. Memory-controller and input-output functions sit on separate dies tuned for bandwidth, connectivity, and cost. Maia 300 also moves from a uniform memory organization to a non-uniform one across chiplets, so software can place data near the compute that uses it. Vector-engine instructions now target inference kernels such as softmax and low-precision data casting, and the chip adds enhanced MXFP4 and sub-3-bit storage formats with hardware conversion. Eight HBM4 stacks deliver 288 GB of capacity, 33% more than Maia 200, and 18 TB/s of bandwidth, 2.5 times Maia 200 and roughly twice the bandwidth per watt.
Table 1. Maia 300 key specifications.
Each tray holds four accelerators plus its own host processor, memory, storage, and cooling, which removes separate host trays. Six switch ASICs in three liquid-cooled switch trays connect all 96 accelerators in a rack. Direct links join accelerator pairs inside a tray, and a switched electrical fabric covers the rest of the rack. Optical second-tier switches join up to 128 racks into a superpod of 12,288 accelerators. One Ethernet-based fabric carries both scale-up and scale-out traffic, so the design needs no separate scale-out network card. A rack draws up to 167 kW, about 1.7 kW per accelerator once you count everything else in the rack. Liquid cooling removes more than 80% of the thermal load, and air handles memory and storage. Operators can connect to facility water or add a heat-exchanger unit for air-cooled data centers.
The software mirrors the hardware. Microsoft introduced its software-defined dataflow architecture with Maia 200: The control path stays separate from data-path execution, and software coordinates placement, movement, conversion, and synchronization. Tensor parallelism stays inside a rack. Expert and pipeline parallelism span racks. Microsoft sizes prefill workers and decode workers independently, and the key-value state moves over the scale-up fabric. A service that handles long documents gives more hardware to prefill, and a chat service gives more to decode. Schedulers can also draw on hot-spare capacity in another rack of the superpod without changing the kernels.
Figure 2. Maia 300 system-on-chip architecture. (Source: Microsoft)
Microsoft says a fully populated Maia 300 rack produces up to seven times the tokens of a Maia 200 rack and more than 2.5 times the performance per watt. The rack holds twice the accelerators, so the per-chip gain lands near 3.5 times. The memory figures explain much of it. Twice the accelerators at 2.5 times the bandwidth gives five times the aggregate memory bandwidth per rack, and compute rises by up to 36%. Decode waits on memory, so that profile fits the workload Microsoft targets. Microsoft also expects 40% more tokens per megawatt than leading GPU-based systems. If the estimate holds under comparable conditions, Maia produces seven tokens for every five at the same power budget. The Azure deep dive names Maia 200 as its only comparator, and Microsoft gives no test conditions for the GPU claim.
Settling the question takes matched tests. Teams would run the same models at the same numerical precision, the same context lengths, and the same number of simultaneous users, record response times and answer quality, and count hardware, networking, power, and operating costs. Microsoft has published no such results, no independent validation, and no measured dollars-per-token comparison. Silicon teams and ISVs should treat the 40% figure as an expectation.
Microsoft also stays committed to Nvidia-powered capacity through IREN, a data center operator. Last November, IREN announced a five-year contract worth about $9.7 billion to supply Microsoft with computing on Nvidia GB300 systems, supporting 200 MW of load. According to IREN’s annual report, Microsoft accepted the first of four phases in August. IREN targets delivery of the other three in the fourth quarter of this year, with contractual grace periods running to the start of the second quarter of 2027. Nvidia capacity may still be arriving when the first Maia 300s go in.
The Maia plan covers inference for Microsoft’s own products, such as Copilot. It says nothing about the GPUs that outside customers rent across Azure, Microsoft’s cloud, or the hardware that trains models. The plan tests Microsoft’s reliance on Nvidia in one workload and does not end it. Maia and Nvidia systems will run side by side, and no commitment quantifies Maia’s future share of Microsoft’s traffic.
What do we think?
Maia 300 targets a real constraint: decode-phase memory bandwidth today, and the 2.5× gain per accelerator addresses it directly. The 128-rack superpod also suits long-context and agent workloads. Every performance claim comes from Microsoft, and the 2027 date gives Nvidia more than a year to respond. CIOs should keep Azure inference planning flexible and vendor-neutral, and demand independent, matched benchmarks.
Inflection point. Maia 300 points toward an inflection point in AI infrastructure. Hyperscalers now design silicon for inference, the workload that grows with every user, and buy training hardware separately. Microsoft joins Google, Amazon, and Meta with its own accelerators, and a 12,288-accelerator superpod puts real scale behind the effort. It also shifts leverage in talks with Nvidia and memory suppliers. The inflection point arrives when a hyperscaler publishes matched cost-per-token results against Nvidia. Ignite may offer the first public evidence.
LIKE WHAT YOU’RE READING? TELL YOUR FRIENDS; WE DO THIS EVERY DAY, ALL DAY.