DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - GB300 NVL72, Huawei, MI355X, B200

Below top-k · 55

• implicationOpen engines get Day 0 support first, closed engines follow latec 0.80

  Because diagnosing TRT-LLM's hidden-size mismatch took until Day 9 and the patch took further days to merge, native SGLang and native vLLM will consistently beat TensorRT-LLM and ATOM to Day 0 readiness on new open models.

• mechanismAMD's gains came from replacing PyTorch fallbacks with real kernelsc 0.80

  The MI355X improvement came almost entirely from swapping PyTorch-native fallback paths for real AITER, Triton, TileLang, and FlyDSL kernels, with the largest jump arriving from the first post-baseline commit cleaning up low-hanging fruit.

• evidenceGB300 MTP gains came from rewriting the decode topology, not kernelsc 0.80

  The dramatic June 2nd jump for GB300 SGLang MTP came almost entirely from a decode topology rewrite — widening from EP=8 to EP=16, scaling prefill workers from one or two up to four to twelve per decode worker, and lifting concurrency from 16,384 to 21,504 — rather than from kernel or precision changes.

• mechanismNVL72 keeps MoE all-to-all on NVLink instead of the scale-out fabricc 0.80

  GB300's rack-scale advantage is fundamentally about scale-up domain size: 72 GPUs in a single NVLink domain let expert parallelism go wide enough to keep DeepSeek V4's MoE dispatch/combine traffic on NVLink rather than spilling onto slower scale-out fabric, while amortizing weight loads across more ranks.

• claimHuawei's CANN strategy is full-stack co-design with Chinese model releasesc 0.80

  Huawei open-sourced CANN in August 2025 to attract developers and chip away at CUDA dominance inside China. The strategy is to make Ascend competitive by shipping kernels, graph paths, quantization, and serving recipes in lockstep with every major Chinese model release.

• evidenceInterleaving CSA and HCA yields 50x KV cache reduction at 1M contextc 0.75

  By alternating CSA and HCA layers, DeepSeek v4 compresses the KV cache aggressively enough to achieve a 50x reduction at 1M context length.

• mechanismDeepSeek wrote custom batch-invariant kernels for RL determinismc 0.75

  To stabilize RL training, DeepSeek made computation deterministic by writing custom kernels — including split-KV attention forward, GEMMs, and MoE backward — that enforce a fixed reduction order regardless of batch size.

• mechanismMegaMoE schedules experts in waves to hide communicationc 0.75

  DeepSeek v4's fused MoE kernel splits experts into waves and schedules each wave separately, enabling finer-grained overlap between dispatch/combine all-to-alls and the Linear1/Activation/Linear2 chain than implementations that sync across all experts at operation boundaries.

• claimCUDA vLLM and SGLang worked out of the box on Day 0c 0.70

  On the latest Nvidia hardware, native vLLM and SGLang supported DeepSeek v4 Pro the moment the model dropped, demonstrating the strength of the open inference engine ecosystem.

• evidenceNvidia's own TensorRT-LLM failed at launch and required an outside patchc 0.70

  TensorRT-LLM did not work for DeepSeek v4 on release, and SemiAnalysis ended up fixing Nvidia's mHC kernel launch code before Nvidia engineers rebased and merged the patch.

• mechanismMTP helps memory-bound decode but hurts compute-bound large batchesc 0.70

  Multi-token prediction exploits compute slack during memory-bound decode, so it delivers big wins at low batch sizes. At high throughput with large batches, decode becomes compute-bound and the cost of running MTP outweighs the benefit of draft tokens.

• evidenceB200 vLLM throughput per provisioned MW rose 1.7x from pure softwarec 0.70

  At 50 tok/s/user, B200 with vLLM went from 300,000 tok/s/MW on Day 0 to nearly 500,000 by June 5th. Because B200's all-in power envelope is fixed at ~2.17 kW/GPU, the 1.7x jump is entirely a software gain.

• claimTokens per all-in MW is the right ROI metric for inference fleetsc 0.70

  Token throughput per all-in provisioned-utility megawatt is the best figure of merit at fleet scale because it captures PUE and datacenter overhead that raw per-GPU throughput misses.

• evidenceGB300 with MTP serves DeepSeek V4 at $0.156 per million output tokensc 0.70

  With MTP enabled, the GB300 leads across all interactivity levels analyzed. Cost per million output tokens hits $0.156 at 50 tok/s/user with 8k input and 1k output.

• caveatAMD is prioritizing ATOM over vLLM despite zero production usagec 0.70

  ROCm vLLM is lagging far behind CUDA vLLM, partly because AMD is refocusing on ATOM — an engine serving zero production tokens — instead of native vLLM, which many of their major customers actually use.

• claimDeepSeek V4 was the first major open model with Day 0 Huawei Ascend supportc 0.70

  DeepSeek V4 was the first major open model to receive first-class Day 0 support on Huawei Ascend, and part of DeepSeek's official API has been served on Huawei since launch.

• evidenceA year ago only CUDA worked on Day 0c 0.70

  When DeepSeek v3/R1 released, only Nvidia's CUDA stack worked on Day 0, making Huawei's current Day 0 readiness a sharp improvement over the prior generation.

• caveatDay 0 readiness is not the same as catching a moving targetc 0.70

  Unlike the biblical Goliath, Nvidia keeps shipping a new architecture every year and improving existing ones, so Huawei proving it can land a Day 0 stone is not the same as proving it can fell a constantly-evolving competitor.

• contextDeepSeek v4 abandons MLA for CSA and HCAc 0.65

  DeepSeek v4 walks away from Multi-head Latent Attention in favor of Compressed Sparse Attention and Heavily Compressed Attention, a design driven primarily by the goal of shrinking the KV cache.

• evidenceNaive MoE kernel spends ~50% of time on communicationc 0.65

  The DeepSeek v4 paper claims a 1.92x theoretical speedup for the fused MoE kernel over the naive version in the Flash configuration, implying the naive kernel was spending nearly half its time on Dispatch and Combine communication.

• evidenceSwitching MoE experts from FP8 to native MXFP4 unlocked the second big jumpc 0.60

  Once AMD got FP4 weight MoE working, expert-weight bandwidth improved, FlashMLA and the sparse-attention indexer moved off the Torch fallback onto TileLang, and HIP graphs were enabled.

• evidenceMI355X eventually overtook H200 at low interactivityc 0.60

  After AITER mHC kernels were introduced at every layer, the MI355X surpassed H200 performance on DeepSeek v4 Pro at lower interactivity levels for the first time.

• evidenceATOM also clawed back the frontier but still has zero production customersc 0.60

  ATOM expanded from a single conc=1 point to a full pareto frontier — some points beating H200 — after AITER fix #2916, FP4 experts moving to AITER's fused MoE, and batching support landing. Yet the engine still has zero production customers.

• caveatAMD distributed inference for DeepSeek v4 Pro still doesn't workc 0.60

  AMD SGLang and AMD vLLM distributed inference remained broken for DeepSeek v4 Pro at the time of writing, leaving a meaningful gap versus CUDA's out-of-the-box distributed support.

• evidenceGrouped FP4 MegaMoE GEMM delivered 3x in under a week on B300c 0.60

  On B300 with SGLang, switching to a grouped FP4 MoE GEMM that keeps experts resident and dispatches once, plus moving from EP8 to EP4, produced a 3x improvement in under a week.

• exampleHuawei's elegant fix for MTP benchmark variancec 0.60

  Benchmarking MTP is hard because acceptance rates vary across deployments. Huawei reports time per decode step rather than time per token, timing the full decode to coincide with the last MTP module and asking users to multiply by their own acceptance length.

• mechanismAscend separates matrix and vector cores with dual-master modec 0.60

  Ascend's AI Core splits matrix work (AIC, used for GEMM and FFN linear layers) from vector work (AIV, for activations, norms, reductions). Unlike a TPU MXU, both cores can independently load and run their own code in 'dual-master mode' rather than having the vector unit drive the matrix unit through messages.

• mechanismDedicated CCU offloads collectives from compute coresc 0.60

  Like TPU and Trainium, Ascend 950 adds a dedicated CCU communication engine that handles collectives with remote-read/reduce/local-write patterns. This lowers communication latency, reduces HBM traffic, and frees compute cores from comm orchestration that previously went through the AICore→AICPU→SDMA path.

• mechanismHuawei pushes sparse-attention planning onto the on-device AI CPUc 0.60

  In DeepSeek v4, the value-dependent scheduler stage for sparse attention and the LightningIndexer runs on the AI CPU instead of bouncing to the host, mirroring FlashInfer's host-side planning phase but keeping it on-device.

• implicationCSA and HCA complicate KV cache management in serving frameworksc 0.60

  The novel attention designs force serving stacks like vLLM to set logical block sizes that divide both compression rates and use page-size bucketing to avoid fragmentation across KV cache, compressor states, and indexer KV — each with different per-entry sizes.

• mechanismToken-granular write-ahead log makes rollouts resumablec 0.60

  DeepSeek built a token-level write-ahead log per generation request so any request preempted mid-prefill or mid-decode can resume without recomputation, making rollouts reproducible.

• mechanismMC² fused operators embed communication into compute kernelsc 0.55

  CANN's MC² operators, introduced in 2024, fuse communication and compute into a single kernel rather than running ordinary kernels alongside HCCL collectives. DeepSeek v4 decode uses MoeDistributeDispatchV2 and MoeDistributeCombineV2 MC² EP operators.

• caveatBatch invariance costs performance, offset by workload specializationc 0.55

  Enforcing deterministic reduction order rules out many fast algorithmic techniques, so DeepSeek recovers performance by specializing kernels to its specific matrix shapes and workload.

• contextDeepSeek v4 is another Chinese open-model release and was partly co-designed for Huawei Ascendc 0.50

  DeepSeek v4 continues the pattern of Chinese labs leading the open model landscape, and its architecture was co-designed in part for Huawei Ascend inference.

• exampleATOM started usable only at batch size of onec 0.50

  Early ATOM #650 hardcoded kv_cache[:1,...], pinning the KV cache to a single sequence slot so a second concurrent request had nowhere to live, limiting runs to batch size one until batching infrastructure landed.

• evidenceGB200 disaggregated inference recipe arrived fast and replicated 5x over B200c 0.50

  vLLM and Nvidia shipped a GB200 Dynamo vLLM disaggregated-inference recipe in srt-slurm quickly, and independent replication delivered up to 5x better results than a B200 run at lower interactivity.

• evidenceMTP arrived on Day 3 and boosted throughput at high interactivityc 0.50

  SGLang shipped the first DeepSeek v4 MTP support on Day 3, substantially improving throughput at higher interactivities by accelerating memory-bound small-batch decode.

• evidenceMI355X couldn't use the native FP4+FP8 checkpoint on Day 0c 0.50

  Because the native FP4 MoE + FP8 Attention checkpoint did not work on MI355X at launch, the only option was a full FP8 non-native checkpoint, likely reflecting the less mature ROCm ecosystem.

• evidenceMay 19th sweep retired the last fallbacks and extended the throughput frontierc 0.50

  FlashMLA moved from TileLang to Triton, the AITER FlyDSL FP4 MoE kernel landed, and fusions for hash-topk, radix attention, store-cache, WQA/WKV projection, and paged-compress shipped — pushing the concurrency sweep to 1024 and drawing in a previously empty high-throughput corner of the pareto frontier.

• implicationThroughput optimizations translate directly into power efficiencyc 0.50

  Because all-in utility power is fixed, the same optimizations that pushed the throughput frontier — MegaMoE grouped FP4 GEMMs, wider EP, FP4 weight paths, scheduler tuning — drop straight through to power efficiency.

• contextB200, B300, and MI355X hit the scale-up wall earlierc 0.50

  B200 and B300 in 8-GPU NVLink islands scaled out over InfiniBand hit the scale-up wall much earlier than GB300, and MI355X lags on both scale-up domain size and collective-stack maturity.

• contextSGLang's roadmap is about fusing small ops and CUDA-graphing decodec 0.50

  SGLang's optimization tracker walks DeepSeek V4 block by block with three goals: CUDA graph support for decode, piecewise graphs for prefill, and no runtime weight processing. The work amounts to replacing chains of small ops with fused kernels and making attention variants read caches in place.

• contextAscend 950 ships in PR and DT variants from a shared dual-diec 0.50

  Ascend 950 comes as 950PR (Prefill/Recommendation, lower cost, better cost-perf) and 950DT (Decode/Training, higher bandwidth and performance). Both share the same dual-die UMA architecture but are packaged with different memory.

• mechanismOn-device AI CPU keeps irregular control work localc 0.50

  Ascend's AI CPU is a device-side ARM64 unit with direct access to device memory that handles branchy control flow, dynamic shapes, and value-dependent scheduling metadata. Keeping this work local avoids host round-trips that cause latency and pipeline bubbles.

• evidenceCANN hides C4A Compressor and overlaps shared and routed expertsc 0.50

  CANN uses independent compute and communication operators on multiple streams, and tunes Cube/Vector allocation to avoid contention. Prolog, Compressor, and LightningIndexer can overlap, C4A Compressor can be hidden completely, and shared-expert computation can hide under routed-expert execution without slowing the routed path.

• mechanismHCA combines a sliding window with aggressively compressed KV entriesc 0.50

  HCA's KV cache pairs a sliding window of recent KV embeddings with compressed entries, where each entry merges key/value into one representation across m' tokens (m'=128 in DeepSeek V4 Pro).

• mechanismCSA uses lighter compression plus lightning-indexer sparsityc 0.50

  CSA applies the same compression scheme as HCA but at a lower rate (m=4), then runs sparse attention over the compressed entries by using a lightning indexer to pick which tokens to attend to, inheriting DeepSeek v3.2's sparse attention.

• mechanismMetadata streams precompute scheduler/tiling info once per passc 0.45

  Streams 145-148 carry AI CPU metadata ops that fire once per decode pass to precompute value-dependent scheduler and tiling metadata reused by later kernels, fully overlapped with AI Core compute and taking a tiny fraction of total time.

• evidenceATOM ran almost everything on fallback paths at launchc 0.40

  ATOM's FP4 MoE was forced onto Triton because AITER's fused_moe was broken on GFX950, and the mHC pre-projection had to be patched to Torch because AITER's kernel crashed, forcing eager execution.

• claimTRT-LLM now wins at high batch sizes but lags at high interactivityc 0.40

  After the dust settled, TensorRT-LLM delivers superior throughput at high batch sizes on DeepSeek v4, but it falls behind the open engines at higher interactivity levels.

• contextTRTLLM is faster on B200 at low interactivity but does not work out of the boxc 0.40

  B200 performance roughly tracks B300, with TRTLLM edging ahead at lower interactivity. Unlike CUDA vLLM and SGLang, TRTLLM does not run out of the box.

• contextvLLM's remaining DeepSeek V4 work is mostly surrounding systemsc 0.40

  vLLM has landed FP4 Indexer, initial MegaMoE, and Hopper support. Remaining work centers on a new Model Runner V2, pipeline parallelism, KV cache offloading, and broader hardware coverage rather than core model support.

• contextDecode steps split work across concurrent streamsc 0.40

  Within a single decode step, operations are distributed across multiple streams that run concurrently when resources allow, since layers contain branches that only need to synchronize at result-combination points like shared-expert overlapping with routed-expert compute.

• mechanismTail work pushed to a separate stream so the main graph completes earlyc 0.40

  Stream 152 holds the LM head, last layer, and second-to-last layer's o_proj and MoE, a choice by the npugraph_ex compiler that lets the runtime treat the main graph as complete on stream 144 while tail work continues asynchronously.

• evidencevLLM and SGLang are commercially important enough to spawn well-funded companiesc 0.30

  The teams behind vLLM and SGLang have founded Inferact and RadixArk respectively, each raising hundreds of millions to grow their open-source inference engines.