DeepSeek DeepInfer: What Was Open-Sourced and Who Can Run It

DeepSeek’s October 20 post announcing DeepInfer passed 3.8 million views on X in under a day. It’s easy to read the news as “now you can run DeepSeek’s production stack yourself.” That reading is half right. DeepInfer is the inference engine that serves DeepSeek-V4.1-Flash-1017 in production, and the code is public under Apache 2.0. But the repository says plainly that it ships offline inference, targets two accelerator families, and leaves out the KV cache system and the load balancers that make a real service work.

This post is for infrastructure engineers deciding whether DeepInfer changes their plans for serving V4.1-Flash-1017. It covers what is in the repo, what is not, the hardware floor DeepSeek itself describes, and the production numbers it disclosed. Everything here comes from the DeepInfer repository and its deployment Q&A as of October 21, 2026. I have not built or benchmarked DeepInfer; I don’t have SM100 or Ascend 950 hardware to run it on.

Key takeaway

• DeepInfer is DeepSeek’s in-house inference framework, open-sourced on October 20, 2026 under Apache 2.0. Its only supported model today is DeepSeek-V4.1-Flash, and its default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017.
• Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It needs Linux, Python 3.12, GCC 13+, and CUDA 13 or CANN 9.2.
• The release is offline inference. A KV cache system, prefill/decode disaggregation, and data-parallel load balancing are listed as “not included.”
• DeepSeek’s Q&A puts the smallest efficient production unit at 64 prefill GPUs plus 64 decode GPUs, and says one cluster serves 155 trillion tokens a day.
• For a single 8-GPU node, the examples README gives --ep 8 for offline inference. Mac and DGX Spark demos circulating on X use other engines, not DeepInfer.

What DeepSeek actually released

The README opens with a scoped claim: DeepInfer “powers DeepSeek’s large-scale production services,” and “this open-source release provides offline inference with the core features needed for serving at scale.” The word that matters is offline. You get an engine you can drive from Python against a batch of prompts. You do not get an HTTP server with routing, caching, and autoscaling in front of it.

The README scopes the release in one screen: one model, two hardware families, and three service components “not included in this release” (captured 2026-10-21).

The feature list is still substantial, and it is the part worth reading even if you never run the code:

Feature	What the repo says it does
Optimized kernels	FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, custom TileLang kernels
Parallelism	Attention data parallel or balanced context parallel; MoE expert parallel with EPLB
DSpark speculative decoding	Draft tokens plus confidence-scheduled verification; default max draft length 5
SWA bounded replay	Cuts decoder prefill by replaying only a bounded suffix of tokens
Asynchronous scheduling	Overlaps CPU batch scheduling with GPU execution; on by default
Batch-invariant inference	Same results regardless of batch composition (NVIDIA only)
Deterministic sampling	Same output for the same input tokens and seed (NVIDIA only)
Structured outputs	Regex, JSON Schema, or EBNF constraints

Two of these stand out for anyone who has fought with evaluation noise. Batch-invariant inference (DEEPINFER_BATCH_INVARIANT=1) and deterministic sampling (DEEPINFER_ENABLE_MTP_VARLEN=0 plus a fixed seq_seed) mean the same prompt can produce the same tokens whatever else shares the batch. That is the property you want when a regression test has to tell a model change apart from scheduling jitter. Both are NVIDIA-only, and there is a trade-off: deterministic mode turns off DSpark’s variable-length verification, so you give up some decode speed for repeatability.

The README also credits its lineage: DeepInfer “was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.” Read it as DeepSeek’s production fork of familiar ideas, not a clean-sheet engine.

What is missing, and why it matters

Three things are listed as “required for a complete inference service (not included in this release)”:

1. A KV cache system. No persistent prefix cache across requests.
2. VE/Prefill/Decode disaggregation. The README notes “most required engine functionality is already implemented,” but the orchestration around it is not shipped.
3. Load balancing across data-parallel replicas for both prefill and decode.

The deployment Q&A explains the omission: the KV cache management and load-balancing components “are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced.” DeepSeek points to a Zhihu post for the design instead.

This is not a small gap. The KV cache is where DeepSeek’s cheap cached-input price comes from. DeepSeek’s pricing page lists deepseek-flash cache-hit input at $0.006 per million tokens at peak, against $0.30 for a cache miss. The Q&A describes the machinery behind that: a dedicated 3FS storage cluster with roughly 100 PiB of SSD for KV cache, a TTL of 3 days for agent traffic and 7 days for chatbot traffic, and about 14.4 TB of logical cache per online GPU. None of that ships in the repo. If you self-host with DeepInfer alone, you will not get that price structure for free; you have to build or bring the cache tier.

The production numbers DeepSeek disclosed

The Q&A that ships in doc/ is the most informative part of the release. It opens with two figures that explain why people shared it so widely:

DeepSeek’s own deployment Q&A: 155 trillion tokens per day from one cluster, US$38 revenue per GPU-hour at off-peak (“valley”) pricing, and the note that the KV cache and load-balancing code stay internal (captured 2026-10-21).

• 155 trillion tokens per day from a single cluster serving V4.1 Flash.
• US$38 of revenue per GPU-hour at valley (off-peak) pricing.

Both are DeepSeek’s statements, not audited figures, and the revenue number is not a cost figure. It tells you what the service earns per GPU-hour at off-peak rates. It does not tell you DeepSeek’s hardware, power, or staffing cost, so it cannot settle the “is DeepSeek pricing below cost” debate on its own. What it does show is that a cheap per-token price and healthy per-GPU revenue are compatible when utilization and cache hit rates are high.

The Q&A also gives the production design in enough detail to copy the ideas:

• Disaggregated E/P/D. Vision encoder, prefill, and decode run as separate pools. Prefill uses EP2–8 on ordinary 8-GPU machines; decode uses “Wide-EP” at EP64–128 on supernodes. Tensor and context parallelism are not used in production.
• Four load balancers. Prefill, decode, vision encoder, and expert-parallel. The prefill balancer buckets requests into four context tiers (<16K, 16–64K, 64–256K, ≥256K). DeepSeek says the system reaches about 80% of a perfectly balanced offline benchmark.
• Low precision. Main KV is stored in a per-16 e4m3 format, close to NVFP4 without the global scale. Routed-expert weights and indexer KV use MXFP4.
• DSpark gains are modest. Dynamic-length verification cuts verified draft tokens per step by about 30% and adds 4–7% throughput at the production operating point (~60 concurrent sequences per rank). That is a solid engineering result, but a modest one.

The last point deserves weight. It’s tempting to read DeepInfer as the secret behind V4.1 Flash’s speed. DeepSeek’s own table shows speculative-decoding tuning worth single-digit percentages at the operating point. The Q&A doesn’t rank what contributes most, but my read is that the bigger levers are the model’s compressed KV cache, Wide-EP decode, and a cache tier that avoids repeat prefill. The engine matters, but it is one layer among several.

The hardware floor

Here is the sentence that should end most home-lab plans:

The Q&A says one prefill plus one decode instance is deployable, but because decode is recommended at EP64 or above, the smallest efficient unit is 64 + 64 GPUs (captured 2026-10-21).

“1P + 1D is deployable,” the Q&A says, but since recommended decode is EP64 or above, “the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode.” That is DeepSeek’s efficiency floor for a production service, not a hard minimum to load the weights.

For evaluation, the floor is much lower. The examples README runs offline inference on a single node:

# One 8-GPU node, expert parallel across all 8 GPUs
export DEEPINFER_ACCELERATOR=nvidia   # or "ascend"
./build.sh
export WORLD_SIZE=1 RANK=0
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
python examples/offline_inference.py --ep 8

That downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 into the Hugging Face cache on first run, or uses a local copy if you set DEEPINFER_HF_MODEL_DIR. The examples also include --ep 2 --dp 8 and --ep 2 --cp 8 layouts and a benchmark.py that reports prefill throughput, decode steps per second, and decode tokens per GPU.

The hardware list is the other hard constraint. SM100 is NVIDIA’s data-center Blackwell generation; the README does not list Hopper, consumer GPUs, Apple silicon, or AMD. That matters because two local V4.1 Flash demos posted on X this week ran elsewhere. One post, with over 220,000 views, showed the model “this fast on 2 DGX Sparks.” It went up at 03:48 UTC on October 20, almost six hours before DeepInfer was announced, so it can’t have used it. Another ran it on a Mac Studio M5 Ultra with an LSQ3 quant through the oMLX engine and reported 63.8 tok/s decode. Neither tells you what DeepInfer does on its target hardware. They show that the MIT-licensed weights travel well; they say nothing about this engine.

Who should use DeepInfer

You are…	DeepInfer today	What to do instead or alongside
A team with SM100 or Ascend 950 nodes serving V4.1 Flash	Strong candidate for the engine layer	Build or bring your own KV cache tier and request routing
Evaluating V4.1-Flash-1017 on one 8-GPU node	Usable for offline batch runs and reproducible evals	Use batch-invariant mode for regression tests
Running on Hopper, AMD, or consumer GPUs	Not supported per the README	Use your existing engine and the open weights
Running on a Mac or DGX Spark	Not supported	Community engines and quants, as in this week’s demos
Needing an OpenAI- or Anthropic-compatible endpoint	Not provided by DeepInfer	Look at deepseek-recipe for protocol encoding, or use a hosted API

The companion release, deepseek-recipe, fills one piece of that last row. It converts Chat Completions, Responses, and Messages requests into DeepSeek’s token format and parses outputs back, in Rust with Python bindings. DeepSeek says its logic “is fully aligned with the official DeepSeek API.” It also guards against special-token injection: strings such as <｜User｜> inside user input are encoded as ordinary text, not control tokens. If you serve the open weights behind your own API, that is the piece worth adopting first, whichever engine you choose.

For most developers, though, the practical path to V4.1-Flash-1017 is still an API. On SandBase it is deepseek/deepseek-v4.1-flash, listed at $0.30 input and $1.20 output per million tokens, the same as DeepSeek’s peak rate. That is a gateway to the provider’s model, not a SandBase deployment of DeepInfer. The Python tutorial shows tool calls and JSON output against that id, and the Claude Code test runs it as a coding agent.

What to watch next

DeepSeek says inference code “for future DeepSeek models, along with subsequent performance work, will continue to be open-sourced in DeepInfer.” When I checked on October 21, the public history had four commits: the initial commit, two Q&A doc updates, and a DeepEP dependency bump. Watch for three things: support for a second model (a V4.1 Pro would be the obvious test), any release of the KV cache or balancer code, and hardware beyond SM100 and Ascend 950. Each would change the self-hosting math more than another speed chart.

FAQ

What is DeepSeek DeepInfer?

DeepInfer is the inference framework DeepSeek uses to serve DeepSeek-V4.1-Flash in production. DeepSeek open-sourced it on October 20, 2026 under the Apache 2.0 license. The public release covers offline inference with optimized kernels, expert parallelism, DSpark speculative decoding, structured outputs, and deterministic modes.

Can I run DeepInfer on a consumer GPU or a Mac?

Not according to the README, which lists only NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Viral demos of V4.1 Flash on DGX Spark and Mac Studio used other engines and quantized weights.

Is DeepInfer a drop-in replacement for vLLM or SGLang?

Not yet. It is built largely on vLLM, but the release is offline inference only. It does not ship a KV cache system, prefill/decode disaggregation orchestration, or data-parallel load balancing, which a production server needs.

How many GPUs does DeepInfer need?

The examples run on one 8-GPU node with --ep 8 for offline inference. For an efficient production service, DeepSeek’s Q&A puts the smallest unit at 64 prefill GPUs plus 64 decode GPUs.

Does DeepInfer explain why DeepSeek’s API is so cheap?

Partly. The Q&A describes KV cache compression, Wide-EP decode, and a long-retention KV cache on about 100 PiB of SSD, without ranking them. The cache tier is the piece behind the low cache-hit price, and it is not in the open-source release.

Bottom line

DeepInfer is a real release with real engineering inside it, and the deployment Q&A is one of the most detailed public accounts of a frontier-scale serving system. It is not a turnkey way to run DeepSeek-V4.1-Flash-1017 at DeepSeek’s prices. You get the engine; you do not get the cache tier, the balancers, or support for hardware outside SM100 and Ascend 950. If you own that hardware, start with the single-node offline example and the batch-invariant mode. If you don’t, read the Q&A for its design ideas and call the model through an API such as deepseek/deepseek-v4.1-flash on SandBase while the open-source stack fills in.