DeepSeek releases DeepInfer inference framework as open source

DeepSeek has released DeepInfer as open-source software, making public the inference framework that the company says powers its large-scale production services. The Apache 2.0-licensed package supports offline inference for DeepSeek-V4.1-Flash, but it is not a complete online serving system.

The package omits three systems that the repository says a complete inference service requires: a key-value cache, disaggregation across virtual engines and the prefill and decode stages, and load balancing across data-parallel replicas for both prefill and decode. Those service-level components remain outside the released code.

DeepInfer currently lists one supported model, DeepSeek-V4.1-Flash, and uses the 1017 checkpoint in its default offline-inference example. DataPhoenix previously covered DeepSeek’s V4.1 Flash 1017 refresh. The supported accelerators are NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Building the software requires Linux, GCC 13 or newer and Python 3.12, along with CUDA 13 for NVIDIA hardware or CANN 9.2 for Ascend.

The published code includes optimized kernels, model and sampler runtimes, attention and mixture-of-experts parallelism, asynchronous scheduling and structured-output support. It also provides DSpark draft-token decoding with confidence-scheduled verification and SWA bounded replay. DeepSeek marks batch-invariant inference and deterministic sampling as NVIDIA-only capabilities.

DeepSeek says the package contains the core features needed for serving at scale, even though the repository describes it as offline-inference code. The company says most of the engine functionality required for the omitted disaggregation layer is already implemented, while the service-level orchestration remains outside the release.

The repository presents company-run performance benchmarks from one scale-up domain configured with four SM100 GPUs for prefill and 64 SM100 GPUs for decode. The official repository provides no quantified cost saving or dollar-denominated serving-cost comparison for DeepInfer.

DeepSeek says it built DeepInfer primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. The code is available under the Apache License 2.0.