DeepSeek released DeepInfer, the inference framework it uses to run its models in production, on 20 October under the Apache-2.0 licence, Crypto Briefing reported. The Chinese AI lab uses it for DeepSeek-V4.1-Flash and has tuned it for Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. The release follows DeepSeek’s partnership with Huawei on programming tools for Ascend chips.
DeepInfer combines chip-specific routines with ways to divide work across processors. Its speculative decoding feature guesses several upcoming tokens and checks them before accepting them, allowing output to arrive faster when the guesses are right. Preparing a batch of written summaries could benefit from faster processing if the reported throughput carried over to that work. The framework’s main target is offline inference rather than a live exchange.
For DeepSeek-V4.1-Flash, DeepSeek reports throughput of 155 trillion tokens per day per cluster, a 96% input cache hit rate and an average wait of around 400 milliseconds for the first token. DeepSeek puts output speed at 230 tokens per second, with coding tasks running at up to 300 tokens per second. DeepSeek also released deepseek-recipe, a tool for converting text to tokens and back that is designed to match the logic of its official API.
