DeepInfer: DeepSeek open-sources the engine that runs its AI models
Representative image: kuszapro / Pixabay
DeepSeek released DeepInfer, the inference engine behind its own AI service, as open source on October 20, 2026; it supports one model for now.
What Happened
DeepSeek, the Chinese artificial intelligence company, released DeepInfer, the software it uses to run its own AI models for paying customers, as free open-source code on Tuesday, October 20, 2026. DeepInfer is an "inference framework": the engine that takes a trained model and serves its answers to users quickly and cheaply. The company published it on GitHub under the Apache 2.0 licence, which lets anyone use, change and build on it, including for commercial products. DeepSeek said it was sharing its production stack to help others lower computing costs and energy use.
Key Facts
- Release: DeepInfer, DeepSeek's in-house inference framework, open-sourced on October 20, 2026, under the Apache 2.0 licence.
- Model support: for now it runs only DeepSeek-V4.1-Flash, the model DeepSeek serves through its own services.
- Hardware: Nvidia SM100-generation GPUs and Huawei's Ascend 950 chips.
- Origins: DeepSeek says it was first built mainly on vLLM, a popular open-source serving engine, with ideas from SGLang and TensorRT-LLM.
- Company claims: one cluster serves about 155 trillion tokens a day, with a 96% input-cache hit rate, 400 milliseconds average time to first token and 230 tokens a second of output.
- Company claim on earnings: about $38 per GPU-hour at its off-peak price; outside analysts have estimated this implies a gross margin of roughly 80%, which is their estimate, not DeepSeek's.
- Gaps: the release leaves out key layers, such as the system for storing and reusing earlier work (the KV cache), splitting work between stages, and load balancing, which users must supply.
- Independent check: the vLLM team reported decode speeds within 6% of DeepInfer's published results in a matched set-up.
Why It Matters
Training an AI model gets most of the attention, but running it every day for millions of users is where much of the money goes. Every reply a chatbot gives is "inference", and its cost depends on how well the serving software uses expensive chips. A "token" is a small piece of text, roughly a word or part of a word, so the number of tokens served is a measure of how much work the system does. By publishing the engine that runs its own production service, DeepSeek is giving other companies and researchers a detailed look at how it keeps those costs low.
The numbers need care. The throughput and earnings figures are DeepSeek's own and have not been audited. One analysis points out that the 155 trillion tokens a day includes a very high share of cached input, text the system has already processed and can reuse, so it is not the same as 155 trillion newly generated tokens. The roughly 80% margin is an outside estimate based on assumed rental prices for chips. The vLLM team's result, coming within 6% of DeepInfer in a matched test, suggests the advantage lies less in secret technology than in careful engineering on top of open tools.
The release also has a strategic side. It supports Huawei's Ascend 950 chips alongside Nvidia's, which matters to Chinese firms facing US export limits on advanced Nvidia hardware. It came one day after the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, a move Beijing objected to. DeepSeek has not linked the two.
For India, where start-ups, research labs and government-backed projects are building and serving their own large language models, open serving software of this kind can reduce the cost of offering AI services in Indian languages. Any team using it would still need to build the missing pieces, check its security, and judge whether a model from a Chinese company fits its data and compliance rules.
Impact
Short-term: AI engineers and cloud providers are testing DeepInfer and comparing it with existing engines such as vLLM and SGLang. Because it supports only one model today, its immediate use is limited to teams running DeepSeek-V4.1-Flash.
Long-term: If DeepSeek widens support to more models and adds the missing layers, DeepInfer could push down the price of running large models across the industry and add pressure on companies that sell AI access at higher prices.
Who is affected: AI start-ups and cloud firms that serve models; chip makers Nvidia and Huawei; rival AI labs; developers in India and elsewhere building AI products; and governments weighing the security of Chinese AI software.
Key Takeaway
DeepSeek has open-sourced DeepInfer, the engine behind its own AI service, under a permissive licence, though for now it supports one model and its striking cost figures are the company's own claims.
Questions and Answers
What is DeepInfer?
DeepInfer is DeepSeek's in-house inference framework, the software that runs its AI model to answer users' requests. DeepSeek released it as open source on October 20, 2026, under the Apache 2.0 licence.
Can anyone use DeepInfer for free?
Yes. The Apache 2.0 licence lets anyone use, change and build on the code, including commercially. Users still pay for the computing hardware and must add parts DeepSeek did not release, such as the cache and load-balancing layers.
Which models and chips does DeepInfer support?
For now it supports only DeepSeek-V4.1-Flash, running on Nvidia SM100-generation GPUs and Huawei Ascend 950 chips.
Are DeepSeek's DeepInfer performance numbers verified?
Not fully. The figures of about 155 trillion tokens a day and $38 per GPU-hour are DeepSeek's own. The vLLM team independently reported speeds within 6% of DeepInfer's published results in a matched test.
Disclaimer: Prepared by the Peepals newsroom from publicly available sources with AI assistance. Information is accurate to the best of our knowledge at the time of publication and may change. Images may be representative. Not professional advice. Report an error via our contact page.
Sourced and fact-checked by the Peepals Global Editorial Team
ISRO clears Gaganyaan crew and service module separation tests before G1 flight
57 Manipur TechBee trainees join HCLTech; state signs AI skilling MoU with Microsoft
Reported, fact-checked and published by the Peepals Global Editorial Team.
