DeepSeek Chooses OmniDocBench: Our Benchmark Just Became the Industry Standard
The new DeepSeek-OCR paper validates its state-of-the-art performance using OmniDocBench, a benchmark developed by our partner 2077AI. As a core contributor, Abaka AI‘s advanced data construction and annotation expertise was foundational in building this complex, industry-standard dataset. This citation from a leading team like DeepSeek is a direct validation of our capability to produce the high-fidelity, high-difficulty data required to train and evaluate the most advanced AI models.
In the fast-paced world of AI research, validation is everything. How do you prove that your new, groundbreaking model is truly state-of-the-art? You test it against the hardest, most comprehensive benchmark you can find.
That’s why we’re thrilled to announce that DeepSeek-AI has selected 2077AI's OmniDocBench as a core evaluation benchmark for their revolutionary new paper, "DeepSeek-OCR: Contexts Optical Compression."
This isn't just a casual citation; it's a powerful statement. When a leading team like DeepSeek needs to prove the efficiency and power of their novel architecture, they turn to OmniDocBench. This confirms what we knew when we built it: OmniDocBench is quickly becoming the new industry standard for Document AI evaluation.
The Challenge: Why DeepSeek Needed OmniDocBench
DeepSeek's latest paper introduces a fascinating concept: using "optical 2D mapping" to highly compress long text contexts. Their goal is to create an OCR model that is not only highly accurate but also incredibly efficient, using a minimal number of "vision tokens."
This created a specific challenge:
- They needed to prove their model was accurate on complex, real-world documents.
- They needed to prove their model was efficient, outperforming others that use far more resources.
This is precisely the problem OmniDocBench was designed to solve.
As the DeepSeek paper states in its abstract:
"On OmniDocBench, it surpasses GOT-OCR2.0 (256 tokens/page) using only 100 vision tokens, and outperforms MinerU2.0 (6000+ tokens per page on average) while utilizing fewer than 800 vision tokens."
This single chart tells the whole story. DeepSeek-OCR achieves top-tier accuracy (low Edit Distance) while living in the "low token" zone on the right. In contrast, many other powerful models like MinerU2.0 are clustered on the far left, requiring over 6,000+ vision tokens to perform at a similar level.
OmniDocBench was the only benchmark that provided the granularity and real-world complexity needed to demonstrate this groundbreaking efficiency.
What Makes OmniDocBench the New "Proving Ground"?
Why did DeepSeek choose OmniDocBench over simpler, older benchmarks? Because Document AI in 2025 is about more than just reading clean, single-column academic papers.
The real world is messy. Our benchmark, OmniDocBench, was built to reflect such reality.
- Unmatched Diversity (9 Types): While other benchmarks focus on one or two document types, OmniDocBench includes 9, featuring notoriously difficult categories like:
- Financial Reports (dense tables)
- Handwritten Notes (messy text)
- Newspapers (complex multi-column layouts)
- Textbooks (intricate formulas and diagrams)
- Exam Papers & Slides
- Fine-Grained Evaluation: We don't just give a single pass/fail score. OmniDocBench provides multi-level evaluations across 19 layout categories and 15 attribute labels. This allows researchers to pinpoint exactly where their model excels or fails—whether it's on tables, formulas, or rotated text.
- Real-World Complexity: Our benchmark tests everything from layout analysis and OCR to table recognition and reading order estimation, providing a holistic "acid test" for any true end-to-end Document AI model.
Its diversity brings complex evaluation challenges, as different models have different strengths. Our own benchmark results clearly demonstrate this:
As the chart shows, no single model can conquer all categories. Some (like MinerU-0.9.3) excel at academic papers but fail at notes, while others have different profiles. This is why a simple "overall score" on a single document type is no longer enough to measure true capability.
How an Industry Standard is Built
Crafting a benchmark for this complex and reliable is a massive engineering feat. It requires a systematic, multi-stage process to ensure every data point is accurate.
Our three-stage process—Intelligent Annotate, Annotator Correct, and Expert Quality Inspection—ensures that the final benchmark is precise, comprehensive, and ready to challenge the world's best models. We’re incredibly proud of this validation and congratulate the DeepSeek-AI team on their fantastic research. This achievement highlights a core philosophy of 2077AI: great AI is built on great data.
The creation of complex, high-fidelity benchmarks like OmniDocBench is a massive undertaking. It is powered by the cutting-edge data construction and annotation capabilities of our entire ecosystem, including our core contributors to Abaka AI. This validation from DeepSeek proves that the advanced data engineering pipelines we've built are setting the new industry standard.
2077AI and our partners will continue to build high-quality benchmarks that push the entire field of artificial intelligence forward.
🔗 Explore Research & Data
- OmniDocBench Tech Blog: https://www.2077ai.com/blog/OmniDocBench
- OmniDocBench Paper: https://arxiv.org/abs/2510.11652
- Hugging Face Dataset: https://huggingface.co/datasets/Quivr/OmniDocBench
- DeepSeek-OCR Paper: https://arxiv.org/html/2510.18234v1