Here's something that probably cost you more time than you'd admit: feeding a compliance report into an AI model and watching it hallucinate half the table data. Happened to me last week with a SOC 2 evidence package from one of our cloud providers. The model confidently asserted that three access control findings applied to a different audit period entirely. Three months of remediation work, nearly wasted.
The problem isn't the model. It's the document format. PDFs, HTML pages, even Markdown files — they were built for humans staring at screens, not for machines digesting tokens. And when you're a security & compliance analyst drowning in thousands of documents across your 365 environment, that translation layer between human-readable format and AI-ready structure becomes a costly, brittle bottleneck.
Enter DocLang. The LF AI & Data Foundation — under the Linux Foundation umbrella — just formed a working group to develop this open-standard document format optimized specifically for AI tokenization and semantic parsing. IBM, NVIDIA, Red Hat, ABBYY, HumanSignal, and Forgis are founding members. Early benchmarks show 4x to over 30x cost reductions depending on the model.
Let's get into why this matters for your day-to-day work, and where I think the coalition might stumble.
The PDF Problem Nobody Wants to Talk About
I'll be blunt: a PDF is not an API. It's a rendering instruction set designed to make text look consistent across monitors and printers, not to preserve semantic meaning for machines.
When you throw a PDF into an AI pipeline, the model has to reverse-engineer structure from visual layout. What's a heading versus body text? Which table cell belongs where? Which paragraph flows into the next? It's impressive what modern OCR and parsing tools can do, but it's inherently noisy. Every time a PDF enters an AI pipeline, structure, meaning and layout get lost — as Jon Knisley of ABBYY put it to The Register. "The model's accuracy ends up bottlenecked by document quality rather than model quality."
Think about a typical compliance report from your cloud provider. The document looks perfect to you — well-formatted tables, clear section breaks, maybe even complex financial charts. But feed that PDF into your compliance analyzer and the model starts hallucinating:
- A table of access logs gets flattened into a single block of text, losing column relationships entirely
- Section headers become body text, so your model can't tell which findings apply to which control framework
- Complex formulas in financial annexes turn into garbled text strings the model struggles to parse
That's why companies end up building custom parsers for every integration point. One pipeline for vendor contracts, another for incident reports, yet another for audit logs. Each one introduces a new place where things break, adds latency, and multiplies the hallucination surface area.
It's not just about accuracy either. It's about cost. The LF AI & Data Foundation points out that parsing a PDF with an AI model can require 1,200 input tokens plus 150 output tokens as a baseline on a single document. Scale that to thousands of documents across your security stack and you're burning serious compute cycles deciphering layout instead of extracting meaning.
"PDFs were designed for rendering, not understanding," Knisley said. "With DocLang, customers can expect better accuracy, lower costs, fewer tokens consumed, faster performance and more consistent outputs."
What DocLang Actually Is (And Isn't)
DocLang isn't trying to replace PDFs. It's trying to replace the thinking that PDFs force you into.
The specification — open source and free to implement — uses a minimal XML vocabulary designed specifically for LLM tokenizers. Here's what makes it different from the usual format-wars nonsense:
- 1-to-1 mapping between DocLang elements and LLM tokens, making prompt construction deterministic
- Lossless representation of structure, layout, metadata, and provenance information
- Built-in support for complex graphical elements like tables, formulas, charts, and multimodal content
- Native support for governance data, keeping document lineage attached through each transformation
The spec draws on IBM's open-source Docling toolkit, which it developed in late 2024 to help organize AI document parsing. Docling provides a way to convert various file formats into structured, AI-ready data — similar in spirit to Microsoft's MarkItDown or the Marker project. DocLang takes that foundation and adds a standard for exchanging structured output across systems, making it more than just an IBM-specific tool.
"DocLang is designed to solve one of the foundational problems in enterprise AI: documents were built for humans, not machines," said Maxime Vermeir, VP of AI Strategy at ABBYY. "By introducing a minimal, standardized, and AI-native representation of document structure, layout, meaning and governance, DocLang creates a far more deterministic foundation for modern AI systems."
The ABBYY Interactive Benchmark demonstrates the potential token savings pretty clearly. A PDF of IBM's 2025 annual report produces 8,421 input tokens and 512 output tokens. The DocLang version requires only 5,310 input tokens and 498 output tokens. Latency drops from 4.2 seconds to 2.7 seconds. And the AI missed one subsection and mangled a table merger in the PDF version — no major errors with DocLang.
That's a 37% reduction in input tokens alone. The difference compounds dramatically when you're ingesting hundreds or thousands of documents.
Why Security & Compliance Analysts Should Pay Attention Now
Most coverage around DocLang focuses on enterprise efficiency. But for security and compliance teams, the implications go deeper than cost savings.
Reduced hallucination risk in critical analyses
Compliance analysts parsing SOC 2 reports, security researchers reviewing vulnerability disclosures, or threat hunters analyzing incident logs all need high-fidelity data. When a model misreads a table of access controls or mixes up findings across different audit periods, the consequences can be serious. DocLang's lossless representation means your security AI knows exactly where each element lives — no more guessing whether a table cell belongs to the "Read" column or "Write" column. That's not just about accuracy; it's about trust in your AI-driven security conclusions.
Preserved provenance for audit trails
Most document formats strip metadata when converted or passed between systems. DocLang keeps that information attached, including who created the document, when it was modified, and what version of a framework it references. For security professionals who need to trace findings back to source evidence, this is genuinely useful. You can prove not just what your AI found, but where it found it — even when that finding traces back to a PDF you received from a third-party vendor three months ago.
Better context for model inference across your 365 environment
When an AI can reliably parse that a section header, table data, and footnote reference belong together as a single logical unit, it dramatically improves the model's ability to reason across documents. Instead of treating each page as a separate blob of text, the model understands that "Section 5.2" on page 42 connects to "Appendix B" on page 178. That contextual awareness becomes crucial when you're training security models on historical incident reports or looking for patterns across compliance frameworks spanning your entire 365 environment.
The Coalition Behind It — And Why That Matters
The founding members read like a who's-who in enterprise AI infrastructure:
- IBM: Provided the initial Docling toolkit foundation and enterprise use case validation
- NVIDIA: Supporting GPU-optimized parsing pipelines for the new format
- Red Hat: Ensuring open-source compatibility and ecosystem integration
- ABBYY: Leading the security and compliance use cases, plus building the interactive benchmark
- HumanSignal: Contributing expertise in data labeling and AI training pipelines
- Forgis: Bringing cloud-scale document processing experience
The specification is openly available, and the working group actively invites additional technology providers and enterprises to join.
"It's still early, and we won't overstate adoption," said Knisley. "The standard is open and free to build on, and the group is actively inviting more technology providers and enterprises to join. The early response has been encouraging, and we're optimistic about where it goes from here."
That's a cautiously optimistic stance — understandable given the history of document format standards that promised much but delivered little. But this time feels different for two reasons: the motivation is clear (enterprises are losing money and accuracy to brittle document parsing), and the standard is grounded in real, working tools rather than theoretical frameworks.
Your Move: Preparing Without Waiting for the Final Spec
You don't need to wait for the final spec to start thinking about AI-native documents. Here's where I'd focus if I were guiding a security team through this transition:
Inventory your document intake points
Map out every place where documents enter your AI pipelines: vendor risk assessments, compliance reports (SOC 2, ISO 27001, etc.), security bulletins and threat intelligence, internal audit findings, incident response documentation. How many of these currently come as PDFs? How often do you need custom parsers for each source?
Pilot a DocLang converter now
Even though the spec is still emerging, tools like Docling already support conversion to AI-ready formats. Test feeding your high-priority documents through a DocLang pipeline and compare hallucination rates, latency, and token usage against your current approach. You'll likely find that the benefits kick in fast — even with a partially implemented spec, the structural clarity alone improves model performance.
Engage your tool vendors
Ask your security automation, compliance management, and AI document processing vendors about their DocLang roadmap. The LF AI & Data Foundation is openly inviting contributors, so vendor adoption will likely accelerate once customers start asking. The more pressure you put on vendors to support DocLang, the faster the ecosystem will shift.
Build governance around your AI-native documents early
With DocLang, provenance data travels with each document. That's a double-edged sword: you can track lineage with unprecedented precision, but you also need to define who owns document versions and how governance policies flow through the pipeline. Start thinking now about version control for AI-ready documents, audit trails that survive format conversion, and metadata standards for security-related documents.
Bottom Line
The LF AI & Data Foundation's DocLang initiative represents a rare convergence of real technical need, tangible cost savings, and serious industry backing. It's not about making documents look better for AI — it's about making them speak the right language in the first place.
For security and compliance teams, this could mean moving from reactive, error-prone AI analysis to proactive, reliable insights that you actually trust. The transition won't happen overnight. PDFs aren't going away tomorrow, and neither are their limitations. But if early benchmarks are any indication, the cost of ignoring AI-native document formats may soon outweigh the effort to adopt them.
It's time we stopped asking AI models to decode our documents and started giving them documents built for their language instead.