Adaptive document parsing: a deterministic Rust engine first, with model-arbitrated upgrade tiers a document walks through only as far as it needs. PDF, Markdown, HTML, DOCX, PPTX, XLSX, a raster image, a meeting transcript or an email in; Markdown, XHTML, JSON, plain text or a DoCO JSON-LD graph out.
| tier | adds | overall¹ | typical cost/document |
|---|---|---|---|
| 1 | deterministic extraction + layout | 0.894673 | 8 ms (CPU) |
| 2 | layout-detector arbitration (headings, table regions) | 0.902501 | ~0.2 s (CPU) |
| 3 | deep reading of pixels-only content and doubted structure | 0.934448 | ~1.5 s² |
¹ 200-document public evaluation corpus, measured 2026-10-07 — the standings below. Every rung reproduces from the committed model-output caches with no GPU and no API key.
² Averaged across the corpus: 87 of the 200 documents escalate (median 1.7 s), the other 113 never leave tier 1 at 8 ms. Tiers 1 and 2 are CPU-only, so the deep reader is the sole reason to think about a GPU or an API key. The full timing accounting is on the benchmarks page.
Two principles produced those numbers:
- Models arbitrate structure; the page tier owns its page. Where the deterministic pass produced a reading, a model's replaces it only when their shapes disagree, and two independent readers agreeing against the deeper model win. Where the deterministic pass is what failed — no usable glyph layer, or a hierarchy resting on nothing — the reading owns the page outright, because there is no good text to prefer to it.
- Escalation is earned, not configured. Each tier's outputs are the sensors that decide whether the next tier runs, per document, per region, per table.
cargo build --release
fdoc convert document.pdf # markdown (default)
fdoc convert notes.md -f doco # Markdown in, DoCO graph out
fdoc convert report.docx -f doco # DOCX in, same graph out
fdoc convert page.html -f doco # HTML in, same graph out
fdoc convert deck.pptx -f doco # PPTX in, slides become pages
# (charts become tables)
fdoc convert book.xlsx -f doco # XLSX in, sheets become pages
# (blocks of cells become tables)
fdoc convert call.vtt -f doco # WebVTT or SubRip in, one paragraph
# per speaker turn, with who and when
fdoc convert reply.eml -f doco # .eml or .msg in, the thread split
# into its messages
fdoc convert document.pdf -f xhtml # XHTML
fdoc convert document.pdf -f json # DoCO-typed elements + bboxes
fdoc convert document.pdf -f doco # DoCO JSON-LD graph (sections,
# containment, table cells, char
# offsets, page/bbox provenance)
fdoc convert document.pdf -f text # plain-text projection (the string
# doco char offsets index into)
fdoc convert ./docs/ --out-dir out -j 8 # parallel batch
fdoc forms document.pdf # AcroForm fields + filled values
fdoc triage document.pdf # what would escalate, and why
fdoc config gemini --credentials key.json # set a deep reader up, once
fdoc convert document.pdf -f doco # …and any format escalates
fdoc convert document.pdf --no-escalate # deterministic for this run
Escalation is off until a provider is configured, and with none configured the
binary opens no network connection. Readings can also be supplied as
JSON sidecars from any model you run
yourself — see fdoc config.
Measured 2026-10-07 on
opendataloader-bench:
200 public PDFs with hand-checked ground truth, scored by a harness neither
written nor tuned by this project. NID scores reading order and text, TEDS
table structure, MHS heading structure; overall is their per-document mean.
All 17 engines scored:
| # | engine | overall | NID | TEDS | MHS | s/doc |
|---|---|---|---|---|---|---|
| 1 | fluree-doc-parse (cascade) | 0.934448 | 0.9484 | 0.9420 | 0.8814 | ~1.5 |
| 2 | opendataloader-hybrid | 0.906572 | 0.9337 | 0.9276 | 0.8208 | 0.463 |
| 3 | fluree-doc-parse (deterministic) | 0.894673 | 0.9238 | 0.8616 | 0.8172 | ~0.009 |
| 4 | nutrient | 0.885067 | 0.9250 | 0.7081 | 0.8190 | 0.008 |
| 5 | docling | 0.881679 | 0.8984 | 0.8871 | 0.8240 | 0.762 |
| 6 | opendataloader-hybrid-hydrogen | 0.876816 | 0.9260 | 0.7958 | 0.7685 | 5.068 |
| 7 | pdf-inspector | 0.875348 | 0.9147 | 0.8141 | 0.7879 | 0.006 |
| 8 | marker | 0.860836 | 0.8897 | 0.8076 | 0.7956 | 53.932 |
| 9 | opendataloader-hybrid-helium | 0.845058 | 0.8789 | 0.8068 | 0.7549 | 8.620 |
| 10 | unstructured-hires | 0.841377 | 0.9038 | 0.5883 | 0.7486 | 3.008 |
| 11 | edgeparse | 0.836959 | 0.8938 | 0.7174 | 0.7061 | 0.036 |
| 12 | opendataloader | 0.831209 | 0.9023 | 0.4887 | 0.7395 | 0.015 |
| 13 | mineru | 0.831135 | 0.8574 | 0.8730 | 0.7430 | 5.962 |
| 14 | pymupdf4llm | 0.731621 | 0.8851 | 0.4010 | 0.4122 | 0.091 |
| 15 | unstructured | 0.685777 | 0.8818 | 0.0000 | 0.3877 | 0.077 |
| 16 | markitdown | 0.588504 | 0.8437 | 0.2729 | 0.0000 | 0.114 |
| 17 | liteparse | 0.575604 | 0.8660 | 0.0000 | 0.0000 | 1.061 |
The deterministic engine, which uses no model, GPU or API key, places third on its own. The full cascade places first, 0.028 ahead of the next engine.
Notes on how both s/doc figures were timed, and the caveats that apply to
these scores, are on the benchmarks page. That
page also lists where our output differs from the
reference and is scored lower as a
result. Per-document results, the steps to reproduce them, and the approaches
measured and rejected are in the scoreboard.
PDF, Markdown, HTML, DOCX, PPTX, XLSX, transcripts, email and raster images all converge on one element
model, so every source produces the same DoCO graph. PDF is the geometric
path — structure inferred from glyph and rule positions, with escalation to
model tiers where the inference is weak. The others declare their structure,
so those readers map rather than measure, and carry no geometry: their
elements have no bbox rather than a zeroed one.
The doco output is insertable into a Fluree ledger as-is:
the JSON-LD context carries the DoCO/NIF/pattern ontologies, containment edges
are IRI-coerced, and --doc-iri stamps every element for retract-on-rerun.
fdoc dev exposes pipeline internals (glyphs, lines, blocks, table geometry)
for debugging; its output is not a compatibility surface.
Full documentation is in docs/ — an mdBook, so
mdbook serve docs renders it locally.
- Output formats — what each of the five guarantees
- CLI reference — every command, flag by flag
- Integration — wiring the model tiers, ledger ingest, entity overlay
- Rust library — the supported API
- Design — how the deterministic engine works
- Benchmarks — the standings, the method, the caveats
crates/fluree-doc-model— element model and emitters, source-agnosticcrates/fluree-doc-markdown— Markdown readercrates/fluree-doc-docx— DOCX (OOXML) readercrates/fluree-doc-html— HTML readercrates/fluree-doc-pptx— PPTX (OOXML) readercrates/fluree-doc-xlsx— XLSX (OOXML) readercrates/fluree-doc-transcript— WebVTT and SubRip transcript readercrates/fluree-doc-email— email reader (.eml,.msg)crates/fluree-doc-pdf— extraction, layout, routing, arbitrationcrates/fluree-doc-cli— thefdocbinarybench-adapter/— opendataloader-bench engine adapters (incl. hybrid tiers)docs/— the documentation bookeval/— test plan, corpus notes, reproduction caches, analyses and evidence
Copyright 2026 Fluree PBC. Licensed under the Apache License, Version 2.0; see LICENSE.
PDF interpretation and rendering use the hayro crates (Apache-2.0 OR MIT). HTML parsing uses html5ever (MIT OR Apache-2.0) directly — Servo's spec-compliant parser, without a CSS selector layer the reader does not need. Markdown parsing uses pulldown-cmark (MIT), and the OOXML readers use quick-xml and zip (both MIT).
No model code is linked and no weights ship here. Model readings enter
through two doors: a deep reader configured over HTTPS, or JSON sidecars
from any layout detector or reader you run yourself. The committed eval/
caches — kept so every published score reproduces without a GPU or an API
key — include outputs produced during evaluation with PaddleX and PaddleOCR
models (Apache-2.0), whose code and weights are not redistributed here.
A configured deep reader (fdoc config) speaks HTTPS
through ureq and
rustls (both MIT OR Apache-2.0), with the
Mozilla CA set from webpki-roots
(CDLA-Permissive-2.0, a permissive data licence). Service-account assertions
are signed with the pure-Rust rsa crate
(MIT OR Apache-2.0), so no system crypto library is required. None of this is
reached with no provider configured.
Run cargo tree for the full dependency set. Every crate in it is under a
permissive licence — MIT, Apache-2.0, BSD, ISC, Zlib, Unicode-3.0 or
CDLA-Permissive-2.0 — with no copyleft anywhere in the tree. The per-user
config directory is resolved directly rather than through the usual crate,
whose transitive option-ext is MPL-2.0 and would be the only exception.