Products How it works Security About Pricing Newsletter Live demo Talk to us
Technical report · June 2026

Answer accuracy: 95%.

We ran Datavera's Document Assistant against 632 questions over 2,625 public documents, about 134,000 pages from seven industries. Every reference answer was verified by a person before scoring. This report covers the protocol, the system under test, the results, and the error analysis.

Read the protocol Browse the 632 questions
95% answers judged correct under hand adjudication
632 questions in the gold set, each verified at the source page
2,625 documents in the corpus, about 134,000 pages, seven industries
98% questions where retrieval returned the right source document

Summary

We measure answer accuracy on a fixed benchmark: 632 questions over a corpus of 2,625 public, openly licensed documents drawn from finance, legal, medical, scientific, engineering, humanities and government sources. Reference answers were verified by a person against the cited source page before any measurement. Answers are scored twice, first by a strict automated judge, then by hand adjudication; headline figures use the hand-adjudicated score. The system answered 95% of the questions correctly and retrieved the correct source document for 98% of them. Retrieval, not reading, was the dominant failure mode. After in-house fine-tuning of the open-source models in the stack, no industry scores below 93%. The full question set, with reference answers and source pages, is published here.

Protocol.

The benchmark was fixed before scoring began: corpus, question set, verified answer key, scoring rules. Nothing in it was adjusted afterwards to improve a number.

01 · CORPUS

Corpus construction

2,625 documents, roughly 134,000 pages: annual reports and SEC filings, court opinions and federal rules, drug labels, engineering manuals, scientific papers, policy reports. Ingested, the corpus amounts to about one million retrievable passages.

  • Sector weights follow page volume as it occurs in real archives. Finance dominates: dense, footnoted, multi-table filings are the hardest input we process.
  • Every document is public and openly licensed, so the benchmark can be reconstructed and audited by anyone.
  • Customer documents are never used for benchmarking.
02 · QUESTION SET

Question design

632 questions, written in-house against the corpus. 456 of them, 72 percent, are graded hard, built on the distractors real archives produce: adjacent fiscal years, near-identical tables, sibling documents.

  • 302 needle questions target a single fact: one number in a footnote, one date on one page out of 134,000.
  • 330 gather questions require evidence from several passages; 174 of those span more than one document.
  • 211 questions are finance, the deliberately over-weighted hardest sector.
03 · VERIFICATION

Answer verification

Before measurement, a person opened every cited page and confirmed the reference answer. The answer key contains no machine-generated entries.

  • In the main audit, 596 of 612 answers passed as written. 16 were corrected, several against our own draft key. The corrections are documented and kept.
  • Questions added after the audit went through the same page-level check.
04 · SCORING

Scoring

Each system answer is scored twice. A strict automated judge grades the full set; a person then adjudicates answers against the verified reference.

  • The automated judge counts honest refusals and formatting deviations as errors, so it reads systematically low. We keep it as a lower bound.
  • Headline figures use the hand-adjudicated score.

The complete question set, with reference answers and source pages, is published at datavera.ch/accuracy/questions.

The system under test.

The benchmark ran on the configuration we deploy, not on a lab build. What follows is the pipeline a document passes through, from PDF to cited answer. We name the techniques; the tuning that makes them work together is our own.

STAGE 1Extraction
Every page passes two extractors in parallel: the PDF's native text layer, and a GPU OCR engine that detects layout, reconstructs tables as structured data, and reads scanned or rotated pages. The two outputs are aligned and merged, so tables keep their structure and nothing on the page is lost. Charts and figures, which OCR cannot read, are captioned by a vision model so their content becomes searchable text.
STAGE 2Indexing
Documents are chunked along their structure, headings, sections and tables, not at fixed character counts. Each passage is stored with a short generated context prefix that situates it in its document, which makes an isolated passage findable by questions that share none of its literal words. Passages are indexed three ways at once: dense semantic vectors, classic exact-term search, and a knowledge graph of the entities that co-occur across the corpus. All of it lives in standard PostgreSQL on one machine. No proprietary vector store, no external service.
STAGE 3Retrieval
A question runs against several retrieval signals in parallel: dense semantic search, exact-term search, an entity lookup, and a graph traversal that follows entity connections across documents for multi-hop questions. A hypothetical-answer expansion switches on only when the gap between question phrasing and document phrasing is large. The rankings are fused with reciprocal rank fusion, then a cross-encoder re-ranks a deepened candidate pool. Each signal catches failures the others miss. The stack was assembled by measurement, and the variants that did not earn their place were dropped and documented.
STAGE 4Cross-document assembly
Questions that span documents break naive retrieval: a single top-k merge lets the loudest document crowd out the rest. The system instead decomposes the question into one sub-query per subject, retrieves for each independently, and merges the results under a coverage quota that guarantees every subject its evidence slots. The model then reads the complete evidence in one pass, which measured better than splitting the question into fragments and recomposing.
STAGE 5Grounded answering
The model answers from the retrieved passages at temperature zero and attaches file and page to every claim. A deterministic grounding check verifies each claim against the retrieved text; unsupported claims are rejected and the system returns "not found". Abstention is a trained behaviour, not a prompt.

Benchmark environment

Models
Open-source models throughout, fine-tuned in-house. The fine-tuning covers behaviour (answer only from the shown evidence, abstain when it is absent) and task specialisation; it does not write facts into weights. Facts stay in the retrieval layer, where each one is traceable to a page and updated when the document changes. No component is tied to a vendor.
Hardware
A single workstation-class GPU runs ingestion, retrieval and answering. The benchmark machine is configured like a customer deployment.
Data path
No cloud API is called between document and answer. The same holds in production: documents stay on the customer's hardware or in a Swiss data centre.

Results.

Full 632-question set, June 2026. Hand-adjudicated scores.

Answers judged correct 95%
Right source document retrieved 98%
Answers correct where the evidence reached the model 95%
Per-industry floor after fine-tuning, finance included 93% or better
Behaviour when the evidence is absent from the corpus Abstains: returns "not found"

Error analysis

Failures concentrate in retrieval, not reading. Where the right passage reached the model, the answer was almost always correct. Where an answer was wrong, the usual cause was a missing or partial passage, most often in finance: values inside dense tables, footnotes that override the table above them, figures restated in a later filing.

Earlier runs showed a wide per-industry spread from exactly that effect. Clean prose sectors scored high from the start; finance sat well below them. We closed most of the gap by fine-tuning the open-source models in the stack. Not to teach them facts. Facts stay in the retrieval layer, where every one is traceable to a page and updated the day a document changes. The fine-tuning targets behaviour, answering strictly from the shown evidence and abstaining when it is absent, and specialisation: a small model trained for one task can match a much larger general model on that task while running on ordinary hardware.

After that work, no industry scores below 93%. Finance remains the hardest sector and remains over-weighted at 211 of the 632 questions.

Limitations.

The figures are measured on this benchmark corpus. Results on a different archive vary with document quality, layout and domain. The free 30-day pilot exists to run the same measurement on a customer's own documents.

95% correct means one answer in twenty is wrong or incomplete. Citations keep the error cost bounded: every answer names its file and page, so a wrong answer can be checked against its source in seconds, and a missing answer is reported as "not found" rather than papered over.

The benchmark is versioned and re-run as the system changes. Figures on this page refer to the most recent fully adjudicated run, currently June 2026.

Questions about the benchmark

How accurate is Datavera's Document Assistant?
On the published benchmark: 95% of answers correct under hand adjudication, across 632 verified questions over 2,625 public documents from seven industries. The correct source document was retrieved for 98% of questions. After in-house fine-tuning of the open-source models in the stack, every industry scores 93% or better.
Who wrote and checked the test questions?
The questions were written in-house against the corpus. Before any measurement, a person opened every cited source page and confirmed the reference answer. The audit corrected 16 entries in the draft answer key, and the corrections are documented. No machine-generated answer key was used.
Does it hallucinate?
The system is built to abstain instead. A deterministic grounding check runs on every answer; when the retrieved evidence does not support a claim, the system returns "not found" rather than a guess. Every answer carries its file and page, so a claim can be checked against its source in seconds.
What documents was it tested on?
2,625 public, openly licensed documents, about 134,000 pages: annual reports and SEC filings, court opinions and federal rules, drug labels and medical research, engineering manuals, scientific papers, and government policy reports. Customer documents are never used in benchmarks.
Will I get the same accuracy on my documents?
Not necessarily. Accuracy depends on the documents: dense financial tables score lower than clean prose, which is why finance is the largest and hardest part of the benchmark. The free 30-day pilot runs the same measurement on your own archive.
Did the test run in the cloud?
No. The benchmark ran on a single workstation-class GPU with open-source models and no cloud API in the answering path. That is the same configuration Datavera deploys on customer hardware or in a Swiss data centre.
Can I see the actual questions?
Yes. All 632 questions are published at datavera.ch/accuracy/questions, each with its verified reference answer and the source document and page. The answers listed are the verified gold-set answers, not the system's output.

Questions about the benchmark, or about your own documents.

Write to the engineering team. We reply within one business day.

Ask about the benchmark

By submitting, you agree to our privacy policy. We never sell or share your data.
Message received. We will be in touch within one business day.