A checked answer makes a better test. Illustrative example.
Sections06
Reading time8 min
EvidenceTesting protocol
Overview
Test document extraction against a human-reviewed answer. Compare field errors, missing rows, totals, failed runs and the time needed to correct each result.
Use files your team actually processes. Keep the document set, expected output, and scoring rules fixed across runs.
Choose what to measure
Text recognition, model behavior and structured field quality answer different questions. Use the same reviewed documents to connect the results.
Learn how to benchmark LLM document extraction with Dynamite Docs. Compare field errors, failed runs, latency, costs, and correction time on the same files.
Verify GSTIN, HSN or SAC, tax splits and bill totals.
A fair test
Dynamite Docs
The handoff
Source
Review
Export
01Read the source
02Review the result
03Export checked data
A checked answer makes a better test. Illustrative example.
Build a reference answer for each document
Confidence helps a reviewer choose where to look first. Accuracy asks whether the extracted value matches a checked answer. Only that ground truth can tell you how often the workflow is right.
01
Choose representative files
Start with 10 to 30 documents that resemble the work you receive. Include ordinary files, awkward layouts, scans and photos.
02
Create the reviewed answer
Record the correct fields, line items, totals and row count in a spreadsheet. A second reviewer should check uncertain values.
03
Keep every run comparable
Use the same files, pages, schema and scoring rules for Local (no AI) and each LLM you want to test.
04
Score the errors and rerun
Count correct fields, missing rows, wrong totals and correction time. Repeat the test to catch output that changes between calls.
Compare processing paths
Compare local parsing and LLM extraction on equal terms.
No model call / 0 PE
Local (no AI)
Local parsing suits CSV, TSV, XLSX and clean text-layer documents with a clear table or label-value structure. If it cannot produce a dependable table, it stops rather than filling gaps with a guess.
Choose Local (no AI) in the extraction model control.
Run every file in the fixed test set.
Record unsupported files as failures, not skipped results.
Rerun to confirm the schema and values stay identical.
Hosted, BYOK, or local Ollama
LLM model
Use an LLM for scans, photos, irregular layouts and fields that need context. A text-only model can work from extracted text. Image-only pages need vision support.
Open AI Settings, then All Models.
Test compatibility before selecting a model.
Choose a passing model and run the same files and schema.
Repeat the run because LLM answers can change.
The model test confirms API and image compatibility. It cannot tell you whether the output matches your documents. The reviewed answer does that.
What to score
One accuracy percentage can hide the costly mistakes.
Score fields, rows, totals and reruns separately. Time the corrections too. The best model for your workflow is the one that meets your error limit with an acceptable amount of review.
Correct fields
correct expected fields / all expected fields
Normalize harmless formatting differences before comparison.
Complete rows
correctly matched expected rows / expected rows
A perfect header with missing invoice rows is still a bad result.
Reconciled totals
subtotal + tax = total
Reconcile money fields separately because one digit can matter.
Stable reruns
same schema and values on rerun
Local code should be stable. LLM output can vary between calls.
Confidence scores
Use confidence to choose what gets checked first.
Data Studio shows an overall score and field-level badges. These are model signals, not measured probabilities that a value is correct. Compare the value with the source before relying on it.
Dynamite Docs
The review
Source value
Compare, then accept
Read the source
Check against the source
Review the result
Check against the source
Export checked data
Check against the source
A checked answer makes a better test. Illustrative example.
Invoice total34%
$8,7B2.10
Compare the ambiguous character with the source before export.
80-100
Higher overall confidence
Review the critical fields, then spot-check the rest.
50-79
Moderate overall confidence
Check highlighted fields and compare totals before export.
0-49
Low overall confidence
Treat the result as a draft. Review it closely or try another model.
Field badges use tighter bands: 70 and above, 40 to 69, and below 40. Sort your review around the lowest fields, then reconcile identifiers, dates, and money even when their score is high.
When to switch
Switch models when the same error keeps returning.
One bad result may come from a poor scan. The same miss across several documents points to the model, its vision support or the extraction path.
The same required field is missing or wrong across several files.
Switch now
A scan or photo was sent to a model without reliable vision support.
Test first
Confidence is low and the source is clear enough for another model to read.
Fix the input
Every model struggles with blur, cropped totals, glare, or unreadable text.
Keep the model
It meets your error limit and review time on the fixed test set.
Use Auto
Your documents vary enough that one fixed model is not the best route.
Run a repeatable document extraction benchmark
Compare models using the same documents and reviewed answers. Check field accuracy, missing rows, totals, processing time, and the corrections each result needs before you choose a model for everyday work.
Essential cookies keep you signed in. With your permission, optional analytics cookies and masked session replays help us understand usage and diagnose errors. Privacy policy. Change your choice anytime in Cookie settings.