Sections06
Reading time8 min
EvidenceTesting protocol

Overview

Test document extraction against a human-reviewed answer. Compare field errors, missing rows, totals, failed runs and the time needed to correct each result.

Use files your team actually processes. Keep the document set, expected output, and scoring rules fixed across runs.

Choose what to measure

Text recognition, model behavior and structured field quality answer different questions. Use the same reviewed documents to connect the results.

Character error rate / word error rate

Measure OCR accuracy using AI vision models

Learn how to measure AI OCR accuracy with Dynamite Docs. Compare extracted text with a reviewed transcript using character and word error rates.

Model comparison / repeated runs

Benchmark field extraction using LLMs

Learn how to benchmark LLM document extraction with Dynamite Docs. Compare field errors, failed runs, latency, costs, and correction time on the same files.

Apply the test in Dynamite Docs

Choose a document workflow, run your sample, and compare the result with your reviewed answer.

  • Invoices

    Check identifiers, tax, totals and line items.

  • Receipts

    Test faint text, tips, dates and merchant names.

  • Bank statements

    Check transaction coverage, signs and balances.

  • PDFs

    Separate text-layer, scanned and mixed-page results.

  • API

    Retain outputs and track failures across repeatable runs.

  • PDF tables

    Check table boundaries, page breaks, row counts and totals.

  • Statement to Excel

    Test transaction coverage, signs and balance reconciliation.

  • BUSY invoices

    Verify GSTIN, HSN or SAC, tax splits and bill totals.

A fair test

Dynamite Docs logoDynamite Docs
The handoff
Reviewed reference: Check every resultRead the source. Review the result. Export checked data. Not extracted customer data.FROM SOURCE FILE TO A RESULT YOU CAN USEPDFReviewed…🔍 Click to zoomReview workspaceCheck everyresult1Read the source2Review the result3Export checkeddata
  1. 01Read the source
  2. 02Review the result
  3. 03Export checked data
A checked answer makes a better test.

Build a reference answer for each document

Confidence helps a reviewer choose where to look first. Accuracy asks whether the extracted value matches a checked answer. Only that ground truth can tell you how often the workflow is right.

  1. 01

    Choose representative files

    Start with 10 to 30 documents that resemble the work you receive. Include ordinary files, awkward layouts, scans and photos.

  2. 02

    Create the reviewed answer

    Record the correct fields, line items, totals and row count in a spreadsheet. A second reviewer should check uncertain values.

  3. 03

    Keep every run comparable

    Use the same files, pages, schema and scoring rules for Local (no AI) and each LLM you want to test.

  4. 04

    Score the errors and rerun

    Count correct fields, missing rows, wrong totals and correction time. Repeat the test to catch output that changes between calls.

Compare processing paths

Compare local parsing and LLM extraction on equal terms.

No model call / 0 PE

Local (no AI)

Local parsing suits CSV, TSV, XLSX and clean text-layer documents with a clear table or label-value structure. If it cannot produce a dependable table, it stops rather than filling gaps with a guess.

  • Choose Local (no AI) in the extraction model control.
  • Run every file in the fixed test set.
  • Record unsupported files as failures, not skipped results.
  • Rerun to confirm the schema and values stay identical.
Hosted, BYOK, or local Ollama

LLM model

Use an LLM for scans, photos, irregular layouts and fields that need context. A text-only model can work from extracted text. Image-only pages need vision support.

  • Open AI Settings, then All Models.
  • Test compatibility before selecting a model.
  • Choose a passing model and run the same files and schema.
  • Repeat the run because LLM answers can change.

The model test confirms API and image compatibility. It cannot tell you whether the output matches your documents. The reviewed answer does that.

What to score

One accuracy percentage can hide the costly mistakes.

Score fields, rows, totals and reruns separately. Time the corrections too. The best model for your workflow is the one that meets your error limit with an acceptable amount of review.

Correct fields

correct expected fields / all expected fields

Normalize harmless formatting differences before comparison.

Complete rows

correctly matched expected rows / expected rows

A perfect header with missing invoice rows is still a bad result.

Reconciled totals

subtotal + tax = total

Reconcile money fields separately because one digit can matter.

Stable reruns

same schema and values on rerun

Local code should be stable. LLM output can vary between calls.

Confidence scores

Use confidence to choose what gets checked first.

Data Studio shows an overall score and field-level badges. These are model signals, not measured probabilities that a value is correct. Compare the value with the source before relying on it.

Dynamite Docs logoDynamite Docs
The review
Reviewed reference: Check every resultA checked answer makes a better test.. Keep the source reference. Review before export. Not extracted customer data.SOURCE DOCUMENTReviewedreference🔍 Click to zoomReview before exportRead the sourceCheck against thesourceCompare with the source on the leftAccept after reviewA checked answer makes a bettertest.Original stays…
Read the source
Check against the source
Review the result
Check against the source
Export checked data
Check against the source
A checked answer makes a better test.
Invoice total34%

$8,7B2.10

Compare the ambiguous character with the source before export.

80-100

Higher overall confidence

Review the critical fields, then spot-check the rest.

50-79

Moderate overall confidence

Check highlighted fields and compare totals before export.

0-49

Low overall confidence

Treat the result as a draft. Review it closely or try another model.

Field badges use tighter bands: 70 and above, 40 to 69, and below 40. Sort your review around the lowest fields, then reconcile identifiers, dates, and money even when their score is high.

When to switch

Switch models when the same error keeps returning.

One bad result may come from a poor scan. The same miss across several documents points to the model, its vision support or the extraction path.

Compare available models

Switch now

The same required field is missing or wrong across several files.

Switch now

A scan or photo was sent to a model without reliable vision support.

Test first

Confidence is low and the source is clear enough for another model to read.

Fix the input

Every model struggles with blur, cropped totals, glare, or unreadable text.

Keep the model

It meets your error limit and review time on the fixed test set.

Use Auto

Your documents vary enough that one fixed model is not the best route.

Run a repeatable document extraction benchmark

Compare models using the same documents and reviewed answers. Check field accuracy, missing rows, totals, processing time, and the corrections each result needs before you choose a model for everyday work.

Read extraction docs

Can we save optional cookies?

Essential cookies keep you signed in. Optional analytics and masked session replays help diagnose errors. Change your choice in Cookie settings. Privacy policy.