Measured extraction accuracy and latency benchmarks across model providers

Build a reviewed test set, measure OCR and field errors, compare LLMs, and find the extraction workflow that needs the least correction.

Choose what to measure

Text recognition, model behavior and structured field quality answer different questions. Use the same reviewed documents to connect the results.

Character error rate / word error rate

Measuring character and word error rates across modern vision models

Measure OCR accuracy against a reviewed transcript with character error rate and word error rate, then inspect the mistakes that can damage extracted data.

Model comparison / repeated runs

Field extraction precision and recall metrics across frontier LLMs

Compare LLMs on the same documents, schema and scoring rules. Track field quality, failed runs, latency, provider cost and human correction time.

Precision / recall / F1

Stress-testing multi-page table extraction on complex real-world layouts

Reproduce a document extraction benchmark with public fixtures, reviewed ground truth, scoring code, model version, latency, cost and limitations.

Apply the test in Dynamite Docs

Choose a document workflow, run your sample, and compare the result with your reviewed answer.

Invoices
Check identifiers, tax, totals and line items.
Receipts
Test faint text, tips, dates and merchant names.
Bank statements
Check transaction coverage, signs and balances.
PDFs
Separate text-layer, scanned and mixed-page results.
API
Retain outputs and track failures across repeatable runs.
PDF tables
Check table boundaries, page breaks, row counts and totals.
Statement to Excel
Test transaction coverage, signs and balance reconciliation.
BUSY invoices
Verify GSTIN, HSN or SAC, tax splits and bill totals.

Start with a ground-truth test set

Choose 10 to 30 representative documents, including clean files, unusual layouts, scans, and photos.
Record the correct fields, line items, totals, and row counts in a reviewed spreadsheet.
Run unchanged copies through Local (no AI) and each LLM using the same page selection and schema.
Count exact field matches, missing rows, wrong totals, correction time, and changes between repeated runs.

Test Local (no AI)

The deterministic code path works with CSV, TSV, XLSX, and clean text-layer documents that contain a clear table or label-value structure. Choose Local (no AI), run the full test set, count unsupported files as failures, and rerun to confirm that the result stays identical.

Test an LLM model

Open AI Settings, choose All Models, run the compatibility test, then select a passing model. Process the same files and repeat the run. Scanned pages require a vision-capable model. The built-in compatibility test checks whether the API and image request work; it does not measure accuracy.

Read confidence scores

Data Studio shows overall confidence and a badge for each field. Overall scores from 80 to 100 indicate a higher-confidence result, 50 to 79 indicate moderate confidence, and scores below 50 require close review. Field badges use bands of 70 and above, 40 to 69, and below 40.

Confidence is a review signal, not measured accuracy or a guarantee. Compare identifiers, dates, money, and low-confidence fields with the source document before export.

Know when to choose another model

Switch when the same required field fails across several files, a scanned document needs stronger vision support, or another model scores better on the fixed test set. Fix the input instead when blur, cropping, or glare makes the source unreadable. Keep the current model when it meets your error limit and correction-time target.

Run the repository benchmark

The engineering benchmark measures field accuracy, row completeness, totals, schema stability, latency, and PE usage. Run npm run benchmark:mock for the deterministic baseline or npm run benchmark for an authenticated live test.

Run a test document or open model settings.