Model comparison / repeated runs

LLM benchmark testing for document extraction

Compare extraction models on a fixed document set. Track field quality, failed runs, latency, cost and correction time with a repeatable protocol.

Define what you are comparing

Write down the decision before running models. For example, you may need to choose a pipeline for scanned supplier invoices that returns every line item with the correct amount. A general language-model leaderboard cannot answer that question for your files.

Separate a controlled model comparison from an end-to-end pipeline comparison. For the first, hold the input representation, schema, prompt and scoring rules fixed. For the second, allow each system its normal preprocessing and report those differences. A model receiving OCR text and one receiving page images have different inputs.

Keep a held-out set for the final decision

Use a development set to adjust prompts and schemas, then evaluate on documents that were not used for those adjustments. Include typical work and known difficult cases. Keep duplicates and near-identical templates from inflating the apparent breadth of the test.

A small pilot of 10 to 30 documents can expose obvious failures. It cannot establish a universal accuracy claim. Record how many suppliers, layouts, languages and scan conditions it covers, then expand the set where the decision remains uncertain.

Version the source files and reviewed answers. Save the model identifier, provider, run date, prompt, schema, page selection, preprocessing and available generation settings alongside each output. Keep original predictions before applying human corrections.

Use the same scorecard for every model

Score structured fields with the precision, recall and F1 protocol in the extraction guide. Separately record invalid output, missing pages, unsupported files, timeouts and retries. Declare whether quality is scored on final pipeline output or the first attempt, and retain an all-document completion rate.

Repeat runs to observe variation. Compare models on the same documents and report the spread across runs, median and tail latency, provider charges, hosted PE use and minutes spent correcting results. A cached rerun measures cache behavior; label it separately from fresh inference.

Example decision rule: every account identifier and invoice total must match the reference, with no missing line items, before a document passes unattended. Set your own tolerances before viewing results. If models trade quality for review time, show that tradeoff rather than selecting a winner from one aggregate score.

Apply the protocol in Dynamite Docs

Choose models in AI Settings and run the same documents. The compatibility check confirms that requests work; accuracy still needs comparison with reviewed answers. The repository mock benchmark checks deterministic scoring behavior. It provides no evidence of live provider quality. Export and retain your actual outputs for a scored comparison.

Method reference: OpenAI evaluation best practices. The worked examples and proposed test protocol above are illustrative guidance.

Run a test document

Choose what to measure

Text recognition, model behavior and structured field quality answer different questions. Use the same reviewed documents to connect the results.

All accuracy testing methods

Character error rate / word error rate

OCR accuracy testing with CER and WER

Compare recognized text with a reviewed transcript. Calculate character and word error rates, then inspect the errors that affect document fields.

Precision / recall / F1

Extraction accuracy with precision, recall and F1

Score document fields and line items against ground truth. Define matching rules, count missing and incorrect values, and calculate precision, recall and F1.

Apply the test in Dynamite Docs

Choose a document workflow, run your sample, and compare the result with your reviewed answer.

Invoices
Check identifiers, tax, totals and line items.
Receipts
Test faint text, tips, dates and merchant names.
Bank statements
Check transaction coverage, signs and balances.
PDFs
Separate text-layer, scanned and mixed-page results.
API
Retain outputs and track failures across repeatable runs.