Precision / recall / F1

Extraction accuracy with precision, recall and F1

Score document fields and line items against ground truth. Define matching rules, count missing and incorrect values, and calculate precision, recall and F1.

Define a correct field before scoring

Create a reviewed answer for each document, including required fields, line items and intentionally absent values. Identify a field by document, field name and row identity where relevant. A correct amount attached to the wrong invoice or line item is an incorrect extraction.

Use a written normalization policy. You might accept equivalent date formats when the source date is unambiguous, or ignore surrounding whitespace in a supplier name. Preserve meaningful differences such as currency, sign and identifier leading zeros. Define any numerical tolerance before seeing predictions.

Count matches, extra predictions and missed answers

Use one-to-one matching between predicted and expected field-value pairs. A correct match is a true positive, TP. An unmatched prediction is a false positive, FP. An unmatched expected answer is a false negative, FN. Under this protocol, a wrong value creates both one FP and one FN. A duplicate prediction cannot match the same answer twice.

Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). Precision asks how much returned data is correct. Recall asks how much expected data was found. F1 combines the two; it does not measure business impact.

Worked example: 20 expected fields produce 18 predictions, of which 16 match. TP = 16, FP = 2 and FN = 4. Precision is 88.9%, recall is 80.0% and F1 is 84.2%. These are illustrative counts, not measured product performance.

If no values are predicted, precision has a zero denominator. With no expected values, recall does too. State the reporting convention, such as undefined, and retain the raw counts. Pool counts for a micro score; label any average across field types separately so rare fields remain visible.

Match rows before comparing their cells

Define how rows align when order changes. An invoice line number can anchor matching if it is present and reliable. Otherwise document a deterministic matching rule and inspect repeated descriptions. Do not let several predictions claim the same reference row.

Report correctly matched expected rows divided by expected rows for row recall, alongside duplicate and extra rows. Simply dividing extracted row count by expected row count can reward duplicates. Reconcile totals separately, accounting for discounts, shipping, credits and rounding in the source.

Turn field scores into a review decision

Break results out by field, document type and input condition. Keep a document pass rate for the exact acceptance rule you need, such as all required values correct and every transaction present. Include failed documents in the completion report.

In Dynamite Docs, compare the extracted table with the source before correcting it. Save those initial predictions for scoring, then record correction time. Confidence badges can guide the review order, but their values do not replace ground-truth measurements. Use OCR error rates to diagnose text recognition problems and repeated LLM runs to investigate variation.

Method reference: Google Document AI evaluation metrics. The worked examples and proposed test protocol above are illustrative guidance.

Run a test document

Choose what to measure

Text recognition, model behavior and structured field quality answer different questions. Use the same reviewed documents to connect the results.

All accuracy testing methods

Character error rate / word error rate

OCR accuracy testing with CER and WER

Compare recognized text with a reviewed transcript. Calculate character and word error rates, then inspect the errors that affect document fields.

Model comparison / repeated runs

LLM benchmark testing for document extraction

Compare extraction models on a fixed document set. Track field quality, failed runs, latency, cost and correction time with a repeatable protocol.

Apply the test in Dynamite Docs

Choose a document workflow, run your sample, and compare the result with your reviewed answer.

Invoices
Check identifiers, tax, totals and line items.
Receipts
Test faint text, tips, dates and merchant names.
Bank statements
Check transaction coverage, signs and balances.
PDFs
Separate text-layer, scanned and mixed-page results.
API
Retain outputs and track failures across repeatable runs.