Test table extraction accuracy with AI across complex layouts

Start with a reviewed answer for every document. Include required fields, line items and values that should be absent. Match each value to its document, field name and, when needed, table row. An amount copied correctly but attached to the wrong invoice or line item is still an error.

Write the normalization rules before you run the test. You might accept equivalent date formats or trim space around a supplier name. Keep differences that affect meaning, including currency, signs and leading zeroes. Set number tolerances before you see the predictions.

Match predicted and expected field-value pairs one to one. A correct match is a true positive, TP. An extra prediction is a false positive, FP. A missed answer is a false negative, FN. One wrong value therefore creates one FP and one FN, and a duplicate prediction cannot claim the same answer twice.

Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). Precision tells you how much of the returned data was right. Recall tells you how much of the expected data was found. F1 combines the two, but it cannot tell you whether the remaining errors are costly.

Worked example: 20 expected fields produce 18 predictions, of which 16 match. TP = 16, FP = 2 and FN = 4. Precision is 88.9%, recall is 80.0% and F1 is 84.2%. These are illustrative counts, not measured product performance.

If no values are predicted, precision has a zero denominator. With no expected values, recall does too. State the reporting convention, such as undefined, and retain the raw counts. Pool counts for a micro score; label any average across field types separately so rare fields remain visible.

Decide how rows align when their order changes. A reliable invoice line number can anchor the match. If there is no usable key, document a deterministic rule and inspect repeated descriptions by hand. Never let two predicted rows claim the same reference row.

For row recall, divide correctly matched expected rows by all expected rows. Report duplicate and extra rows beside that figure. A raw extracted-row count can look complete even when it contains duplicates. Check totals separately, including discounts, shipping, credits and source rounding.

Break the results out by field, document type and input condition. Add a document pass rate tied to a real acceptance rule, such as every required value being correct and every transaction being present. Failed documents belong in the completion rate, not in a footnote.

In Dynamite Docs, save the first extraction before anyone corrects it. Score that output against the source, then record how long the corrections take. Confidence badges can help reviewers choose what to inspect first, but they do not replace ground truth. Use OCR error rates to diagnose text recognition and repeated model runs to measure variation.

This reference run tests the benchmark harness, not a live model. It uses corpus and scoring version 2026-08-19.v1, 11 generated fixtures, 44 reviewed fields and two repeats. The runner is the deterministic mock parser in benchmark/extractors.ts, version deterministic-mock-parser@2026-08-19.v1. No LLM or external provider was used.

We recorded the result on 2026-09-23 UTC with Node.js on the project runner. The mock parser waits 120 ms on purpose, so the latency can catch changes in the harness. It says nothing about production extraction speed.