Character error rate / word error rate

OCR accuracy testing with CER and WER

Compare recognized text with a reviewed transcript. Calculate character and word error rates, then inspect the errors that affect document fields.

Prepare a transcript you can trust

Choose pages that represent the documents you receive. Keep clean digital PDFs, scans and phone photos in separate groups so the easy pages cannot hide failures on faint receipts or rotated bank statements. Record the source file, page number and reading order for every transcript.

Have a person transcribe each page and a second person check ambiguous text against the image. Mark unreadable regions explicitly and document whether they are excluded. Decide in advance how to handle whitespace, punctuation, capitalization and Unicode. Keep the same rules for every engine.

Calculate CER and WER

CER = (substitutions + deletions + insertions) / reference characters. WER uses the same edit-distance calculation on words, divided by reference words. Both measure the edits needed to align a prediction with the reference. Lower is better; extra inserted text can push either rate above 100%.

Worked example: the reference is "Total 125.00" and the prediction is "Total 128.00". Counting the space, there are 12 reference characters and one substitution, so CER is 1/12 = 8.33%. With whitespace tokenization there are two words and one wrong word, so WER is 1/2 = 50%. This is an illustrative calculation, not a Dynamite Docs benchmark result.

For a corpus score, sum edit counts and reference lengths before dividing. Also show page-level results and the number of pages in each input group. If a reference is empty, the denominator is zero; report inserted text separately and state the scorer convention instead of silently recording a perfect score.

Inspect where the character errors landed

An incorrect digit in a total may need immediate correction even when thousands of surrounding characters match. Keep an error register for account numbers, dates, amounts and identifiers. Do not remove decimal points or minus signs during normalization just to improve the score.

CER and WER do not establish whether a value belongs to the right field or table row. A transcript can contain every word while mixing two columns. Use the extraction accuracy guide to score those assignments, and the LLM benchmark guide when comparing complete processing pipelines.

Method reference: Hugging Face CER metric definition and limitations. The worked examples and proposed test protocol above are illustrative guidance.

Run a test document

Choose what to measure

Text recognition, model behavior and structured field quality answer different questions. Use the same reviewed documents to connect the results.

All accuracy testing methods

Model comparison / repeated runs

LLM benchmark testing for document extraction

Compare extraction models on a fixed document set. Track field quality, failed runs, latency, cost and correction time with a repeatable protocol.

Precision / recall / F1

Extraction accuracy with precision, recall and F1

Score document fields and line items against ground truth. Define matching rules, count missing and incorrect values, and calculate precision, recall and F1.

Apply the test in Dynamite Docs

Choose a document workflow, run your sample, and compare the result with your reviewed answer.

Invoices
Check identifiers, tax, totals and line items.
Receipts
Test faint text, tips, dates and merchant names.
Bank statements
Check transaction coverage, signs and balances.
PDFs
Separate text-layer, scanned and mixed-page results.
API
Retain outputs and track failures across repeatable runs.