Test document extraction accuracy in Dynamite Docs
Compare the local no-AI parser with hosted, own-key, or local Ollama models. Use a human-reviewed ground-truth set, inspect confidence scores, and switch models when errors repeat.
Start with a ground-truth test set
- Choose 10 to 30 representative documents, including clean files, unusual layouts, scans, and photos.
- Record the correct fields, line items, totals, and row counts in a reviewed spreadsheet.
- Run unchanged copies through Local (no AI) and each LLM using the same page selection and schema.
- Count exact field matches, missing rows, wrong totals, correction time, and changes between repeated runs.
Test Local (no AI)
The deterministic code path works with CSV, TSV, XLSX, and clean text-layer documents that contain a clear table or label-value structure. Choose Local (no AI), run the full test set, count unsupported files as failures, and rerun to confirm that the result stays identical.
Test an LLM model
Open AI Settings, choose All Models, run the compatibility test, then select a passing model. Process the same files and repeat the run. Scanned pages require a vision-capable model. The built-in compatibility test checks whether the API and image request work; it does not measure accuracy.
Read confidence scores
Data Studio shows overall confidence and a badge for each field. Overall scores from 80 to 100 indicate a higher-confidence result, 50 to 79 indicate moderate confidence, and scores below 50 require close review. Field badges use bands of 70 and above, 40 to 69, and below 40.
Confidence is a review signal, not measured accuracy or a guarantee. Compare identifiers, dates, money, and low-confidence fields with the source document before export.
Know when to choose another model
Switch when the same required field fails across several files, a scanned document needs stronger vision support, or another model scores better on the fixed test set. Fix the input instead when blur, cropping, or glare makes the source unreadable. Keep the current model when it meets your error limit and correction-time target.
Run the repository benchmark
The engineering benchmark measures field accuracy, row completeness, totals, schema stability, latency, and PE usage. Run npm run benchmark:mock for the deterministic baseline or npm run benchmark for an authenticated live test.
Run a test document or open model settings.