A demo of document extraction on ten clean invoices reassures a room quickly. Production receives scans of uneven quality, mixed formats, handwritten marks, and business exceptions nobody listed. In that gap the model is often only part of the problem. What decides whether the system holds is the engineering around it: test sets, scores, recovery queues, observability.
Test sets and per-field scores
A representative test set is not limited to “pretty” documents. It should mirror the real distribution - including rare cases that are expensive in manual recovery. We prefer a confidence score per field over a single score per document: a misread amount does not have the same impact as an approximate label. Without that granularity, accept/reject decisions are too coarse. The same demo / production gap appears in enterprise RAG.
The human recovery queue is not an admission of failure; it is a pipeline component. Every recovery should be traceable: which value was proposed, who corrected it, within what delay. Those corrections then feed error analysis. Without a closed loop, the error rate measured at go-live drifts before anyone notices - usually at the cost of a business incident.
Production-ready under real load
We treat an extractor as production-ready only when residual errors are acceptable to the business and recoverable within a defined delay. If that condition is not met, the tool moves work around rather than reducing it. The question is therefore not “is the model good enough?”, but “is the system around the model operable under real load?”. In finance, that continues into pre-accounting automation.
When the flow must reach the accounting tool, a brick like AYA 360 helps think end to end rather than isolated OCR. To frame an extraction pipeline, see also our AI offerings.
