From Character Accuracy to Workflow Accuracy: An Enterprise Guide to Evaluating OCR Solutions

An organization tests an Optical Character Recognition (OCR) solution on a clearly printed invoice. The tool correctly reads most words and numbers, and the results displayed on screen appear entirely satisfactory. However, when the same solution is applied to actual business documents, it mixes up table columns, assigns the total amount to the tax field, and separates an explanatory footnote from the clause it qualifies.

This raises a question that should precede any purchasing decision: Has the solution succeeded because it can read the page, or has it failed because it cannot produce data the organization can trust?

The distinction between reading a page and producing reliable data explains much of the disappointment organizations experience after adopting OCR solutions. A tool may recognize every word on a page yet place the correct value in the wrong field. Character-level accuracy may be high, while the business process that depends on the document still fails.

Before examining the details, four key points are worth keeping in mind:

• A single accuracy percentage is insufficient. It can conceal significant differences between error types and their actual business impact.
• Arabic documents introduce additional complexities, including right-to-left text direction, connected letterforms, diacritics, and complex tables. Evaluating them requires a dedicated approach rather than simply adapting English-language tests.
• True success must be measured across three complementary dimensions: text accuracy, structural accuracy, and workflow accuracy.
• Sound enterprise decisions begin with a test sample that reflects the organization's actual documents, not a demonstration using a carefully selected, clean document.

Why Is an Overall Accuracy Percentage Not Enough?

When a solution provider presents a single figure, such as "97% accuracy," that number may conceal more than it reveals. Not all errors are alike, and their consequences for business operations vary considerably.

Five categories of errors should be distinguished when evaluating any solution:

• A minor spelling error that neither changes the meaning of a sentence nor prevents its use.
• A reading-order error that produces text in a sequence different from its logical order on the page.
• A table-structure error that places a value in the wrong column or merges rows that should remain separate.
• An error in an identification number, financial amount, or date. This is particularly critical because it affects a field used to make a decision or execute an action.
• An error that prevents a downstream system from using the output altogether because of an unexpected format or a missing field.

The first type may require little attention. The last may bring an entire workflow to a halt.

An organization that compares solutions using a single accuracy percentage effectively assigns equal weight to these errors, even though their operational consequences are vastly different.

The impact of an error, rather than the number of errors, should guide the decision.

Arabic Documents Test More Than Character Recognition

Testing OCR solutions on English documents and assuming the results apply equally to Arabic documents is an unreliable approach.

Arabic documents have characteristics that make their evaluation a distinct challenge:

• Right-to-left writing direction, which complicates the identification of the correct reading order.
• Connected characters whose shapes change depending on their position within a word.
• Diacritics and variations in typography across printed, handwritten, and designed documents.
• Arabic and English appearing on the same page, as commonly found in official correspondence and contracts.
• Hijri and Gregorian dates appearing alongside different numeral formats.
• Multilevel tables and multicolumn page layouts.
• Stamps, footnotes, and watermarks.
• Variations in scanning quality in historical archives, from yellowed paper to images captured at an angle.

These are not minor considerations.

The KITAB-Bench study, a comprehensive multidomain benchmark for evaluating Arabic OCR and document recognition, demonstrates the importance of genuinely diverse evaluation samples.

Its dataset contains approximately 8,800 samples distributed across nine major domains and 36 subdomains, covering text, tables, charts, and multiple document types.

Any benchmark that evaluates text alone or relies on a single document category provides only a partial picture of real-world performance.

Similarly, the research paper introducing Baseer, MISRAJ's Arabic document understanding model, explains that writing direction, font diversity, diacritics, and complex page structures directly influence extraction quality.

Addressing these challenges requires specialized adaptation rather than merely adjusting a general-purpose model originally developed primarily for other languages.

References: KITAB-Bench | Baseer Research Paper

The Three Levels of Accuracy

The success of an OCR solution cannot be measured solely by the number of characters it recognizes correctly.

It must also be evaluated by its ability to preserve the document's meaning and structure, extract critical fields, and produce outputs that enterprise systems can use with minimal human intervention.

This requires distinguishing between three levels of accuracy.

1. Text Accuracy

Does the tool correctly recognize characters, words, and numbers?

This is the most commonly evaluated dimension and can be measured using:

• Character Error Rate (CER)
• Word Error Rate (WER)

Both metrics are essential, but neither reveals where an error occurs or how it affects downstream operations.

2. Structural Accuracy

Does the tool preserve reading order, the relationships between headings and their associated paragraphs, table rows and columns, merged cells where appropriate, lists, and footnotes?

This dimension can be measured using metrics designed to evaluate table structures, such as Tree-Edit-Distance-based Similarity (TEDS).

TEDS was specifically developed to compare both table structure and content, rather than evaluating extracted text alone.

Reference: Research on TEDS for Table Evaluation

3. Workflow Accuracy

Can the extracted output be used directly in an archive, database, or business workflow without extensive reformatting or manual intervention?

This is the dimension that ultimately matters most to enterprise decision-makers.

It can be evaluated through:

• Accuracy of critical fields rather than overall page accuracy.
• Percentage of documents processed successfully without manual intervention.
• Human review time per document or document batch.
• Percentage of outputs requiring reformatting before use.
• Impact of extraction errors on subsequent workflow steps.
• Successful integration with the target system, rather than merely the availability of export functionality.

A solution may achieve excellent results at the first level and still be unsuitable for enterprise adoption.

Business decisions are not made at the character level. They depend on the operational processes that documents support.

How to Build a Test Sample That Reflects Your Organization

Many evaluations fail before testing even begins because the selected samples do not reflect actual working conditions.

To build a meaningful evaluation dataset:

• Identify document types directly relevant to the intended use case, rather than including every document the organization produces.
• Select easy, moderate, and difficult samples from each category instead of choosing only the best-quality examples.
• Include older, skewed, and low-quality documents, as these are more likely to reveal the solution's actual limitations.
• Include tables, bilingual pages, and documents containing footnotes.
• Define critical fields before reviewing any results to prevent the evaluation criteria from being influenced by the performance of a particular solution.
• Prepare an accurate reference version validated by a human specialist. This should serve as the authoritative ground truth for comparison.
• Avoid evaluating capabilities that the solution does not claim to support, such as handwriting recognition, unless they are genuinely required for the intended use case.

Organizations should preferably begin with one to three document types rather than assembling a large archive that does not support a specific decision.

A thorough evaluation of one relevant document category is more valuable than a broad but unfocused evaluation across many categories.

How to Conduct a Fair Comparison Between Solutions

Comparing OCR solutions without controlling testing conditions produces misleading results, regardless of how precise the reported numbers may appear.

All solutions being compared should receive:

• The same files, image quality, and number of pages.
• The same processing settings wherever possible.
• A predefined required output format rather than each solution's default format.
• Exactly the same list of fields and questions.

When comparing generative models in particular, whose outputs may vary depending on the version and instructions used, the evaluation must document:

• The exact model name and version.
• The date of testing.
• The exact prompts used.
• The number of evaluation runs, since results may vary between runs.
• Any preprocessing applied before submitting the document to the model.
• Any manual corrections made after extraction and before calculating the results.

No comparison results should be published, internally or externally, without specifying the evaluation metric, sample size, and nature of the documents tested.

A number without context is not a reliable basis for a purchasing decision.

The Enterprise Evaluation Scorecard

Consolidating these considerations into a single scorecard helps procurement and governance teams compare solutions against consistent criteria rather than relying on general impressions following a product demonstration.

The relative weights assigned to these dimensions should vary according to the use case, not according to a general preference.

For invoice processing, financial values may receive greater weight than other fields.

For regulations and official circulars, article numbers and issuance dates are often more important than the accuracy of surrounding descriptive text.

The scorecard remains consistent; the weighting is customized.

From Technical Evaluation to Adoption Decisions

Organizations should define their success criteria before testing begins, not after reviewing the results.

Key criteria include:

• Minimum acceptable accuracy for critical fields.
• Maximum percentage of documents that may be routed for human review.
• Maximum acceptable processing time.
• Cost per page or document.
• Data retention requirements and retention periods.
• Deployment requirements, whether on-premises or cloud-based.
• The solution's ability to integrate with the organization's existing Application Programming Interfaces (APIs).

Defining these criteria in advance prevents a common problem: adjusting success thresholds after seeing the results to favor a solution that was already preferred.

A well-designed evaluation should lead to one of three clear decisions rather than a general impression:

• Ready for adoption: The solution meets the requirements of the specific use case for which it was tested.
• Requires customization or improvement: The solution needs further work before adoption, with the required improvements clearly identified.
• Not suitable: The solution does not meet the requirements of the document category or risk level and can be confidently excluded from consideration.

How Do These Evaluation Principles Apply in Practice?

The methodology described above is not merely theoretical.

Baseer, the Arabic document understanding model developed by MISRAJ, was built around a fundamental principle: accurate text extraction alone is not enough.

The goal is to transform documents into structured content that preserves both text and document structure, using Markdown output that can be indexed and integrated directly with downstream systems.

During Baseer's development, the team adopted a multilayered evaluation methodology aligned with the principles discussed in this article.

This approach included measuring text accuracy through CER and WER, evaluating structural accuracy using metrics that account for table structure rather than content alone, and testing diverse Arabic document samples that include tables, bilingual pages, and footnotes, instead of relying on a single clean document.

In published experiments on the Misraj-DocOCR benchmark, an internal benchmark reviewed by human specialists and comprising Arabic document samples of varying difficulty, Baseer achieved a Word Error Rate (WER) of 0.25.

On that benchmark, using the same samples and evaluation metric, Baseer outperformed a range of open-source and commercial solutions.

This result is specific to the benchmark and its evaluation sample. Any comparison in a different context requires independent testing.

However, these findings do not eliminate the most important step for any organization considering Baseer: evaluating it on a representative sample of its own documents using the methodology described in this article before expanding adoption.

Conclusion

A meaningful OCR evaluation does not begin with the easiest page a tool can recognize. It begins with the most challenging document an organization depends on.

The right decision is not based on character accuracy alone. It depends on whether the extracted output can reliably support the intended workflow, with minimal manual intervention and without losing the document's meaning.

Start with a sample of actual business documents, identify the fields where errors are unacceptable, and evaluate text accuracy, structural accuracy, and workflow accuracy together rather than treating them as isolated measures.

If your organization processes large volumes of Arabic documents and is considering evaluating Baseer using this methodology, you can contact the MISRAJ team to design a limited pilot using a representative sample of your documents and predefined, mutually agreed success criteria.