AIMultiple ResearchAIMultiple Research

OCR Benchmark Methodology in 2024

AIMultiple aims to help enterprises identify the right OCR for their business. These enterprises should expect to process a high volume (i.e. at least tens of thousands of pages per month) of documents and images.

What will be the guiding principles?

AIMultiple’s benchmark methodology is designed for an objective and transparent assessment. It also explains participation requirements.

What will be benchmarked?

Extraction of text in English from documents and images.

Dataset is expected to include 500 pages:

  • 300 pages of long-form PDF documents (e.g. technical manuals, whitepapers, contracts) which include text in image form. PDFs of varying legibility will be used. PDFs will be collected online.
  • 100 pages of transactional documents (e.g. invoices and receipts). They will be collected online and selected from AIMultiple’s and its partners’ documents.
  • 100 pages of handwritten documents (e.g. receipts, insurance claims forms). They will be collected online and selected from AIMultiple’s and its partners’ documents.

In certain documents, parts of the document will be digitally altered to protect PII.

How will AIMultiple perform the benchmark?

First, AIMultiple will run a test batch with a few documents and share its results with the participating vendors to ensure that AIMultiple correctly uses vendors’ services. Then, AIMultiple will run the entire benchmark.

Which criteria will be used?

AIMultiple’s OCR benchmark aims to closely match the preferences of OCR buyers. They want a flexible, cost-effective solution. Therefore, AIMultiple will measure these metrics:

Accuracy

It will be measured by cosine similarity. We will not use Levenshtein distance because different products output texts in different orders especially in case of multi-column text. While Levenshtein distance takes these positional differences into account, we are interested in how accurate the text is detected but not where it is located.

Speed

Average response time and distribution of response times will be measured. A maximum of 5 seconds of data processing and transfer time will be allowed per page.

Scalability

The same metrics may be tested with a fixed number of simultaneous connections. This metric may be similar for all providers (i.e. simultaneous connections may not slow down processing). In such a case, AIMultiple may not publish the results for this metric.

Cost

Public cost data published by the vendors will be used to calculate the cost of the benchmark. Vendors’ pricing models will also be shared to help buyers compare prices of different loads.

Customer service

Reviews on B2B review platforms will be analyzed to assess customer satisfaction.

How will the results be published?

They will be published on AIMultiple.com and will feature graphs that users can leverage to find the right vendor for their business. Top vendors in each of the above categories will be presented.

Each participant will receive

  • their detailed results for each document and page along with timestamps
  • the average results for each document and page
  • the dataset

Please note that AIMultiple is in the design phase of the benchmark and changes will be made as AIMultiple gets end user feedback and finalizes the benchmark.

Reach out to AIMultiple team via info@aimultiple.com if you would like to participate in the AIMultiple OCR benchmark.

Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on

Cem Dilmegani
Principal Analyst

Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 60% of Fortune 500 every month.

Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised businesses on their enterprise software, automation, cloud, AI / ML and other technology related decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

To stay up-to-date on B2B tech & accelerate your enterprise:

Follow on

Next to Read

Comments

Your email address will not be published. All fields are required.

0 Comments