AIMultiple Benchmark Methodology & Its Rationale in 2024

AIMultiple benchmarks aim to make relevant measurements in a transparent and objective manner.

Transparent, data driven benchmarks of product performance are rare. Legacy industry analysts rely on opaque and potentially biased assessments where only these data are published:

– High-level qualitative (i.e. market understanding) and quantitative criteria that products are evaluated against¹

– High-level assessments of these criteria without disclosing the values driving the assessment

These assessments rely on data provided by vendors which have undisclosed commercial relations with analysts.

Therefore the results are subject to numerous issues such as:

• Analyst bias: Analysts evaluate vendor representatives’ responses including qualitative responses. Vendor representatives with commercial relationships with the industry analyst have the chance to build relationships with them by scheduling calls over the year. However, vendor representatives without such commercial relationships would present their product over a single call.
• Conflict of interest: For these assessments, vendor representatives are asked about their private data (e.g. revenues, features, roadmap etc.). Since it would be clear which responses lead to better outcomes for the vendor (e.g. higher product revenues are likely to result in a higher rank), vendor representatives face a conflict of interest.

The B2B tech industry can make better choices with objective and data-driven benchmarks.

How does AIMultiple ensure objectivity?

To ensure that AIMultiple does not favor any solution and does not rely on other sources of income to run the benchmark: Each participant

• Pays a participation fee. There are exceptions when all participants participate freely.
• Provides free access to their solution during the assessment and necessary documentation so AIMultiple can utilize their solution correctly
• Allows AIMultiple to use its logo for the purposes of the benchmark by signing a short contact.
• Has the same exposure to the dataset. Vendors are given enough time to provide an automated answer. For example, in the case of the invoice capture benchmark, each vendor is required to return their response within 5 seconds of receiving a page of an invoice.

AIMultiple allows participants to validate its objectivity with its transparent approach outlined below.

How does AIMultiple ensure transparency?

Detailed results of the assessments (excluding any human judgement where possible) are available to all participating parties in all paid benchmarks.

For example, if the assessment involves measuring a value using automated systems, the participants will all receive these values:

• Assessment timestamps for all participants
• Measured metrics for their product and the average product

Why should you join?

Marketing & sales

To have a document that benchmarks your solution in the market. If the benchmark supports your marketing messages, you can back up marketing claims with 3rd party data gathered via an objective and transparent process.

Product & technology

To understand your product’s strengths and weaknesses in a data-driven manner.

How much is the participation fee?

Depends on the cost of the benchmark and the number of participants. Fees start from free benchmarks where we are testing a technology. Fees can also be a few thousand euros to around ten thousand euros per participant. AIMultiple publishes from one to four benchmark reports per topic. The publishing frequency for specific benchmarks are shared in each benchmark’s related documents.

AIMultiple completes these activities for the benchmarks and distributes their cost to the participants:

• Participant identification
• Data collection
• Data processing
• Results analysis
• Report preparation

Reach out to AIMultiple team via email or Linkedin if you would like to have an AIMultiple benchmark in your domain.

External Links

• 1. How Markets and Vendors Are Evaluated in Gartner Magic Quadrants. Gartner. January 22, 2016. Retrieved June 21, 2023.