AIMultiple ResearchAIMultiple Research

AIMultiple Benchmark Methodology & Its Rationale in 2024

AIMultiple benchmarks aim to make relevant measurements in a transparent and objective manner.

Transparent, data driven benchmarks of product performance are rare. Legacy industry analysts rely on opaque and potentially biased assessments where only these data are published:

– High-level qualitative (i.e. market understanding) and quantitative criteria that products are evaluated against1

– High-level assessments of these criteria without disclosing the values driving the assessment

These assessments rely on data provided by vendors which have undisclosed commercial relations with analysts.

Therefore the results are subject to numerous issues such as:

  • Analyst bias: Analysts evaluate vendor representatives’ responses including qualitative responses. Vendor representatives with commercial relationships with the industry analyst have the chance to build relationships with them by scheduling calls over the year. However, vendor representatives without such commercial relationships would present their product over a single call.
  • Conflict of interest: For these assessments, vendor representatives are asked about their private data (e.g. revenues, features, roadmap etc.). Since it would be clear which responses lead to better outcomes for the vendor (e.g. higher product revenues are likely to result in a higher rank), vendor representatives face a conflict of interest.

The B2B tech industry can make better choices with objective and data-driven benchmarks.

How does AIMultiple ensure objectivity?

To ensure that AIMultiple does not favor any solution and does not rely on other sources of income to run the benchmark: Each participant

  • Pays a participation fee. There are exceptions when all participants participate freely.
  • Provides free access to their solution during the assessment and necessary documentation so AIMultiple can utilize their solution correctly
  • Allows AIMultiple to use its logo for the purposes of the benchmark by signing a short contact.
  • Has the same exposure to the dataset. Vendors are given enough time to provide an automated answer. For example, in the case of the invoice capture benchmark, each vendor is required to return their response within 5 seconds of receiving a page of an invoice.

AIMultiple allows participants to validate its objectivity with its transparent approach outlined below.

How does AIMultiple ensure transparency?

Detailed results of the assessments (excluding any human judgement where possible) are available to all participating parties in all paid benchmarks.

For example, if the assessment involves measuring a value using automated systems, the participants will all receive these values:

  • Assessment timestamps for all participants
  • Measured metrics for their product and the average product

Why should you join?

Marketing & sales

To have a document that benchmarks your solution in the market. If the benchmark supports your marketing messages, you can back up marketing claims with 3rd party data gathered via an objective and transparent process.

Product & technology

To understand your product’s strengths and weaknesses in a data-driven manner.

How much is the participation fee?

Depends on the cost of the benchmark and the number of participants. Fees start from free benchmarks where we are testing a technology. Fees can also be a few thousand euros to around ten thousand euros per participant. AIMultiple publishes from one to four benchmark reports per topic. The publishing frequency for specific benchmarks are shared in each benchmark’s related documents.

AIMultiple completes these activities for the benchmarks and distributes their cost to the participants:

  • Participant identification
  • Data collection
  • Data processing
  • Results analysis
  • Report preparation

Reach out to AIMultiple team via email or Linkedin if you would like to have an AIMultiple benchmark in your domain.

Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on

Cem Dilmegani
Principal Analyst

Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 60% of Fortune 500 every month.

Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised businesses on their enterprise software, automation, cloud, AI / ML and other technology related decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

To stay up-to-date on B2B tech & accelerate your enterprise:

Follow on

Next to Read

Comments

Your email address will not be published. All fields are required.

0 Comments