AIMultiple ResearchAIMultiple Research

Holistic Tech Procurement in 2024: Go Beyond Software Reviews

Reviews are a good input to forming a vendor shortlist but various factors including fake or vendor motivated reviews reduce their reliability. Using additional metrics like how much a vendor’s product is being searched on search engines, their number of employees, their references can help paint a more complete picture quickly. And once tech buyers have a shortlist, they can run a more detailed assessment of these vendors taking into account the product’s feature set, pricing etc.

What are software review aggregators?

Software review aggregators aggregate software reviews which include users’ experiences with software products. Aggregators like G2, Capterra, Software Advice, GetApp etc. operate 2 sided markets. On the one side, vendors provide them with profiles of their products and pay them to get more visibility. There are 3 ways different software review aggregators provide visibility to vendors:

  • Top payers get ranked at the top: All review aggregators offer a way to rank different software solutions. They have various ways to rank companies. Almost all companies calculate a score to estimate product quality like G2’s G2 Score. Other alternative ranking methodologies can be things like number of reviews or alphabetical ranking etc. Some companies like Capterra and Software Advice also have another ranking called “Featured”. In this ranking, companies that pay the most per click are shown at the top of the list to ensure that they get the most clicks and attention from review aggregators’ visitors.
  • Paying companies’ ads are added on other companies profiles: Imagine that you as a vendor create a profile on G2 as a non-paying customer. Later, you will see that paying companies advertising are added to your profile page. So your data which will be used to attract search engine traffic will also be used to advertise your competitor’s products.
  • Selling data on companies showing interest in payer’s categories: G2 also makes use of this method. Using the data on companies browsing their category, paying vendors can guess which roles in those companies could be searching for their product and either target them or email them.

On the other side of the market, these companies collect software reviews.

What is a B2B software review?

A B2B software review explains the benefits and challenges of a B2B software. There are millions of online reviews in sources like G2, Capterra, Gartner etc. One of the largest review aggregators, G2 has collected 1.1 million reviews as of August/2020. Though this number may seem large, compared to the total number of software users, it is tiny. Every internet user uses multiple software products and with ~4 billion software users, at most a few users out of 10,000 users ever leave reviews.

We said at most here because reviewers could be leaving multiple reviews of different products, even reducing the share of users who leave reviews. Another complicaiton is that G2 is not the only review aggregator but other review aggregators are likely to have fewer reviews or they do not disclose their number of reviews.

Why don’t most users write reviews?

Writing reviews without any compensation is a thankless task that a rational human pursuing self-benefit would not engage in.

A sad fact of B2B marketing is that you will never know the the reactions of your most valuable users. They will use your service, pay for it and even if they get great value and benefit out of it, they will never write a review. This is not so surprising since writing a review by an individual yields almost negligible benefits to that individual. The only benefit of writing reviews is that every written review may encourage others to write reviews, too. This may help you evaluate products in the future. So in short by investing 5-10 minutes, you get almost no benefit.

If it is not rational to write reviews, why do reviewers write them?

Why do reviewers write reviews?

We can think of 6 ways that positive reviews get created: vendors writing reviews under pseudonyms (e.g. fake reviews), incentives provided by the brand or review aggregators, complaints or attempts to get support, an emotional connection to the brand, personal branding and altruism.

Reviews by vendors (fake reviews)

I expect but can not positively establish that most reviews by small software providers or in categories with few reviews are fake. When a vendor has no reviews or when there are only a few reviews in a category, the incentives to create a review is quite high. Vendors work with agencies or just write fake reviews themselves. These can be impossible to catch as they could be written by users with multiple reviews and use carefully selected language to make the reviews look genuine. They could also include screenshots or other details that make the reviews appear realistic by mentioning specific details.

I also suspect reviews in large categories to be partially driven by vendors especially in competitive categories where having a few more reviews can move a vendor a few points up in the rankings.

Fake reviews can also be negative. Competitors can easily leave horrible reviews to the competition.

These may seem like hypotheses but there are numerous anecdotes that point to the presence of fake reviews in all areas of business:

  • Any review aggreagator: A search engine query for review agency reveals numerous agencies focusing on increasing clients’ reviews. It is hard to believe that they do something other than generate fake reviews. Some claim to be working with their clients’ clients to get them to submit reviews but that seems implausible. Why should any business need 3rd parties to ask their customers for reviews?
  • Amazon: Fake reviews are quite common on Amazon. The size of Amazon’s marketplace makes it easier to expose the problem even though it probably exists for all review websites.
    • Washington Post highlighted how common Amazon fake reviews are
    • A data breach exposed communication between fake reviewers and businesses involving hundreds thousands of reviews
  • Local businesses: A fake reviewer created London’s highest rated restaurant on Tripadvisor just based on reviews generated by him and his friends.
  • Services: Real estate agencies went from a few bad reviews to hundreds of positive ones after becoming customers of Trustpilot as reported by the Times.

Incentives

Incentives come from two sources: vendors or the review aggregators. One of these parties pay the reviewer for the review. Incentives include Amazon credits, free coffee etc. Incentives create misalignment of incentives between the review reader who wants to hear facts and the reviewer who either

  • wants to minimize his/her effort in leaving a review (in case of review platform incentives)
  • wants to help the brand giving him/her a gift (in case of vendor incentives)

You could argue that these are insignificant incentives but they are not so insignificant. Writing a review is a 5-10 minute affair. Getting $10 for 5 minutes is not a bad outcome for most workers considering that minimum wage is $7-12 per hour in most US states. And if the incentives were not significant there wouldn’t be thousands of people complaining when they did not receive them. Even leaders of review platforms publicly share reviews which underline that reviewers leave reviews for incentives and stop working with brands that don’t provide the promised incentives:

Review shared by Trustradius CEO on how users react to incentives

Incentives introduce bias. Most of the time, reviewers are invited to submit their reviews by the vendor or the review aggregator invites them to submit a review on behalf of the vendor. If you value the incentive and if you are reviewing a company that enabled you to get that incentive, it is impossible to be objective.

We have not crawled the review aggregators to identify what percentage of reviews receive incentives and most review aggregators do not disclose that data. However, there are data points showing that a significant portion of reviewers get incentives. When you look at the reviews about any review aggregator on another review aggregator and you will notice people complaining that they did not get the incentive provided by the review aggregator. Ideally such reviews should also exist on the individual review aggregator website but they probably find a way to declare them invalid. We could prepare a honesty index for a review aggregator by measuring the reviews about itself on its own platforms vs. its reviews on other platforms.

Vendor incentives

Especially incentives provided by vendors can create a strong bias in reviews. Trustradius and Capterra disclose sources of their reviews. According to Trustradius, while ~50% of vendor sourced reviews give the perfect score, only ~20% of independently sourced reviews provide the perfect score.

This is the easiest white hat way to boost a product’s reviews. When vendors ask their customers for reviews, they ask for reviews from the best ones and provide incentives to make sure that they complete reviews. This results in glowing reviews of mediocre products. For example, Gartner has accepted that vendors cherry picked their customers to boost their review scores on Gartner’s review aggregator Peer Insights.

Complaints

Most of my experiences in getting support for incorrect charges or bug fixes were slow and painful. Complaints on social media and review aggregators get more visibility and could be resolved faster. Again, we can only provide anecdotal examples but there are reports of startups prioritizing more visible complaints over less visible ones.

Emotional connection

Some B2C brands have formed lasting connections with their customers. One of my friends raised money for his favorite neighborhood bar during the coronavirus shutdown to make sure that it doesn’t go bust. Such a connection is less likely in the B2B world but those reviews exist. When Google added expiring video call links to its meetings by default, I was very pleasantly surprised as I had a lot of trouble with Teams. However, this was not enough for me to leave a review.

Alturism

Reviews are useful to others and helping others is a human indeed. It is impossible to measure what share of reviews just come from altruistic users. However, it is unlikely to be a significant share given how we, as a society, are showing indifference to subjects that are a lot more important than software reviews such as climate change or income equality.

Personal branding

Some influencers build audiences around their views. However, this is likely to form a minor influence in reviewing as we have not seen any B2B enterprise software review focused influencers. Famous professionals’ views are definitely valued but they tend to share it on Linkedin or their own websites rather than submitting it to a review aggregator where their review will be among hundreds of other reviews.

In short,  a significant reviews are likely to be fake or incentivized by vendors which reduces their accuracy. The only other significant source of reviews can be those that are incentivized by review aggregators. We have not yet done the necessary web crawling to figure out the share of reviews incentivized by review aggregators but happy to do that if you think it will be informative. Just leave us a comment.

How review aggregators respond to these issues?

Most of the time by saying that they have a system in place to manage it. To be honest, this is a complex situation because it is not possible to prove that fake reviews do not represent a significant part of their reviews. Neither them nor any outside party can do that. It is possible to show that some reviews are fake but that would not solve this issue as the fake reviews could be just a few examples.

We have seen a few approaches by the review aggregators:

Trustradius

Out of all B2B software review aggregators, they seem to have the most announcements when it comes to review quality. They are still an independent company (not owned by Gartner) and therefore they can still call out others’ deficiencies. It is harder to do for Gartner’s brands like Capterra or GetApp as they need to be careful in their communication to not harm other Gartner brands.

Their initiatives include:

  • Creating transparency about their reviews by including who suggested the reviewer to Trustradius and whether it is incentivized or not. This is definitely a must for all review aggregators to follow and as far as we can see, most follow this approach.
  • True program: It includes vendors committing to certain basic forms of ethical behavior like committing to representing reviews accurately in their sales and marketing. It is definitely a weel meaning initative but we haven’t seen how these commitments are tracked. Without a tracking mechanism, commitments are not valuable. If you know the mechanism to monitor these, please share in the comments.
  • TrScore: The company claims to use a weighted average of reviews to calculate its final score for a product. This is definitely relevant as old reviews or reviews with less details should have less weight.

Gartner Peer Insights

They have rejected 35% of reviews in 2019 which points to their efforts to combat review fraud. However, we or any one else can not tell whether the ratio should be 35%, 50% or some other number.

G2

G2 shares clear guidelines on their reviews and break down on a high level how they calculate their scores, however we have not seen that they undervalue reviews incentivized by vendors. Given the complexity of their algorithm, this can be overlooked but since this is one of the fundamental ways for vendors to get ahead in reviews, we were expecting them to highlight this. Of course, if this is already clarified, please leave a comment so we correct this.

What are reviews useful for?

Given the complexity of achieving honesty in reviews, it is not reasonable to rely only on them. However, they are definitely useful in:

  • Validating popularity of top software solutions: Solutions with hundreds of reviews across multiple review aggregators points to either an extremely diligent vendor or general adoption of the software. However, this approach can not be used for more niche areas where there are a limited number (a couple hundred or less) reviews in total. A vendor could easily flood the review websites with fake reviews and get ahead in such shallow markets
  • Identifying potential issues: Negative reviews are less likely to be fake. Most vendors have numerous competitors. Spending time and effort to create bad reviews of many software solutions is unlikely to be a productive effort.

What are the limitations of reviews?

We have discussed in detail on how reviews can be manipulated by vendors. Another issue is that getting insights from reviews is slow and time consuming. No one has the time to read tens of reviews of each potential vendor to identify potential issues. To solve this issue, we have started collecting commonly used positive and negative phrases about products. Soon, we will be launching this feature in our lists on our main website.

What is the solution?

Reviews are imperfect and knowing their imperfections help us design better ranking systems. In our product lists, we combine several factors and share our input data transparently to make it easier to choose. The data that go into our product lists include:

  • Number of times different products are searched on search engines
  • Number of online visitors of the vendor
  • Number of employees of the vendor
  • All online reviews of the product
  • Momentum: Changes in all of these metrics over time

While these metrics are not all that we have collected on vendors, this is our day 0. We will continue to build the most transparent and effective list of products in all technology categories.

Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on

Cem Dilmegani
Principal Analyst

Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 60% of Fortune 500 every month.

Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised businesses on their enterprise software, automation, cloud, AI / ML and other technology related decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

To stay up-to-date on B2B tech & accelerate your enterprise:

Follow on

Next to Read

Comments

Your email address will not be published. All fields are required.

0 Comments