AIMultiple ResearchAIMultiple Research

Automation Testing Tools in 2024: Code-based, Robotics & AI

According to a 2017 survey, ±87% of development teams have adopted some level of test automation. Choosing the right tool is one of the first steps in automation testing. 70% of developers state that they use open-source tools like Selenium or Appium, while 52% use automation via frameworks. To choose the right automation tool, it is important to consider the nature and complexity of test cases, as well as ensure that the business infrastructure can accommodate the testing tools. Here, we investigate the types of tools and how to choose the best fit:

Selecting the right tool

When selecting the tool for test automation, it is important to consider the following points:

  • Covered platforms (hardware/software and mobile/desktop)
  • Ease of installation and setup
  • Ease of use and user interface
  • Features like
    • Object storage and maintenance: Features for storage and maintenance of test data
    • Scripting languages (Java, Perl, Python, etc.) supported. While examining this, it is important to consider the level of programming and script-writing expertise of team members
  • Product support and documentation
  • Licensing model and total cost of ownership

It is also important to determine the scope of automation such as the complexity of tests and existing databases.

Fortune 500 companies rely on Testifi as a provider of test automation solutions, including Nokia, Amazon, and BMW. Web & API testing capabilities are provided via their CAST & PULSE products, which have tracking and a real-time performance dashboard.

Script-based testing automation tools

There are several types of test automation tools, each has a different type of testing, level of programming, open source options:


Code-based testing automation tools enable testers to write a code for test cases, reuse it for multiple integration, and customize it according to business goal. Typically, code-based tools support different programming languages and frameworks, and are often open-source with a large community to support users and share codes for test cases. Some code-based automation testing tools also have record-and-playback frameworks to run scripts without coding. However, for complex test cases, testers need to have a high level of programming experience to write and run test case codes. Some of the most popular code-based tools are:

  • Selenium (web application testing)
  • Appium (web, iOS, and Android application testing)

Low code

Low code tools provide a GUI platform to enable non-technical team members to write and run test cases, hence reduce the need for developers in the team. However, low code automation testing platforms typically require set up and minimal coding to tweak built-in codes for specific test cases. Some examples of low-code test automation tools include:

  • Cerberus Testing

No code

No code tools are similar to low-code tools such that they allow for codeless test scripts which can be reused in different environment. Users build tests by using the Graphical User Interface (i.e. by dragging-and-dropping, visual development). Additionally, no code tools do not require any further coding on existing scripts. However, no-code and low-code automation testing tools may not be as customizable as code-based tools where the tester writes a code for specific test cases, therefore, most low-code and no-code tools come with a coding option. Some no-code tools include:

  • CloudQA
  • Katalon Studio
  • Ranorex
  • Subject7
  • TestProject
  • TestComplete
  • TestArchitect
  • TestingWhiz

Here’s a list of features of code-based and code-less automation testing tools:

Test case creationWrite customized test casesCreate tests without code
Programming languagesCreate tests using programming languagesCodeless GUI based tools
Time to write and perform test(These numbers are indicative & based on experience)
~6 hours per test~1 hour per test
Level of programming skill requiredRequires expert level coding skillsDoes not require coding skills
Supported testing typesFunctional and non-functionalFunctional, limited or basic non-functional

Robotic testing automation tools

RPA bots are typically used to run repetitive processes in launched products. However, in test automation, bots eliminate the need for test script creation and maintenance, especially in GUI testing, web applications, and automated regression testing (ART) which aims to check for defects in existing features after software/system updates.

Bots can be integrated to a web browser and programmed to perform tasks typically done by the testers, such as:

  • Filling out and submitting forms
  • Clicking links and checking output page
  • Recognizing text or UI elements on a page
  • Validating the resulting information of an API request

Most RPA tools can also be classified as low/no code testing tools so this is not a very different category than those. RPA tools tend to have more data processing capabilities compared to test automation tools. Some tools that rely on bots to automate tests include:

  • Robot Framework
  • T-Plan

AI-enabled testing automation tools

Leveraging AI/ML algorithms in testing automation typically requires large databases to train the data. However, AI enables advanced features in testing automation tools, such as:

  • Test selection: AI/ML algorithms can scan the software and match it to training data in order to:
    • Suggest test cases
    • Estimate test design
  • Test maintenance: AI/ML algorithms are trained to:
    • Detect GUI defects
    • Monitor software changes
    • Update existing tests to align with changes
    • Update UI elements, field names, test gaps, etc.

Some AI-enabled tools include:

  • Applitools Eyes
  • Functionize
  • Mabl
  • Parasoft SOAtest

For more on quality assurance

To learn more about quality assurance automation, feel free to read our articles:

If you are ready to purchase a solution in this space, you can check out these vendor lists:

Or we can guide you through the process:

Find the Right Vendors

This article was drafted by former AIMultiple industry analyst Alamira Jouman Hajjar.

Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on

Cem Dilmegani
Principal Analyst

Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 60% of Fortune 500 every month.

Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised businesses on their enterprise software, automation, cloud, AI / ML and other technology related decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

To stay up-to-date on B2B tech & accelerate your enterprise:

Follow on

Next to Read


Your email address will not be published. All fields are required.