Top 20 Automation Testing Case Studies in 2024
Cem DilmeganiTransitioning to automated testing is the top priority in the software testing environment because of Its benefits, such as:
- Testing time reduction
- Increase testing coverage
- Higher feedback speed
- Quality improvement
In this article, we will review 20 case studies from different vendors and companies and explore the challenges or goals that they had before implementing test automation. Lastly, we will cover the effect of test automation on their business to guide QA executives regarding their investments in testing automation.
Examples of test automation case studies
The following table has 20 case studies gathered from different companies and vendors. The first company listed is an AIMultiple sponsor.
| Company | Country | Vendors | Industry | Challenges/ Goals | Results |
| large e-commerce platform | – | Testifi | e-commerce | Poor product quality because of manual testing.Because of the platform’s
quality issues, customers were leaving.
| Significant quality improvement. 10 minutes feedback loop. Reduction in the development cycle. |
| DHL | USA | Cypress | Enterprise logistics services | labeling tool for shipping labeling | 65% faster run time. Increased coverage. Increase in the number of tests. |
| Optimizely | USA | Cypress | Software, Website Optimization | Slow and low regression testing coverage. | 4x faster test runs.86% less time writing and debugging tests.40%
increase in new feature coverage.
|
| GoFundMe | USA | Cypress | Crowdfunding | Slow and high test failures. | 30x faster test runs.98-99% reduction in test failures.50% increase in
developers writing tests.
|
| Siemens Software | USA | Cypress | CAD/CAM/CAE/PLM software | A large number of codes. Flaky & slow codes. Many false negatives. | 49% reduction in test code.38% increase in productivity.375% increase in
test execution speed.18 minutes of time saved per execution.290+ person hours saved per quarter
|
| PlanGrid | USA | Cypress | Construction | Slow and hard to maintain codes.required a lot of setup
in the UI.
| 2000+ tests run daily.4-minute average run duration (without parallelization).20+ custom
commands for test efficiency.
|
| Lightstep | USA | Cypress | – | Complex QA process. Time-consuming manual test process. | 8-12k tests run daily. 48x faster deployment validation speed. |
| Slido | Slovakia | Cypress | Pooling application | Slow manual testing. Code migration issues. | 100% coverage of crucial user stories.24x increase in test execution.10x
increase in weekly releases.85% test coverage for all features.
|
| SpotAHome | Spain | Cypress | Rental platform | Flaky & slow codes. Low coverage. Hard to use. | 130+ test runs per week.160+ test scenarios.250 deployments per week.
minutes – run time for all tests.Up to 70% of the time is saved by parallelization.
|
| Gatsby | USA | Cypress | Web development tool | Long pull request review time. Slow manual testing. | 54% reduction in the cumulative age of open pull requests.More
features are shipped earlier and more often.Increase in confidence and reliability.
|
| SaltStack | USA | Cypress | Intelligent automation | Slow manual testing. | 93% reduction in regressions.300 tests written in one month by
a single QA engineer.100% test coverage for all new features.75% of all test cases are automated.
|
| Latitutde CG | USA | Subject7 | Learning Management System | – | Recreate test cases 10 times faster. Doubled the test coverage. |
| US government agency | USA | Subject7 | Government | Slow manual testing. The testing team is not experienced in testing. | No code test automation implementation.Fast automated tests.Continuous feedback because of
automated tests.
|
| Global consulting firm | Global | Subject7 | Consulting | Changing test requirements based on the client’s sector. | Easily modifiable automated tests. |
| Jumping elephants | Canada | Subject7 | IT consulting | Automating repetitive testing scenarios.Conducting unit and performance testing.Testing UI and
admin interface.
| Increase in confidence in the code quality.Reduced the time for
input validation testing.The supported performance tuning process.
|
| Leidos | USA | Subject7 | defense, aviation, information technology | Ensuring cross-browser and cross-environment testing.A large number of test codes.Different
levels of skills among testers.
| 90% increase in productivity.42% savings in testing resources. Enhanced ability
to pursue new client opportunities.
|
| Dovel | USA | Subject7 | IT service management | 976 man hours required for regression testing.Different levels of skills
among testers.
| Reduced regression testing time to 7 machine hours.Regression testing happening
nightly compared to once per release.
|
| Cobb systems group | USA | Subject7 | Software development | Regression testing across multiple datasets and browsers. Ensuring adequate coverage. | Reduced regression testing time to 3-4 days from weeks.Ensured test
execution consistency and coverage.
|
| Large independent wine retailer | USA | Subject7 | e-commerce | Constant maintenance. Not scalable. | Reduced testing cycle time by 60%. Improved release quality. Reduced cost. |
| Motionsoft | USA | Subject7 | CRM provider | Automating smoke test cases to increase test coverage.Improve the slow
testing process ( 2000 manual tests that took weeks to test).
| Execution of 3600 automated tests daily. |
As you observe, automation testing can have substantial positive impacts on a company’s QA process and is used by companies in different industries.
Selenium is an open-source tool for test automation. A common characteristic we noticed in 50% of the case studies mentioned is that the companies initially used or experimented with Selenium. However, companies opted for a different test automation provider because they found out Selenium was either too complicated to use or inefficient for their needs.
We will focus on the common problems observed in the case studies, which were solved by automation testing.
Bad code
According to our observation, developers tend to skip or reduce testing if a company has a hard-to-use or ineffective testing system because it increases the effort required without a clear benefit. An ineffective testing system can harm a company substantially by allowing bugs to reach later development stages which can result in:
- Higher cost of bug fixing
- Lower product quality
- Loss of customers
Automation testing effect: Test automation can reduce the effort required by manual testing. It is reported that in 46% of the cases where test automation was implemented, 50% or more of the manual testing was replaced. Additionally, 55% of companies seeking test automation mention quality improvement as their main strategic driver.
Slow testing
Slow testing is a significant hurdle in the age of agile development and CI/CD. It is estimated that 35% of the testing cycle is spent on manual testing. Slow testing increases the development time and reduces the feedback available for each build of a design.
Automation testing effect: Test automation can significantly increase the number of tests that can be run in a time period. 30% of companies that are pursuing test automation indicate time to market as their main driver.
Workforce with different skill levels
Each company’s testing workforce is different. Within a team, skill levels can be drastically different among members; some can be professional testers with advanced programming knowledge, while others might not know how to program.
Automation testing effect: Test automation tools can provide no/low-code solutions that can benefit non-technical users.
To learn more about software testing best practices, you can read our Top 10 Best Practices for Software Testing.
Further readings
- API Testing: 3 Benefits & 8 Different Types
- Top 9 Metrics That Measure Your Software Testing Efficiency
- Continuous Testing: Definition, Top 7 Benefits, & Challenges
If you need more information regarding automation testing or the details of case studies, you can reach us:
Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 60% of Fortune 500 every month.
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised businesses on their enterprise software, automation, cloud, AI / ML and other technology related decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
To stay up-to-date on B2B tech & accelerate your enterprise:
Follow on
Comments
Your email address will not be published. All fields are required.