Automated testing

Test Automation: ROI, Cost Estimation and Effort Analysis

By Bartosz Szpiech on October, 14 2020
11 minute read

Stay up to date

Back to main Blog
Bartosz Szpiech

Bart is a Microsoft Dynamics fanboy. In 2019, Bart took over 130 international flights to help businesses solve hundreds of issues and show other ways how to assess the problems. Aside of test automation he specializes in Security Role based configuration in AX2012 and D365 providing full project leadership in that area.

In this article you will get to know:

  • What are the immeasurable and measurable benefits of test automation.
  • How to calculate the ROI of automated testing.
  • What are key differences in the effectiveness of manual and automation testing.
  • What are additional factors that you should consider before choosing test automation strategy.

 

Immeasurable and measurable benefits of test automation

 

Decision makers in ERP-managed companies realize how important it is to keep their internal system operational and up-to-date. To get and sustain a good-quality ERP product it is not enough just to buy one. After the implementation of the system and “go-live”, it is almost obligatory to perform testing and in many cases it is mandatory. Sooner or later business owners realize how time-consuming manual testing is, and they face the decision about investing in automation.

 

Moving from manual to automated testing is not “optional” and it plays a crucial part in delivering high quality services. Even so, it’s not mandatory to automate all tests! Automation is unquestionably efficient for testing regression scenarios. They require plenty or repetitive actions taken one by one, frequently with large data sets included. It is not only time consuming for testers, but also gives a place for human mistakes that could be avoided while using automation.

 

Keeping in mind the above, the vast majority of businesses agree that test automation is a good idea. But uncertainty around appropriate resources that should be committed, as well as the cost of building and sustaining an automated testing framework, sometimes prevents moving it forward. Also, it usually leads to questions about the return on the investment: what part of the budget should be dedicated, and how much value will this venture deliver in the future?

 

Measuring the fiscal benefits of test automation can dispel the above doubts. A quantitative justification may be also helpful to consider investing in an automated testing tool, especially for those outside of the technology processes in the organization.

 

How to calculate the ROI of automated testing

 

Here we will explain to you all the details about calculation ROI for test automation. 

For better understanding the automated testing ROI we will refer to the three real-case scenarios. Calculation will be presented based on the comparison of:

  • Manual testing,
  • RSAT - free Microsoft Dynamics 365 FSCM testing tool,
  • and Executive Automats (here and after: EA) - commercial test automation tool.

 

All the data indicated below are prepared based on our practice, and real customers cases.

Test automation requires investment, which means more up-front expenses. Over time, the benefit of being able to repeatedly run your created test cases automatically, without having to dedicate a human resource to it, will significantly reduce your costs leaving manual tests far behind in the ROI calculation.

Planning

Let’s take into consideration an average company with 600 Business Process Scenarios (to test in ERP applications). Surely this number depends on application size and business complexity, but on average it is around 500-600 scenarios.

 

The organization is facing system updates every quarter. Estimating 5 minutes spent on each scenario with (positively assumed) 1 case per scenario that needs to be tested, leads us to 25 days per year dedicated by manual testers (Fig. 1).

Calculating Investment

fig 1

Fig. 1

On average 35%-40% of test cases are able to be covered by RSAT. However, the Microsoft tool does not support negative tests, system warnings, "try-catch" functions, or workflows that are also important to test and may be automated. With Executive Automats, which can handle 100% of what RSAT handles and more, businesses are able to cover 70-80% of their test cases.

For the purposes of our calculation, let’s assume a little bit worse scenario in which EA covers “just” 60% of the testing scope.

 

Getting back to the investments. To start generating savings, the first months have to be dedicated to the preparation of automated test scripts.

 

Depending on the test case complexity and validations type, due to complex process of moving files to LCS and Azure DevOps and further execution, an average Test Script creation in RSAT usually takes between 30-180 minutes (approx. optimistic 60 mins). Due to the superior flexibility and script reusability features that EA provides, tests do not need to be continually re-created from scratch so the average time for Test Script creation is 10-30 minutes. For our example, we’ll say pessimistic 30 minutes.

 

fig 2

Fig. 2

 

It gives accordingly 30 (RSAT) or 18 (EA) days invested in the automation while number of automated test cases with EA (288) is significantly higher than with RSAT (240) (Fig. 2).

But initial script creation is just one part of the process. The second part that is critical, considering the number of system updates needed, is test script maintenance. Companies face changes made into Business Process Scenarios.

The third factor to consider is test execution time, which also demands testing team supervision. These 3 factors considered together is how we receive 36 and 20 days spent on preparation, maintenance, and execution of test scripts accordingly for RSAT (6 additional days) and EA (2 additional days) (Fig. 3 and 4).

Once again the difference between both tools is explained by the EA’s flexibility for recreation.

Meanwhile, manual testing takes just the execution time of each and every scenario. We already calculated that it is 25 days per year.

 

fig 3

Fig.3

 

fig 4

Fig. 4

 

fig 5

Fig. 5

It is noticeable that the first year is invested in the preparation of an test automation framework. On the other hand, each subsequent year brings less time spent on manual testing. That saves the tester's precious time and lets them focus on more challenging tasks.

Calculating Savings

Adding up total time spent for automations and manual testing, and comparing the number with the scenario of having just manual tests, we receive the total accumulated savings.

Eg. year 1 with RSAT gives 15 days spent on manual testing and 36 days on automatic scripts preparation (15 + 36 = 52).

52 - 25 (just manual testing scenario) = -26

fig 6

Fig. 6 

Calculating Return On Investment

With all the costs and savings prepared, we finally get to the ROI.

Return on investment is calculated as savings (gained by replacing manual regression tests with automated tests) divided by the cost of investment into test automation:

 

ROI = Savings(Benefits) / Investment (Costs)

 

Here, to present the ROI already compared with manual testing, the formula takes total accumulated savings (time) against manual testing (Fig. 6) and divides it by the total time invested in automation (Fig. 5).

For EA the first year would be: (-5 / 20 )*100% = -24%

fig 7

Fig. 7

 

At first glance, it may seem expensive to pay for extra weeks of work just for constructing the test framework.

But after running the test automation script more times the deficit is made up. For instance, the ROI for EA starts to be positive during the first year. The difference is noticeable at first sight in the second year:

(-8 / 22 )*100% = 39%

 

ROI for RSAT is also continuously getting higher.

fig 8

Fig. 8

 

As a good business practice, it is important to ensure that ROI is positive during the first three years of the automation project. Any investments in automation that will not start returning in this amount of time should be verified and probably dropped.  

    

ROI is not the only factor. What else should be considered in terms of investment in test automation?

 

The ROI model presented above is intentionally simple and keeps the number of inputs low to make it generally applicable to most of the companies. In reality, many other components can impact ROI but are not incorporated in the calculation. These include eg. qualitative measures as security of data or an additional capacity of testers’ workforces.

 

Many benefits come from advanced regression test automation tools like Executive Automats. Aside from quantitative advantages of using EA over RSAT that are visible at a glance in the calculation, there are other immeasurable benefits, like:

 

  • Scripts simultaneous runs: while RSAT does not really support simultaneous scripts runs, Executive Automats is designed to execute multiple session at once with a number of virtual users. This improves regression test execution times (that is also critical in smoke tests) and allows to schedule hundreds of session to provoke a controlled close-to-real system stress (performance and load tests)
  • "End-to-end business process scenarios" and rollouts: infeasible in RSAT, where every script must be recreated. Easy to execute in EA using puzzle style Scripts joining and embedding or switching between predefined test environments or entities.
  • Cross-platform testing: RSAT is limited to Microsoft Dynamics 365 for Finance and Operations while EA may be used to test different web based apps (ie. D365 Sales, Salesforce, e-commerce, custom platforms etc)
  • Business process documentation: in Executive Automats, script execution steps may be screen-shot (during playback) and automatically combined into Word/PDF files. This process is not available in RSAT / Task Recorder (screen shots only during recording).

 

Time and security are the most valuable assets in business. Automations are constructed to fight against every additional minute that employees waste performing repetitive tasks.

That investment pays itself off with use, and the real profits can be made by providing appropriate test automation tool along with a holistic automation strategy.

 

Companies relying just on manual regression testing, not only fail to achieve these profits but put the product and possibly the organization at risk, especially as a product and business grows.

The ROI model can, at a glance, quickly and easily help business decision makers understand the value that test automation delivers over time and drive the right decision making with regards to automation.

Stay up to date