Software quality assurance is complex and multifaceted. Perhaps the quickest definition would be that it is a process that strives to maximize the risk-benefit factor of an IT project.
Our company has released a calculator that focuses on only one aspect of QA - regression testing - and looks into comparison of costs/benefits between manual and automated testing. For a (much) more complete picture of the impact of QA on the overall “profitability” of an IT project, we warmly recommend the reading of an analysis produced by RTI for the National Institute of Software and Technology that can be found here.
The calculator outputs estimates of the costs for different testing scenarios depending on the project parameters that are set by the user.
Here is an example of the output:
Scenario 3: "Test phase" Testing
Regression tests performed only at formal testing sessions
Overall number of test case executions: 5,625
Total cost for bug fixing: £180,000.00
Total testing cost if done manually: £140,625.00
Total testing cost if automated`: £105,000.00
Combined costs if done manually: £320,625.00
Combined costs if automated: £285,000.00
The following are explanations of the main contributing concepts in these calculations:
Amount of testing
How often and how much you test.
The extremes here range from each developer many times a day performing full regression testing, to no testing be done at all until the UAT phase. The calculator will provide you with testing costs for each of these possible scenarios. However, as will be explained below, the savings you make by not testing in the early phases will result in very expensive bug fixing later in the project cyclus.
Manual testing costs
Testing costs if done manually.
In the manual testing scenario, considering that each test execution consumes a chunk of a tester’s time, the testing cost is linearly dependent on the number of test executions, e.i. on the amount of testing.
Automated testing costs
Costs of automated testing.
In the automated testing model, costs of test executions are close to zero as they are performed by computer. The main costs for automated testing fall into three categories: the technology ownership cost, test development costs and test maintenance costs.
Cost of bugs
The cost of impact of the regression bugs in the project.
One of the key factors in the program’s calculations is the fact that the cost of a bug dramatically increases with the “lateness” of its discovery: the later in the project cyclus the bug is discovered, the (much) higher its impact and the total cost to fix it. Here are some arguments behind this thinking:
Cost consists of required developer hours to fix their own bug on the piece of development they are presently working on. The time varies based on complexity, but is usually smaller than fixing a bug found by someone else. When developers discover their own bug, they already understand the problem and know how to fix it.
Cost is increased by system engineer hours. The time is usually at least twice as much, since the problem occurs at a higher level and there is a need to figure out which exact code or configuration is wrong.
Cost combines the hours of the developer, system engineer, PM, and QA hours. This now requires the QA tester to be able to reproduce and document the steps, submit a defect, prioritize the defect, and meet with developers to discuss. Once the developer has fixed the bug, the code needs to be integrated back into the test environment, where testing needs to be redone and the bug needs to be verified as fixed. The defect also needs to be tracked and updated in the defect tracking system.There is also a cost associated with fixing other parts of the code that were built upon the invalid original code.
User Acceptance Testing (UAT)
Cost involves the time of the developer, system engineer, PM, QA hours, customer, and the management. This now requires the user acceptance tester to be able to communicate the bug to the system tester. The system tester will try to reproduce the bug to decide if it's a bug or if it’s working as designed. If the system tester cannot reproduce the bug, he or she will inform the user of their findings. If the system tester can reproduce the bug then he or she must document the steps, submit a defect, prioritize the defect, and meet with developers to discuss. As this is a sign-off phase of the project, the discovered defects usually have high visibility and can produce significant erosion of confidence in the product with very serious consequences on the product’s future.
The calculator produces and compares cost calculations for different testing set-ups. These set-ups are defined by two major factors:
Scenario : In what phase of the development cyclus does the regression testing take place (when the bugs are discovered)
Set-up: How the testing is done (manual or automated)
Here is an overview of the output calculations of the calculator:
|Calculation||Formula||Impacted by scenario|
|Number of test case executions||Number of expected test cases * number of testing rounds over the length of the project||Yes - the more frequently we want to test the more test executions there will be|
|Total cost for bug fixing||Number of expected bugs * cost of a bug for the given testing scenario||Yes - the later we postpone the testing the more expensive it will be to rectify bugs|
|Total testing cost if done manually||Number of test executions * time to complete an execution * tester’s rate||Yes - it is directly proportional to the number of test execution|
|Total testing cost if automated||Number of test cases * (time to develop + time to maintain)* developer’s rate||No - it is the same cost for all scenarios|
|Combined costs if done manually||Total cost for bug fixing + Total testing cost if done manually||Yes - the testing costs go up the earlier you test but at the same time bug costs go down the earlier you test.|
|Combined costs if automated||Total cost for bug fixing + Total testing cost if automated||Yes - although the testing costs are the same, the later you test the higher the cost of bugs|
The user needs to adjust the calculation parameters to reflect their own project. The calculator offers the following parameters for user input:
|Project length in months||The length of the part of the project that involves testing. Generally, from the beginning of the development stage to the end of the project||The longer the project the more testing is required|
|Number of test cases||Estimated or if known, actual number of test cases||The more test cases the more testing|
|Number of developers||Number of developers on the project||Affects the number of executions in the continuous testing scenario, where the developers perform full regression testing before they integrate their development|
|Number of individual developer's full regression test runs per day||How often will a developer run full regression testing suite per day||Ditto|
|Expected number of full regression testing rounds||This depends on how frequently you plan to fully regression test your system. For agile style projects, this should equate to the overall number of sprints.||Impacts the overall number of test executions in “Test phase” testing scenario|
|Expected number of iterations per testing round||In case you find bugs, will you repeat your testing after fixes or will you wait for the next scheduled testing round?||Ditto|
|Expected number of UATs||How many times is the client expected to perform user acceptance testing||Impacts the overall number of test executions in UAT testing scenario|
|Expected number of testing iterations per UAT||Your estimation on how many UAT iterations need to happen before the sign of f||Ditto|
|Expected number of bugs for the production Maximo per year||The average number of bugs that are caught through structured testing for a regular Maximo production run year.||The higher number of bugs the higher overall costs. Note that the estimated number of bugs is independent of testing. Testing only uncovers bugs but does not stop them from happening.|
|Bug cost in continuous testing||Cost of a bug in continuous testing scenario||Impacts the overall cost for this scenario|
|Bug cost in daily code integration testing||Cost of a bug in continuous testing scenario||Impacts the overall cost for this scenario|
|Bug cost in “testing phase” testing||Cost of a bug in continuous testing scenario||Impacts the overall cost for this scenario|
|Bug cost in User Acceptance Testing (UAT)||Cost of a bug in continuous testing scenario||Impacts the overall cost for this scenario|
|Daily rate for automation engineer||This is the cost to your business of using a skilled resource that can create and maintain automated tests.||Impacts the cost of automated testing set-up|
|Average time to develop test in mandays||This largely depends on the technology that is used and also on the skill of the people that are developing tests.||Ditto|
|Average time to maintain test during the project||Ditto||Ditto|
|Licence cost||The cost of ownership of the automated testing technology||Ditto|
|Daily rate for manual tester||Average cost of your manual tester||Impacts the cost of the manual testing set-up|
|Average time to perform a test in mandays||Average time that your manual tester spends on executing and documenting an average scripted test.||Ditto|
We hope you’ll have a go with the calculator and see if your project can benefit from a different set-up and/or different testing scenario. If you have any questions, please leave a comment below.