Jargon Buster: Continuous Testing (CT)

continuous testing

It would appear there is not much to bust here - it is what it is. You just test continuously as all testers do anyway. That’s what they are payed for after all.

Despite its unexciting title, Continuous Testing (CT) is an important SDLC process with the ultimate purpose to give the business an indication of what level of risk exists in a new software release candidate.

CT is a child of the Agile movement and the need of businesses to release their software at much higher frequencies than they would traditionally do. With the good old waterfall approach, you do things in big chunks:  full analysis, overarching design, detail specification, complete build, loads of testing and if all is good, big-bang release it all with the champagne and caviar.

In Agile, you look two to three weeks ahead: pick a few stories, work on them in “just enough” capacity. So you do small chunks of analysis, design, specification and build per each iteration.

Test Coverage

There’s always a catch. One thing you can’t do in small chunks is the testing, more specifically the regression testing. Your small increment in functionality that you put in for this sprint may break a number of existing features. It may not, but how will you know for sure? The only certain way of telling is by retesting all of it.

Cost of Bugs

Another thing - we discussed this in other blogs - is the cost of fixing a bug depending on when you catch it.  If the developer, on his own Dev VM, finds out that they broke the whole system with his latest database change, they don’t even need to tell anyone. They go back to their code and quietly fix it at a minimal cost and fuss to the project. No drama. If the same bug makes it to the day you plan to release, then all sorts of people get involved, project managers, test managers, user representatives. Yes drama. You first need to understand what bit is causing the problem, who built it, can they drop whatever they are doing now to fix it, report back to the upper echelons with estimates, la, la, la. Therefore the sum of the overall costs and time it takes to fix at this stage is much, much higher.

Risk Estimate

The third thing is life itself. In life we take risks, we make do. In life we pay mortgages, go for unforgettable holidays and give iPads to our kids for Christmas. We need to earn money. So when we get to make the “go, no go” decision about the new release, we need to balance the trouble that known bad things in the product may cause us, against all that cash that will be pouring in when the World sees our new sexy functionality.

Continuous Testing is there to address these issues. It aims to:

  • Catch bugs early
  • Give us a good test coverage at reasonable cost
  • Tell us what kind of risk we are taking on by releasing the software with all of its known imperfections


How does it work?

As all good things,  it is a little bit of everything. Here is a diagram that visually shows Agile testing components (credit to Brian Marick - http://www.agiletester.ca/agile-testing )

Agile testing quadrants

As you can see, you need a good mixture of tools and people.

  • Automation -  All of unit testing and a good part of the functional testing needs automation. This automated unit and functional testing is linked with builds and continuous integration: each time you build your software you perform all unit tests. Each time you integrate, you run all automated functional tests.
  • Tools - For performance, load and security testing, you need appropriate tools. Not much choice here.
  • People - User Acceptance Testing, UX testing, Exploratory testing is the domain of humans. Experience and a sadistic desire to break things go a long way here.


Once you have the above in place, you need to design the whole process so that it gives you a solid picture of your risks. You ought to be able to make the following decisions:

  • Do we release or not
  • If we release, what are the worst and best scenarios, how likely they are, what will they cost our business
  • What do we need to fix first and what can wait
  • Are we seeing a pattern, do we need structural changes


Why can’t you do it?

This all may be good and proper but it’s not for you. Your software is specific. You cannot do continuous testing because:

  • We all work on the same Dev environment and can’t have tests run whenever one wants
  • Our tests can’t be executed until all the development is completed as things depend on each other
  • We cannot automate most of the tests; for example it is hard to automate UI testing as it is forever changing


True, these could be genuine obstacles as things stand, but with some restructuring, most can be overcome. There are a great many technical and organisational solutions to these problems. Developers can work in their own environments and integrate periodically. Missing dependencies can be simulated through mocking frameworks. UI tests can be tamed with the right technologies and techniques.

Or it could be that the above is not at all what is stopping you from continuous testing, you have a completely different set of issues.

Whatever’s the case, here’s the thing: you cannot deal with testing only after the development is done. For it to be most useful, the testing needs to be an integral part of your overall design. As soon as you think of the great next thing in your software, you must start thinking how you are going to test it, continuously and effectively. It has to be a forethought, not an afterthought.

Leave a reply

Your email address will not be published. Required fields are marked*