Continuous Integration
Objective: replace the old practice of applying quality control after all development has been completed by continuous processes, as small pieces of effort, applied frequently. The advantage of the continuous minimal effort is finding bugs earlier, which results in less effort in correcting them.
A Continuous Integration approach could be built like so:
- Hudson automates the build extracting the code from RedMine.
- Every build runs tests to determine if the last commit breaks old code. This is part of Quality Assurance, just done earlier.
- Note that for proper tests we'll need to deal with the database. Real tests can and must have effect on the database, because they simulate user action. Failed tests may leave the database status dirty or broken. Successful tests need to clean up their inputs. To deal with this, we may:
- use transactions and rollbacks for each test
- use dbUnit more complex but also more powerful. Let's start with transactions and evaluate this as further development.
A possible structure for the environment:
The selenium and hudson servers can coexist. The test server and test clients also can coexist. The four can even all coexist on one pc to start with.
The approach is not intended to be a full scale, up front effort. Rather, we can introduce first the selenium tests, basically reproducing what the testers do every time there's a change. Automating many of the testers actions results in a more complete QA coverage: the testers normally don't run all the tests after a change, but only those related to the new development. This however may result in a bug in an apparently unrelated functionality of the site .
An automated build can either green (success), or red (failure), and generates report pages. If a build fails, it can't be deployed. If a build succeeds, it passes to QA and may then be deployed. The advantages?
- A code change that provokes an error can be immediately corrected, without waiting for the Testers' response.
- More automatic tests mean less Testers' time spent on monotonous tasks. Their time is better spent for testing things that can't be automated.
- We can deploy the builds to a build repository (like artifactory) which is linked from the continuous integration server. We are then very close to generating automatically the distribution. Builds will be linked to features, so that we always know which version offers a new feature for the first time, in case we need to rollback the software to a previous version.
- With DB testing, the state of the database (definitons) is linked to the build. It is therefore always possible to get a build up and running in no time. The actual database data is still to be recovered separately.
The cons?
- Unit testing can be heavy to start using. However, we can introduce it gradually, doing only a login test to start with. We will then build a new test for every code change. Eventually, we'll cover a majority of the code in this way; it would then be plausible to discuss a subproject to complete the testing code coverage.
Other possible tools I'm researching:
- EMMA to provide a code coverage report on the unit tests to see how much of the tests are actually covering our code base. Not sure if it works with CF. Stops us testing in all the wrong areas! And this will probably kick off the above Unit tests as a result.
- Fitnesse will allow BA's and potentially tech savvy Business users a way to automate the functionally tests of the code. FIT testing.
- JProbe to produce reports on performance of the code when these tests have been run against the instances. It works with CF
- Could Findbugs work with ColdFusion?
- A Redmine-Hudson plugin that links the two systems
- Selenium Grid makes it possible to distribute the load of testing on several machines, keeping the hudson automated approach
Some useful references: