A story about how testing failed the business
I was on an agile project. The test team started with great objective of keeping pace with test automation. Iteration 1 to 3 were quite uneventful, every test designed was automated. By this time Customer had first feedback from actual user. Entire UI design was changed, none of the tests were running, development team made changes as per the new UI requirement in a week, it took 2 weeks for automation team to change scripts to make these pass. By this time the test team had 2 weeks of automation backlog. Few more changes and backlog had increased to 6 weeks on a 12 week long project. Project team was getting nervous because of lack of regression testing, and pulled every tester in manual testing.
Later, some business users and developers were available to provide some additional bandwidth. But the automated tool was difficult to learn, proprietary in nature, looking at the scripts it was difficult to understand what the tests were trying to achieve as tests were programs. We couldn’t leverage skills of available resources. Business wanted to push through another release to preempt competition. Lack of automation meant delivery organization needed more time in manual testing effort.
Business was under dilemma- go with release date,may create loss of face or Don’t release give the advantage to competition on a platter.
This is not an isolated case nor is it going to be a one off occurrence in future. As long as current state of testing allows above situations occur, testing will be seen as non agile, may be party poopers or associated with waste.
Business and IT need to think testing strategically
If we see testing in the light of what business or organizations want to do, this perception might change. Testing needs to use business needs to improve efficiency in testing. There are some key business driver for testing as I see from my experience -
• Reduce or stabilize the cost of testing
• Improve release cycle to production. (Time to market, beat the competition)
• Reduce time spent in regression testing
• Reduce defect escape rate - Improve the quality of deployed application.
• Early Feedback of what application does and doesn’t do, allowing business to make informed decisions.
Testing should address Business and IT needs
Well, looking at the above business driver and putting my waste/efficiency hat, I think this is what roughly translates into requirement for testing –
• Testing must keep pace with development
• Test asset created must be able to run on multiple development and deployment platforms when any change happens.
• Business context should drive testing; business must be involved in testing.
• Test must be executable
• Test design and execution happens in real time
• Testing results reported in a way business understands
• Testing should not hinder change in business directions
Testing is a multi skill activity: Testing needs collaboration and innovation
I think I clearly understand the requirement now, however I see there are gaps, sometimes big gaps in the current state of tools and practices which will prevent me to achieve what I wanted to achieve –
• Key Business values and features are often not adequately tested
• Business is not involved in testing
• Test Automation tools lack the ability to write tests in language of business, The intent of tests are not clear.
• Converting test specification or intent of tests into executable tests often requires building a test framework from the ground up.
• Some automation Tools don’t support the programming language used by developers, hindering leveraging developer skills.
• Making changes in executable tests are non trivial activities. Test Automation Engineers spend most of their time maintaining tests. (Waste) rather than adding more tests (Value).
• Cost of tools and lack of flexibility of tools to support multiple programming languages means that organizations can’t leverage combined efforts of development and testing team.
Clearly, this means that testing should be far more collaborative than what it is now and requires far more collaborative effort in building tools and practices which leverages strength of individuals on the team and just not the testers, to make it effective and efficient.