Monday, June 9, 2008

Why Testing is perceived as least agile: What needs to change

A story about how testing failed the business

I was on an agile project. The test team started with great objective of keeping pace with test automation. Iteration 1 to 3 were quite uneventful, every test designed was automated. By this time Customer had first feedback from actual user. Entire UI design was changed, none of the tests were running, development team made changes as per the new UI requirement in a week, it took 2 weeks for automation team to change scripts to make these pass. By this time the test team had 2 weeks of automation backlog. Few more changes and backlog had increased to 6 weeks on a 12 week long project. Project team was getting nervous because of lack of regression testing, and pulled every tester in manual testing.
Later, some business users and developers were available to provide some additional bandwidth. But the automated tool was difficult to learn, proprietary in nature, looking at the scripts it was difficult to understand what the tests were trying to achieve as tests were programs. We couldn’t leverage skills of available resources. Business wanted to push through another release to preempt competition. Lack of automation meant delivery organization needed more time in manual testing effort.
Business was under dilemma- go with release date,may create loss of face or Don’t release give the advantage to competition on a platter.

This is not an isolated case nor is it going to be a one off occurrence in future. As long as current state of testing allows above situations occur, testing will be seen as non agile, may be party poopers or associated with waste.

Business and IT need to think testing strategically

If we see testing in the light of what business or organizations want to do, this perception might change. Testing needs to use business needs to improve efficiency in testing. There are some key business driver for testing as I see from my experience -

• Reduce or stabilize the cost of testing

• Improve release cycle to production. (Time to market, beat the competition)

• Reduce time spent in regression testing

• Reduce defect escape rate - Improve the quality of deployed application.

• Early Feedback of what application does and doesn’t do, allowing business to make informed decisions.

Testing should address Business and IT needs

Well, looking at the above business driver and putting my waste/efficiency hat, I think this is what roughly translates into requirement for testing –

• Testing must keep pace with development

• Test asset created must be able to run on multiple development and deployment platforms when any change happens.

• Business context should drive testing; business must be involved in testing.

• Test must be executable

• Test design and execution happens in real time

• Testing results reported in a way business understands

• Testing should not hinder change in business directions

Testing is a multi skill activity: Testing needs collaboration and innovation

I think I clearly understand the requirement now, however I see there are gaps, sometimes big gaps in the current state of tools and practices which will prevent me to achieve what I wanted to achieve –
• Key Business values and features are often not adequately tested

• Business is not involved in testing

• Test Automation tools lack the ability to write tests in language of business, The intent of tests are not clear.
• Converting test specification or intent of tests into executable tests often requires building a test framework from the ground up.

• Some automation Tools don’t support the programming language used by developers, hindering leveraging developer skills.

• Making changes in executable tests are non trivial activities. Test Automation Engineers spend most of their time maintaining tests. (Waste) rather than adding more tests (Value).

• Cost of tools and lack of flexibility of tools to support multiple programming languages means that organizations can’t leverage combined efforts of development and testing team.

Clearly, this means that testing should be far more collaborative than what it is now and requires far more collaborative effort in building tools and practices which leverages strength of individuals on the team and just not the testers, to make it effective and efficient.


Aslak Helles√ły said...

If a redesign of the UI causes a lot of rework to the suite of automated tests, it usually means that you have coupled the tests too tightly to the UI.

I have painfully experienced this myself in several projects, and it seems to be a common mistake. Here is my advice based on experience.

Automated functional tests that go through the UI are important, but their usage should be limited to a small number "smoke tests": Tests that navigate through the user interface to make sure the UI is correctly hooked up to the underlying business logic. Don't ever use UI tests to verify business logic - or at the very least reduce it to a minimum.

Instead of putting all your automates functional testing eggs in the Watir/Selenium/HTMLUnit/whatever basket, do most of the automated functional testing right below the user interface.

Business logic changes less frequently than UI, and by having the functional tests at the business logic layer, you can reduce the maintenance cost of your tests.

Personally I have had great experience with Fit/Fitnesse and RSpec's story framework for non-UI automated functional tests.

Jason Huggins said...

To Aslak... Amen, brother! People (ab)use Selenium way too much. I'd go one step further and say UI tests would be more useful if repurposed and rebranded as "how-to" documentation, instead of the more broad use of calling it "functional testing".

Eric said...

It's truly amazing how often teams are given a mandate to be more agile without actually assessing the skills, tools and mentality of the test teams.

When things fall apart, too often do test teams fall back on the old UI focused testing and huge, combinatorics generated test suites. It seems like the safest and less-risky approach, because folks are familiar with the terminology and tool names.

However, it's the most risky approach because the time it takes to build the suites of hundreds of functional tests and the time it takes to automate said test is spent not testing. Then the time it takes to update spreadsheets matrices and graphs that provide little information regarding the application's readiness is "empty calories" you've got all of this bloat and documentation, but no real useful information to give the business.

Finally, automation suites (being UI based), go down with any significant release usually not reporting bugs, but a change in the location of a UI element.

Years of seeing software testing as a field for non-technical IT members is part of this. That coupled with the mentality that testing is at the end of a project has caused more failure on projects than poor coding, in my opinion. said...

Thanks for this article. This is very good.
In general, Agile development works with relatively few, very briefly-defined requirements in the form of user stories. Agile’s requirements usually are in the form of user stories, which by definition are very brief and relatively few in the number in keeping with Agile’s focus on just the work to be done in the near term. User story acceptance tests also tend to be limited to the small user story, which is closest in scope to integration tests.
Agile tends not to devote much attention to software design. Instead Agile usually limits its attention to the individual small pieces of code implementing particular features, often without necessarily addressing the design--how those features fit together with each other and with other components. Consequently, typical agile development often lacks system testing. Not surprisingly, such lack of design can cause agile development upstream difficulties when the individual features fail to work together adequately overall.

Source: What is Agile Software Testing?