Task: Analyze Test Failure
This task focuses on locating, isolating and diagnosing test failures and documenting them effectively.
Disciplines: Test

The purpose of this task is to:

  • Investigate the Test Log details and analyze the failures that occurred during test implementation and execution
  • Correct failures that result from errors in testing procedure
  • Record important findings appropriately
RolesPrimary Performer: Additional Performers:
        Examine the Test Logs
        Purpose:  To collate and understand the output from the tests conducted. 

        Start by gathering the Test Logs output during the implementation and execution of the tests. Relevant logs might come from many sources-they might be captured by the tools you use (both test execution and diagnostic tools), generated by custom-written routines your team has developed, output from the Target Test Items themselves, and recorded manually be the tester. Gather all of the available Test Log sources and examine their content. Check that all the scheduled testing executed to completion, and that all the tests were scheduled that should have been.

        Capture Nontrivial Incident Data
        Purpose:  To record the occurrence of any anomalous, nontrivial events for subsequent investigation. 

        It's important to capture any anomalous occurrences-even if you can't reproduce or explain them now, subsequent incidents with similar symptoms will eventually provide enough information to help isolate what the fault is behind them.

        Log as much detail as you can now but indicate that the incident can't yet be resolved.

        Identify Procedural Errors in the Test
        Purpose:  To eliminate human error and other procedural and process errors in the test process from the incident log. 

        It's pretty common that a number of failures will be as a result of errors introduced during the implementation of the test, or in the management of the test environment. Identify and correct these errors.

        If the test has completed abnormally, preventing other tests from being executed, you might need to recover the test close to the point of failure and continue execution of the remaining tests.

        Locate and Isolate Failures
        Purpose:  To identify where the failure is occurring, eliminating Target Test Items from the failure analysis that are not the source of the failure. 

        The more diagnosis of the failure you perform, the more likelihood there will be that the fault will eventually be identified and understood.

        Try to isolate the failure by eliminating Target Test Items that are unlikely to be involved in the failure, and look for trends and characteristics in the remaining items, system status etc.

        Conduct an analysis of the failure by reproducing it under controlled conditions, if the failure cannot be investigated usefully without reproduction. Use diagnostic and debugging tools where helpful.

        Diagnose Failure Symptoms and Characteristics
        Purpose:  To capture a useful analysis of the failure to facilitate fault identification and resolution. 

        Attempt to diagnose the underlying fault using your experience of similar incidents that have occurred.

        If required and available, enlist assistance form developers, taking advantage of the developers' internal knowledge of the software to improve the failure analysis.

        Identify Candidate Solutions
        Purpose:  To provide the person responsible for failure resolution with a better understanding or the nature and impact of the failure, and to assist the developer by providing possible ideas that can optionally be pursued. 

        See Task: Determine Test Results - Create and maintain Change Requests for information on writing effective incident reports and Change Requests.

        Document Your Findings Appropriately
        Purpose:  To present your failure analysis in an appropriate manner for the person responsible for resolving the failure. 

        See Task: Determine Test Results - Create and maintain Change Requests for information on writing effective incident reports and Change Requests.

        Evaluate and Verify Your Results
        Purpose:  To verify that the task has been completed appropriately and that the resulting work products are acceptable. 

        Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".

        Have the people performing the downstream tasks that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input work products to make sure you have represented them accurately and sufficiently. It might be useful to have the author of the input work product review your work on this basis.

        Try to remember that that RUP is an iterative delivery process and that in many cases work products evolve over time. As such, it is not usually necessary-and is often counterproductive-to fully-form a work product that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the work product will change-and the assumptions made when the work product was created proven incorrect-before the work product is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.

        More Information