Regression Testing

A test methodology for an effective regression testing

1. What is regression testing?

Regression testing is selective retesting of the system with an objective to ensure the bug fixes work and those bug fixes have not caused any un-intended effects in the system.

2. Types of regression testing

There are two types of regression testing that are proposed here even though it is not being practiced or popular.

 A "final regression testing" is being done to validate the gold master builds and "Regression testing" being done to validate the product & failed test cases between system test cycles.

The final regression test cycle is conducted on an "unchanged build for a period of x days" or for a period, which was agreed as the "cook-time" for release. The product is continuously exercised for the complete duration of this cook-time. Some of the test cases are even repeated to find out whether there are failures in the final product that will reach the customer. All the bug fixes for the release should have been completed for the build used for the final regression test cycle. The final regression test cycle is more critical than any other type or phase of testing, as this is the only testing, which ensures "the same build of the product that was tested reaches the customer".

A normal regression testing can use the builds for a period that is exactly needed for the test cases to be executed. However unchanged build is highly recommended for each cycle of regression testing.

3. How to select test cases for regression testing?

It was found that some of the defects reported by customers in the past were due to last minute bug fixes creating side effects and hence selecting the test case for regression testing is really an art and not that easy.

The selection of test cases for regression testing

1. Requires knowledge on the bug fixes and how it affect the system
2. Includes the area of frequent defects
3. Includes the area which has undergone many/recent code changes
4. Includes the area which is highly visible to the users
5. Includes the core features of the product which are mandatory requirements of the customer

Selection of test cases for regression testing depends more on the criticality of bug fixes than the criticality of the defect itself. A minor defect can result in major side effect and a bug fix for an Extreme defect can have no or a minor side effect. So the test engineer needs to balance these aspects for selecting the test cases for regression testing.

When selecting the test cases we should not select more test cases, which are bound to fail and has no or less relevance to the bug fixes. You need to select more positive test cases than negative test cases for final regression test cycle as this may create some confusion and unexpected heat. It is also recommended that the regular test cycles before regression testing should have right mix of both positive and negative test cases. Negative test cases here I mean those test cases which are introduced newly with an intent to break the system.

It is noticed that several companies have "constant test cases set" for regression testing and they are executed irrespective of the number and type of bug fixes. Sometimes this approach may not find all side effects in the system and in sometimes it may be observed that the effort spend on executing test cases for regression testing can be minimized if some analysis is done to find out what test cases are relevant and what are not.

It is a good approach to plan and act for regression testing from the beginning of project before the test cycles. One of the ideas is to classify the test cases into various Priorities based on importance and customer usage. Here I am suggesting the test cases be classified into three categories;

• Priority-0 – Sanity test cases which checks basic functionality and are run for pre-system acceptance and when product goes thru major change. These test cases deliver a very high project value to both engineering dept and to customers.

• Priority-1 – Uses the basic and normal setup and these test cases deliver high project value to both engineering and to customers.

• Priority-2 – These test cases deliver moderate project value. Executed part of ST cycle and selected for regression testing on need basis.

There could be several right approaches to regression testing which needs to be decided on "case to case" basis;

• Case 1: If the criticality and impact of the bug fixes are LOW, then it is enough a test engineer selects a few test cases from TCDB and executes them. These test cases can fall under any Priority (0, 1 or 2).

• Case 2: If the criticality and the impact of the bug fixes are Medium, then we need to execute all Priority-0 and Priority-1 test cases. If bug fixes need additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not a must.

• Case 3: If the criticality and impact of the bug fixes are High, then we need to execute all Priority-0, Priority-1 and carefully selected Priority-2 test cases.

• Case 4: One can also go thru the complete log of changes happened (can be obtained from CM engineer) because of bug fixes and select the test cases to conduct regression testing. This is an elaborate process but can give very good results.

4. Resetting the test cases for regression testing

In a big product release involving several rounds of testing, it is very important to note down what test cases were executed with what build and related information. This is called test case result history. In many organizations not all types of testing and all test cases were repeated for each cycle. In such cases resetting the test cases become very critical for the success of regression testing. Resetting a test case is nothing but setting a flag called NOTRUN or EXECUTE AGAIN with zero base thinking.

RESET of test case, are not expected to be done often. Resetting of the test cases needs to be done with following considerations;

a.    When there is a major change in the product

b.    When there is a change in the build procedure which affect the product

c.    Large release cycle where some test cases were not executed for a long time

d.    You are in the final regression test cycle with a few selected test cases

e.    Where there is a situation the expected results of the test cases could be quite different from previous cycles

When the above guidelines are not met, you may want to RERUN the test cases rather than resetting the results of the test cases. There are only few differences between RERUN and RESET states in test cases, either way the test cases are executed but in case of RESET one has to think zero base and expect different result than what was obtained in earlier cycles and therefore those test cases affect the completion rate of testing. In case of RERUN the management need not worry about completion rate as those test cases can be considered complete except for a formality check and are expected to give same results.

To give you an example, if there is a change in Installation of a product, which does not affect the product functionality, then the change can be tested independently by rerunning some test cases and we don't have to RESET the test cases.

RESET is also decided based on how stable the functionalities are. If you are in Priority-1 and have reached a stage of comfort level on Priority-0 (say for example more than 95% pass rate) then you don't RESET Priority-0 test cases unless there is a major change. This is true with Priority-1 test cases when you are in Priority-2 test phase.

4.1 Pre-system test cycle phase

For pre-system acceptance only Priority-0 test cases are used. For each build that is entering the system test, the build number is selected and all test cases in Priority-0 are reset to NOT RUN. The system test cycle starts only if all pre-system test cases (Priority-0) pass. Test manager or CCB, can decide exceptions if any.

4.2 System test cycle – Priority-1 testing phase

After pre-system acceptance is over, Priority-1 test cases are executed. Priority-1 testing can use multiple builds. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are high. A RESET procedure during this phase may affect all Priority-0 and Priority-1 test cases and these test cases are reset to NOTRUN in TCDB.

4.3 System test cycle – Priority-2 testing phase

Priority-2 testing starts after all test cases in Priority-1 are executed with an acceptable pass % as defined in test plan. In this phase several builds are used. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are very high. A RESET procedure during this phase may affect Priority-0, Priority-1 and Priority-2 test cases. 

4.4 In what way regression testing is related to above three phases?

Regression testing is normally done after Priority-2 testing or for the next release involving only few changes. Resetting test cases during the above phases are not called as regression testing as in my assumption regression comes into picture only after the product is stable. A testing for a release can be decided either by saying a regression testing is sufficient or we can do all phases of testing starting from Priority-0 to Priority-2.

A regression testing for a release can use test cases from all priorities (as mentioned before). A regression testing involving multiple priorities of test cases also needs the test cases executed in strict order i.e. Priority-0 test cases are executed first, Priority-1 next and Priority-2 test cases.

4.5 Why we need to RESET the test cases?

Regression testing uses good number of test cases, which would have been executed already and associated with some results and assumptions on the result. A RESET procedure makes them to NOTRUN so that it gives a clear picture about how much of testing is still remaining, and reflect the results of the regression testing on Zero base.

If test cases are not RESET, then the test engineers tend to report a completion rate and other results based on previous builds. This is because of the basic assumption that multiple builds can be used in each phase of the testing and a gut-feel that if something passed in the past builds, it will pass in future builds also. Regression testing doesn't go with an assumption that "Future is an extension of the past".

5. How to conclude the results of a regression testing?

Regression testing uses only one build for testing (if not it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result;

a.    If the result of a particular test case was PASS using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.

b.    If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked.

c.    If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn't be considered for the pass %. This may also mean that such test cases shouldn't be selected for regression.

d.    If the result of a particular test case is FAIL using the previous builds but works with a documented workaround and

a.    if you are satisfied with the workaround then it should considered as PASS for both system test cycle and regression test cycle

b.    If you are not satisfied with the workaround then it should be considered as FAIL for a system test cycle but can be considered as PASS for regression test cycle.

6. Can we apply the regression test guidelines for patch/upgrade releases?.

The regression guidelines are applicable for both cases where

a.    You are doing a major release of a product, executed all system test cycles and planning a regression test cycle for bug fixes.

b.    You are doing a minor release of a product (CSPs, patches …etc) having only bug fixes, and you are planning for regression test cycles to take care of those bug fixes.

There can be multiple cycles of regression testing that can be planned for each release, if bug fixes come in phases or to take care of some bug fixes not working with specific build.

7. How do I find out which test case to be executed for a particular defect fix?

When failing a test case it is a good practice to enter the defect number(s) along so that you will know what test cases to be executed when a bug fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.

Even though, it is easy to do the mapping between test cases and defects using these mechanisms, the test cases that are to be executed for taking care of side effects of bug fixes, may remain as a manual process as this requires knowledge and several other perspectives discussed earlier in this doc.