Supporting Decisions on Regression test Scoping in a Software Product Line Context - from Evidence to Practice

Large software organizations with a product line development approach face many challenges regarding testing. Test managers need to make tradeoffs along three dimensions of repeated testing (abstraction level, time and product space) and consider a range of goals and constraints. In order to keep pace with the decreased development time for new products, which is enabled by the systematic reuse, selective testing of product variants is necessary. A common industrial practice is to base test scoping decisions on practitioners’ expertise and experience. However, as software systems grow in size and complexity, the combinatorial explosion of test possibilities makes it infeasible to assess the reasonableness of the decisions without tool support.
Research on regression test selection propose several systematic strategies for setting a proper test scope when verifying changes of previously tested code. The goal of our research is to improve test management and reduce the amount of redundant testing in the product line context by applying regression test selection strategies. However, despite extensive research on regression testing, gained knowledge has not impacted on industry practices. Thus a secondary goal of our research is to bridge the gap between research and practice in the field of regression testing. Test planning support, like any decision support system, must operate in a complex context and need to be pragmatic, adapted to the context and evolve incrementally within the context.
This thesis explores state of art and state of practice of regression testing and software product line testing, and proposes and evaluates a visualization strategy to support regression test scoping in the product line context. Two extensive systematic literature reviews are conducted as well as four empirical studies in close cooperation with industry. Through visualization of relevant information at a proper
level of detail, test management in general may be supported. A visual analytics tool for test management would also provide a framework which enables research based and context specific regression testing improvements.