University of Minnesota
Software Engineering Center

You are here

On the Danger of Coverage Directed Test Case Generation

Date of Publication: 
March 2012
Associated Research Groups: 
Publication Files: 
In the avionics domain, the use of structural coverage criteria is legally required in determining test suite adequacy. With the success of automated test generation tools, it is tempting to use these criteria as the basis for test generation. To more firmly establish the eectiveness of such approaches, we have generated and evaluated test suites to satisfy two coverage criteria using counterexample-based test generation and a random generation approach, contrasted against purely random test suites of equal size. Our results yield two key conclusions. First, coverage criteria satisfaction alone is a poor indication of test suite effectiveness. Second, the use of structural coverage as a supplement, not a target, for test generation can have a positive impact. These observations points to the dangers inherent in the increase in test automation in critical systems and the need for more research in how coverage criteria, generation approach, and system structure jointly influence test effectiveness.
15th International Conference on Fundamental Approaches to Software Engineering (FASE), Tallinn, Estonia, March 24- April 1, 2012.
@inproceedings{DBLP:conf/fase/StaatsGWH12, author = {Matt Staats and Gregory Gay and Michael W. Whalen and Mats Per Erik Heimdahl}, title = {On the Danger of Coverage Directed Test Case Generation}, booktitle = {FASE}, year = {2012}, pages = {409-424}, ee = {}, crossref = {DBLP:conf/fase/2012}, bibsource = {DBLP,} }