University of Minnesota
Software Engineering Center
/

You are here

Coverage Metrics for Requirements-Based Testing: Evaluation of Effectiveness

Date of Publication: 
April 2010
Associated Research Groups: 
Publication Files: 
Abstract: 
In black-box testing, the tester creates a set of tests to exercise a system under test without regard to the internal structure of the system. Generally, no objective metric is used to measure the adequacy of black-box tests. In recent work, we have proposed three requirements coverage metrics, allowing testers to objectively measure the adequacy of a black-box test suite with respect to a set of requirements formalized as Linear Temporal Logic (LTL) properties. In this report, we evaluate the effectiveness of these coverage metrics with respect to fault finding. Specifically, we conduct an empirical study to investigate two questions: (1) do test suites satisfying a requirements coverage metric provide better fault finding than randomly generated test suites of approximately the same size?, and (2) do test suites satisfying a more rigorous requirements coverage metric provide better fault finding than test suites satisfying a less rigorous requirements coverage metric? Our results indicate (1) that test suites satisfying more rigorous coverage metrics provide better fault finding than test suites satisfying less rigorous coverage metrics and (2) only one coverage metric proposed—Unique First Cause (UFC) coverage—is sufficiently rigorous to ensure test suites satisfying the metric outperform randomly generated test suites of similar size.
Publisher: 
NASA
Venue: 
Proceedings of the Second NASA Formal Methods Symposium
bibtex: 
@InProceedings{Staats10:NFM, author = {Matt Staats and Michael W. Whalen and Ajitha Rajan and Mats P.E. Heimdahl}, title = {Coverage Metrics for Requirements-Based Testing: Evaluation of Effectiveness}, booktitle = {Proceedings of the Second NASA Formal Methods Symposium}, publisher = {NASA}, month = {April}, year = {2010} }