University of Minnesota
Software Engineering Center

You are here

Requirements Coverage as an Adequacy Measure for Conformance Testing

Date of Publication: 
October 2008
Associated Research Groups: 
Publication Files: 
Conformance testing in model-based development refers to the testing activity that verifies whether the code generated (manually or automatically) from the model is behaviorally equivalent to the model. Presently the adequacy of conformance testing is inferred by measuring structural coverage achieved over the model. We hypothesize that adequacy metrics for conformance testing should consider structural coverage over the requirements either in place of or in addition to structural coverage over the model. Measuring structural coverage over the requirements gives a notion of how well the conformance tests exercise the required behavior of the system. We conducted an experiment to investigate the hypothesis stating structural coverage over formal requirements is more effective than structural coverage over the model as an adequacy measure for conformance testing. We found that the hypothesis was rejected at 5% statistical significance on three of the four case examples in our experiment. Nevertheless, we found that the tests providing requirements coverage found several faults that remained undetected by tests providing model coverage. We thus formed a second hypothesis stating that complementing model coverage with requirements coverage will prove more effective as an adequacy measure than solely using model coverage for conformance testing. In our experiment, we found test suites providing both requirements coverage and model coverage to be more effective at finding faults than test suites providing model coverage alone, at 5% statistical significance. Based on our results, we believe existing adequacy measures for conformance testing that only consider model coverage can be strengthened by combining them with rigorous requirements coverage metrics.
Proceedings of the 10th International Conference on Formal Methods and Software Engineering (ICFEM 2008)
@conference{rajan2008requirements, title={Requirements Coverage as an Adequacy Measure for Conformance Testing}, author={Rajan, A. and Whalen, M. and Staats, M. and Heimdahl, M.P.}, booktitle={Proceedings of the 10th International Conference on Formal Methods and Software Engineering}, pages={86--104}, year={2008}, organization={Springer} }