University of Minnesota
Software Engineering Center

You are here

Programs, Tests, and Oracles: The Foundations of Testing Revisited.

Date of Publication: 
May 2011
Associated Research Groups: 
Publication Files: 
In previous decades, researchers have explored the formal foundations of program testing. By exploring the foundations of testing largely separate from any specific method of testing, these researchers provided a general discussion of the testing process, including the goals, the underlying problems, and the limitations of testing. Unfortunately, a common, rigorous foundation has not been widely adopted in empirical software testing research, making it difficult to generalize and compare empirical research. We continue this foundational work, providing a framework intended to serve as a guide for future discussions and empirical studies concerning software testing. Specifically, we extend Gourlay’s functional description of testing with the notion of a test oracle, an aspect of testing largely overlooked in previous foundational work and only lightly explored in general. We argue additional work exploring the interrelationship between programs, tests, and oracles should be performed, and use our extension to clarify concepts presented in previous work, present new concepts related to test oracles, and demonstrate that oracle selection must be considered when discussing the efficacy of a testing process.
33rd International Conference on Software Engineering, Honolulu, Hawaii, May 21-28, 2011. ICSE 2011 Distinguished Paper
@inproceedings{Staats:2011:PTO:1985793.1985847, author = {Staats, Matt and Whalen, Michael W. and Heimdahl, Mats P.E.}, title = {Programs, tests, and oracles: the foundations of testing revisited}, booktitle = {Proceedings of the 33rd International Conference on Software Engineering}, series = {ICSE '11}, year = {2011}, isbn = {978-1-4503-0445-0}, location = {Waikiki, Honolulu, HI, USA}, pages = {391--400}, numpages = {10}, url = {}, doi = {10.1145/1985793.1985847}, acmid = {1985847}, publisher = {ACM}, address = {New York, NY, USA}, keywords = {testing formalism, theory of testing}, }