CoChat

The Oracle Problem in Software Testing: A Survey

Open access
Barr, Earl T., Harman, Mark, McMinn, Phil, Shahbaz, Muzammil, Yoo, Shin · 2015 · IEEE Transactions on Software Engineering

The DOI matches its Crossref record; 218 of the 236 references it cites were found in a registry or an index.

Abstract

Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tse.2014.2372785

Cites

218 of 220 references verified0 citation errors · 2 not checkable

Statistics

Figures

Fig. 1 Stimulus and observations: S is anything that can change the observable behavior of the SUT f ; R is anything that can be observed about the system's behavior; I includes f 's explicit inputs; O is its explicit outputs; everything not in S [ R neither affects nor is affected by f .
Fig. 1 Stimulus and observations: S is anything that can change the observable behavior of the SUT f ; R is anything that can be observed about the system's behavior; I includes f 's explicit inputs; O is its explicit outputs; everything not in S [ R neither affects nor is affected by f .CC-BY · From the PDF
Fig. 2 Cumulative number of publications from 1978 to 2012 and research trend analysis for each type of test oracle. The x -axis represents years and y -axis the cumulative number of publications. We use a power regression model to perform the trend analysis. The regression equation and the coefficient of determination ( R 2 ) indicate a upward future trend, a sign of a healthy research community.
Fig. 2 Cumulative number of publications from 1978 to 2012 and research trend analysis for each type of test oracle. The x -axis represents years and y -axis the cumulative number of publications. We use a power regression model to perform the trend analysis. The regression equation and the coefficient of determination ( R 2 ) indicate a upward future trend, a sign of a healthy research community.CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Fig. 3 Chronological introduction of test oracles techniques and concepts.
Fig. 3 Chronological introduction of test oracles techniques and concepts.CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Image 11
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Items 201–220 of 220

Research works printed but not linked: 6

  1. A. Avizienis and L. Chen, "On the implementation of N-version programming for software fault-tolerance during execution," in Proc. 1st Int. Comput. Softw. Appl. Conf., 1977, pp. 149-155.
  2. P. McMinn, M. Stevenson, and M. Harman, "Reducing qualita- tive human oracle costs associated with automatically generated test data," in Proc. Int. Workshop Softw. Test Output Validation, Jul. 2010, pp. 1-4. · DOI
  3. A. J. Offutt, J. Pan, and J. M. Voas, "Procedures for reducing the size of coverage-based test sets," in Int. Conf. Testing Comput. Softw., 1995, pp. 111-123.
  4. M. Shahbaz, P. McMinn, and M. Stevenson, "Automatic genera- tion of valid and invalid test data for string validation routines," Sci. Comput. Programm., vol. 97, no. 4, pp. 405-425, 2015.
  5. J. M. Spivey, Z Notation-a Reference Manual, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1992.
  6. G. V. Bochmann, C. He, and D. Ouimet, "Protocol testing using automatic trace analysis," in Proc. Can. Conf. Elect. Comput. Eng., 1989, pp. 814-820.

Cited by

See all 1,104

Metadata

Authors
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S.
Affiliations
  • University College London (GB)
  • University of Sheffield (GB)
Venue
IEEE Transactions on Software Engineering
Published
2015-05-01
Volume
41
Issue
5
Pages
507-525
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
OpenAlex
W2041713059
Cited by
1,104
References
236
Citations by year
2026: 123 · 2025: 132 · 2024: 116 · 2023: 96 · 2022: 104 · 2021: 114 · 2020: 102 · 2019: 89 · 2018: 98 · 2017: 56 · 2016: 58 · 2015: 16
FWCI
151.12
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Software Engineering (publishedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
Funders
Engineering and Physical Sciences Research Council (EP/I010165/1, EP/I010386/1)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
oracle problem, software testing, metamorphic testing, informal specification, test automation, fault detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.