Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 218 of the 236 references it cites were found in a registry or an index, and 2 are not checked yet.

The Oracle Problem in Software Testing: A Survey

Open access
Barr, Earl T., Harman, Mark, McMinn, Phil, Shahbaz, Muzammil, Yoo, Shin · 2015 · IEEE Transactions on Software Engineering

Abstract

Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tse.2014.2372785

Cites

218 of 220 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 236 references

Research works printed but not linked: 7

  1. Avizienis, 1977, On the implementation of N-version programming for software fault-tolerance during execution,, 149
  2. 10.4018/978-1-60566-060-8.ch170 · doi:10.4018/978-1-60566-060-8.ch170
  3. 10.1007/978-1-4471-3435-0 · doi:10.1007/978-1-4471-3435-0
  4. 10.1145/1868048.1868049 · DOI
  5. Offutt, 1995, Procedures for reducing the size of coverage-based test sets,, Int. Conf. Testing Comput. Softw., 111
  6. Spivey, 1992, Z Notationa Reference Manual
  7. Bochmann, 1989, Protocol testing using automatic trace analysis, 814

Cited by

See all 1,104

Metadata

Authors
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S.
Affiliations
  • University College London (GB)
  • University of Sheffield (GB)
Venue
IEEE Transactions on Software Engineering
Published
2015-05-01
Volume
41
Issue
5
Pages
507-525
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
OpenAlex
W2041713059
Cited by
1,104
References
236
Citations by year
2026: 123 · 2025: 132 · 2024: 116 · 2023: 96 · 2022: 104 · 2021: 114 · 2020: 102 · 2019: 89 · 2018: 98 · 2017: 56 · 2016: 58 · 2015: 16
FWCI
151.12
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Software Engineering (publishedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
Funders
Engineering and Physical Sciences Research Council (EP/I010165/1, EP/I010386/1)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
oracle problem, software testing, metamorphic testing, informal specification, test automation, fault detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Oracle Problem in Software Testing: A Survey. https://app.cochat.ai/papers/10.1109/tse.2014.2372785
The paper
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2014.2372785

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.