On testing non-testable programs
Abstract
A frequently invoked assumption in program testing is that there is an oracle (i.e. the tester or an external mechanism can accurately decide whether or not the output produced by a program is correct). A program is non-testable if either an oracle does not exist or the tester must expend some extraordinary amount of time to determine whether or not the output is correct. The reasonableness of the oracle assumption is examined and the conclusion is reached that in many cases this is not a realistic assumption. The consequences of assuming the availability of an oracle are examined and alternatives investigated.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although non-testability is attributed to a lack of reliable test oracle, there are several challenges in testing ML systems.
“Although non-testability is attributed to a lack of reliable test oracle (Weyuker, 1982), there are several challenges in testing ML systems.”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Weyuker, 1982
Cites
Faires, J. D. (1978) · Cited by 6,711 · see citing works
Björck, Å., Dahlquist, G. G., & Anderson, N. (1974) · Cited by 1,719 · see citing works
System structure for software fault tolerance
Randell, B. (1975) · ACM SIGPLAN Notices · doi:10.1145/390016.808467 · Cited by 1,523 · see citing works
Gilb, T. (1977) · Cited by 484 · see citing works
Reliability of the Path Analysis Testing Strategy
Howden, W. E. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233816 · Cited by 401 · see citing works
A Domain Strategy for Computer Program Testing
White, L. J., & Cohen, E. I. (1980) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1980.234486 · Cited by 381 · see citing works
A program structure for error detection and recovery
(2005) · doi:10.1007/bfb0029359 · Cited by 296 · see citing works
An Approach to Program Testing
Huang, J. C. (1975) · ACM Computing Surveys · doi:10.1145/356651.356652 · Cited by 231 · see citing works
Theories of Program Testing and the Application of Revealing Subdomains
Weyuker, E. J., & Ostrand, T. J. (1980) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1980.234485 · Cited by 224 · see citing works
Self-citationMatrix Computations and Mathematical Software
Rice, J. R. (1983) · Medical Entomology and Zoology · Cited by 149 · see citing works
Programming by Refinement, as Exemplified by the SETL Representation Sublanguage
Dewar, R., Grand, A., Liu, S.-C., et al. (1979) · ACM Transactions on Programming Languages and Systems · doi:10.1145/357062.357064 · Cited by 107 · see citing works
Pseudo-oracles for non-testable programs
Davis, M. D., & Weyuker, E. J. (1981) · ACM Annual Conference · doi:10.1145/800175.809889 · Cited by 98 · see citing works
Self-citationMulti-Level Adaptive Techniques (MLAT) for Partial Differential Equations: Ideas and Software
Brandt, A. (1977) · Elsevier eBooks · doi:10.1016/b978-0-12-587260-7.50015-7 · Cited by 74 · see citing works
The applicability of program schema results to programs
Weyuker, E. J. (1979) · International Journal of Parallel Programming · doi:10.1007/bf00995175 · Cited by 63 · see citing works
Self-citationIntroduction to computer science
(1975) · Computer Languages · doi:10.1016/0096-0551(75)90036-3 · Cited by 59 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
Testing scientific software: A systematic literature review
Kanewala, U., & Bieman, J. M. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.05.006 · Cited by 150 · see citing works
- Authors
- Weyuker, E. J.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- Corresponding author
- Weyuker, E. J.
- ORCID
- Weyuker, E. J.
- Venue
- The Computer Journal
- Publisher
- Oxford University Press (OUP)
- Published
- 1982-11-01
- Volume
- 25
- Issue
- 4
- Pages
- 465-470
- ISSN
- 0010-4620 (ISSN-L)
- 1460-2067
- OpenAlex
- W2110441383
- Cited by
- 606
- References
- 15
- Citations by year
- 2026: 17 · 2025: 14 · 2024: 23 · 2023: 20 · 2022: 22 · 2021: 29 · 2020: 20 · 2019: 30 · 2018: 27 · 2017: 21 · 2016: 28 · 2015: 24 · 2014: 25 · 2013: 38 · 2012: 34
- FWCI
- 7.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Computer Journal (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, programming, and type systems
- Keywords
- software testing, oracle problem, program correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
