A Survey of Field-based Testing Techniques
Open accessThe DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Abstract
Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements , and how field tests are managed , and identify many challenging research directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The approaches to field testing monitor the applications in production to enrich the test suites with new test cases [16].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [16]
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669
The state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundA brief account of runtime verification
Leucker, M., & Schallhart, C. (2009) · The Journal of Logic and Algebraic Programming · doi:10.1016/j.jlap.2008.08.004 · Cited by 863 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundContinuous software engineering: A roadmap and agenda
Fitzgerald, B., & Stol, K.-J. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.063 · Cited by 723 · cited 1×
“Nowadays, the emergence of more and more agile and distributed paradigms of development, toward the view of a continuous software engineering discipline [35], emphasizes the need for a continuous testing mindset, making developers realize that testing must continue after deployment, and that no clear boundary can be s…”
BackgroundUser experience evaluation methods: current state and development needs
Vermeeren, A. P. O. S., Law, E. L.-C., Roto, V., Obrist, M., Hoonhout, J., & Väänänen-Vainio-Mattila, K. (2010) · Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries · doi:10.1145/1868914.1868973 · Cited by 592 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundA Survey on Metamorphic Testing
Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortes, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 563
Software testing and analysis: process, principles, and techniques
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0935 · Cited by 340
Self-citationBasiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., & Rosenthal, C. (2016) · IEEE Software · doi:10.1109/ms.2016.60 · Cited by 285 · cited 1×
“For example, in NetFlix the concept of Chaos engineering for online testing the robustness of their cloud services has been established a decade ago and still continues to be expanded [9].”
BackgroundOnline Controlled Experiments and A/B Testing
Kohavi, R., & Longbotham, R. (2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_891 · Cited by 179 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
Background
Research works printed but not linked: 5
- Smt. J. Sasi Bhanu , A. Vinaya Babu , and P. Trimurthy . 2015 . A comprehensive architecture for dynamic evolution of online testing of an embedded system . J. Theor. Appl. Inf. Technol. 80 , 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdf Smt. J. Sasi Bhanu, A. Vinaya Babu, and P. Trimurthy. 2015. A comprehensive architecture for dynamic evolution of online testing of an embedded system. J. Theor. Appl. Inf. Technol. 80, 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdf
- 10.5555/829505.830124 · doi:10.5555/829505.830124
- 10.5555/850965.854686 · doi:10.5555/850965.854686
- Alberto Gonzalez-Sanchez , Eric Piel , Hans-Gerhard Gross , and Arjan J. C . van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems . In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10) . IEEE, 37--42. DOI:https://doi.org/10.1109/VALID. 2010 .13 10.1109/VALID.2010.13 Alberto Gonzalez-Sanchez, Eric Piel, Hans-Gerhard Gross, and Arjan J. C. van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems. In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10). IEEE, 37--42. DOI:https://doi.org/10.1109/VALID.2010.13 · doi:10.1109/valid
- Jüri Vain , Leonidas Tsiopoulos , Vyacheslav Kharchenko , Apneet Kaur , Maksim Jenihhin , and Jaan Raik . 2017 . Multi-fragment Markov model guided online test generation for MPSoC . In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17) . 594--607. Jüri Vain, Leonidas Tsiopoulos, Vyacheslav Kharchenko, Apneet Kaur, Maksim Jenihhin, and Jaan Raik. 2017. Multi-fragment Markov model guided online test generation for MPSoC. In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17). 594--607.
Cited by
Latent class analysis in PLS-SEM: A review and recommendations for future applications
Sarstedt, M., Radomir, L., Moisescu, O. I., & Ringle, C. M. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2021.08.051 · Cited by 319
Usability research in educational technology: a state-of-the-art systematic review
Lu, J., Schmidt, M., Lee, M., & Huang, R. (2022) · Educational Technology Research and Development · doi:10.1007/s11423-022-10152-6 · Cited by 143
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Software Engineering by and for Humans in an AI Era
Abrahão, S., Grundy, J., Pezzè, M., Storey, M.-A., & Tamburri, D. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715111 · Cited by 37
“SE might play a more active role in the process of specifying and evaluating such systems in the future.”
Self-citationOnline testing of RESTful APIs: promises and challenges
Martin-Lopez, A., Segura, S., & Ruiz–Cortés, A. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549144 · Cited by 37
“Online testing refers to the testing performed on production software [60, 63], as opposed to offline testing, which is done on a development environment, before the software is released.”
Method
Metadata
- Authors
- Bertolino, A., Braione, P., Angelis, G. D., Gazzola, L., Kifetew, F., Mariani, L., Orrù, M., Pezzè, M., Pietrantuono, R., Russo, S., & Tonella, P.
- Affiliations
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- University of Milano-Bicocca (IT)
- Istituto di Analisi dei Sistemi ed Informatica Antonio Ruberti (IT)
- Fondazione Bruno Kessler (IT)
- Constructor Institute (CH)
- University of Naples Federico II (IT)
- Università della Svizzera italiana (CH)
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2021-05-25
- Volume
- 54
- Issue
- 5
- Pages
- 1-39
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3165293486
- Cited by
- 37
- References
- 106
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 5 · 2023: 10 · 2022: 7 · 2021: 1
- FWCI
- 9.54
- Type
- journal article
- Language
- en
- Open access
- Locations
- ACM Computing Surveys (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- ISTI Open Portal (submittedVersion, other-oa)
- Funders
- Italian MIUR PRIN 2015 (GAUSS - Prot. 2015KWREMX)
- European Research Council (PRECRIME - Advanced Grant 2017 Agreement n. 787703)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- field testing, software testing, software defects, fault detection, software quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Survey of Field-based Testing Techniques. https://app.cochat.ai/papers/10.1145/3447240
- The paper
- Bertolino, A., Braione, P., Angelis, G. D., Gazzola, L., Kifetew, F., Mariani, L., Orrù, M., Pezzè, M., Pietrantuono, R., Russo, S., & Tonella, P. (2022). A Survey of Field-based Testing Techniques. ACM Computing Surveys. https://doi.org/10.1145/3447240
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.