A Survey of Field-based Testing Techniques
Open accessThe DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Abstract
Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements , and how field tests are managed , and identify many challenging research directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The approaches to field testing monitor the applications in production to enrich the test suites with new test cases [16].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [16]
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669
The state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundA brief account of runtime verification
Leucker, M., & Schallhart, C. (2009) · The Journal of Logic and Algebraic Programming · doi:10.1016/j.jlap.2008.08.004 · Cited by 863 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundContinuous software engineering: A roadmap and agenda
Fitzgerald, B., & Stol, K.-J. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.063 · Cited by 723 · cited 1×
“Nowadays, the emergence of more and more agile and distributed paradigms of development, toward the view of a continuous software engineering discipline [35], emphasizes the need for a continuous testing mindset, making developers realize that testing must continue after deployment, and that no clear boundary can be s…”
BackgroundUser experience evaluation methods: current state and development needs
Vermeeren, A. P. O. S., Law, E. L.-C., Roto, V., Obrist, M., Hoonhout, J., & Väänänen-Vainio-Mattila, K. (2010) · Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries · doi:10.1145/1868914.1868973 · Cited by 592 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundA Survey on Metamorphic Testing
Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortes, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 563
Software testing and analysis: process, principles, and techniques
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0935 · Cited by 340
Self-citationBasiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., & Rosenthal, C. (2016) · IEEE Software · doi:10.1109/ms.2016.60 · Cited by 285 · cited 1×
“For example, in NetFlix the concept of Chaos engineering for online testing the robustness of their cloud services has been established a decade ago and still continues to be expanded [9].”
BackgroundOnline Controlled Experiments and A/B Testing
Kohavi, R., & Longbotham, R. (2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_891 · Cited by 179 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
Background
Research works printed but not linked: 5
- Smt. J. Sasi Bhanu , A. Vinaya Babu , and P. Trimurthy . 2015 . A comprehensive architecture for dynamic evolution of online testing of an embedded system . J. Theor. Appl. Inf. Technol. 80 , 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdf Smt. J. Sasi Bhanu, A. Vinaya Babu, and P. Trimurthy. 2015. A comprehensive architecture for dynamic evolution of online testing of an embedded system. J. Theor. Appl. Inf. Technol. 80, 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdf
- 10.5555/829505.830124 · doi:10.5555/829505.830124
- 10.5555/850965.854686 · doi:10.5555/850965.854686
- Alberto Gonzalez-Sanchez , Eric Piel , Hans-Gerhard Gross , and Arjan J. C . van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems . In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10) . IEEE, 37--42. DOI:https://doi.org/10.1109/VALID. 2010 .13 10.1109/VALID.2010.13 Alberto Gonzalez-Sanchez, Eric Piel, Hans-Gerhard Gross, and Arjan J. C. van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems. In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10). IEEE, 37--42. DOI:https://doi.org/10.1109/VALID.2010.13 · doi:10.1109/valid
- Jüri Vain , Leonidas Tsiopoulos , Vyacheslav Kharchenko , Apneet Kaur , Maksim Jenihhin , and Jaan Raik . 2017 . Multi-fragment Markov model guided online test generation for MPSoC . In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17) . 594--607. Jüri Vain, Leonidas Tsiopoulos, Vyacheslav Kharchenko, Apneet Kaur, Maksim Jenihhin, and Jaan Raik. 2017. Multi-fragment Markov model guided online test generation for MPSoC. In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17). 594--607.
Cited by
Latent class analysis in PLS-SEM: A review and recommendations for future applications
Sarstedt, M., Radomir, L., Moisescu, O. I., & Ringle, C. M. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2021.08.051 · Cited by 319
Usability research in educational technology: a state-of-the-art systematic review
Lu, J., Schmidt, M., Lee, M., & Huang, R. (2022) · Educational Technology Research and Development · doi:10.1007/s11423-022-10152-6 · Cited by 143
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Software Engineering by and for Humans in an AI Era
Abrahão, S., Grundy, J., Pezzè, M., Storey, M.-A., & Tamburri, D. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715111 · Cited by 37
“SE might play a more active role in the process of specifying and evaluating such systems in the future.”
Self-citationOnline testing of RESTful APIs: promises and challenges
Martin-Lopez, A., Segura, S., & Ruiz–Cortés, A. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549144 · Cited by 50
“Online testing refers to the testing performed on production software [60, 63], as opposed to offline testing, which is done on a development environment, before the software is released.”
MethodRuntime Verification and Field-Based Testing for ROS-Based Robotic Systems
Caldas, R., García, J. A. P., Schiopu, M., Pelliccione, P., Rodrigues, G. N., & Berger, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3444697 · Cited by 53
“Process (HOW) Typically, understanding the overhead acceptance criteria follows from contrasting the required performance for delivering the required service, aka nominal performance (e.g., time to reaction, latency, speed), against available resources (e.g., computing power).”
BackgroundMethodInfluentialMicroservices integrated performance and reliability testing
Camilli, M., Guerriero, A., Janes, A., Russo, B., & Russo, S. (2022) · doi:10.1145/3524481.3527233 · Cited by 42
“2According to [12], ex-vivo testing “indicates any type of software testing performed in-house using information extracted from the field”.”
MethodSelf-citationEngineering Digital Systems for Humanity: A Research Roadmap
Autili, M., Sanctis, M. D., Inverardi, P., & Pelliccione, P. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712006 · Cited by 35
“The paper concludes with a concrete research roadmap of digital systems for humanity.”
Continuous deployment in software-intensive system-of-systems
Dakkak, A., Bosch, J., Olsson, H. H., & Mattos, D. I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107200 · Cited by 22
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13
Self-citationSelf-Adaptive Testing in the Field
da Silva, S. S., Pelliccione, P., & Bertolino, A. (2023) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3627163 · Cited by 21
Self-citationAn empirical study of vulnerabilities in edge frameworks to support security testing improvement
Malik, J., & Pastore, F. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10330-x · Cited by 15
“Authorization problems (i.e., CWE-284 in RQ7B ) are frequent and, unfortunately, covered by existing field testing approaches only in the case ofWeb services Bertolino (2007); Bertolino et al. (2021); De Angelis et al. (2011); Zhang (2004).”
BackgroundMethodInfluentialKristensson, P. O., Mjelde, M., & Vertanen, K. D. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584093 · Cited by 16
“Such a deployment study [1] of HCI research translated into a product opens up the potential for rich and unique feedback to the research”
BackgroundTesting Software in Production Environments with Data from the Field
Gazzola, L., Mariani, L., Orrù, M., Pezzè, M., & Tappler, M. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00017 · Cited by 12
Self-citationTesting Abstractions for Cyber-Physical Control Systems
Mandrioli, C., Carlsson, M. N., & Maggio, M. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3617170 · Cited by 12
“During the verification and validation process, engineers spend most time on testing [13, 23, 73].”
BackgroundSelf-adaptive testing in the field
da Silva, S. S., Bertolino, A., & Pelliccione, P. (2022) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1145/3524844.3528050 · Cited by 7
“Moreover, in summarizing open challenges related to field-test cases generation, they conclude that these “shall adapt to the production environment” [6], whereas concerning the oracles for field testing, they observe that these “need”
BackgroundMethodInfluentialSelf-citationDo LLMs Generate Useful Test Oracles? An Empirical Study with an Unbiased Dataset
Molinelli, D., Grazia, L. D., Martin-Lopez, A., Ernst, M. D., & Pezzè, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00031 · Cited by 16
“Writing and maintaining test oracles is demanding and error-prone [3].”
Self-citationArchitecting and Engineering Value-Based Ecosystems
Pelliccione, P., Bühnová, B., Gottschalk, S., Weber, I. M., & Engels, G. (2023) · Springer eBooks · doi:10.1007/978-3-031-36847-9_3 · Cited by 8
Methodologies of the Validation of Software Architectures
Murabet, A. E., & Abtoy, A. (2023) · Journal of Computing Theories and Applications · doi:10.33633/jcta.v1i2.9332 · Cited by 5
“Various MBV techniques are available[20] – [23], each with its own strengths and weaknesses.”
BackgroundArasteh, B., Bulut, F., İnce, İ. F., Sefati, S. S., Kusetoğulları, H., & Kiani, F. (2025) · Journal of Electronic Testing · doi:10.1007/s10836-025-06210-5 · Cited by 3
In vivo test and rollback of Java applications as they are
Bertolino, A., Angelis, G. D., Miranda, B., & Tonella, P. (2023) · Software Testing Verification and Reliability · doi:10.1002/stvr.1857 · Cited by 3
“In vivo testing is a specific kind of field software testing [9] where testing activities are launched directly in the production environment during actual end-user sessions.”
BackgroundMethodInfluentialSelf-citationA family of experiments about how developers perceive delayed system response time
Cornejo, O., Briola, D., Micucci, D., Ginelli, D., Mariani, L., Parrilla, A. S., & Juristo, N. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09660-w · Cited by 4
Self-citationVerification of Autonomous Systems [TC Spotlight]
Araiza-Illan, D., Fisher, M., Leahy, K., Olszewska, J. I., & Redfield, S. (2022) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2022.3143966 · Cited by 2
“…a subset of the possible executions of S ; • Simulation-based Testing — as above but where a simulation of the real environment is used for environmental modelling and interactions [1]; • In-situ Testing — where R is assessed against the actual working of the system in a real-world environment [3].”
MethodFlakiness goes live: Insights from an In Vivo testing simulation study
Barboni, M., Bertolino, A., & Angelis, G. D. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107373 · Cited by 1
Self-citationAn Adaptive Testing Approach Based on Field Data
da Silva, S. S., Caldas, R., Pelliccione, P., & Bertolino, A. (2025) · doi:10.1109/ast66626.2025.00013 · Cited by 1
“As said, we here perform ex-vivo testing [2], which means that we get actual data from the field ( FSR s), but use this data to generate DTMCs representing a patient, and later run the DTMCs to generate DSR s and perform test sessions in-house.”
InfluentialSelf-citationLeveraging AI to Bridge Gaps in Mental Health Counseling for Deaf Populations
Oladunmoye, E. T. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-4317-4.ch011 · Cited by 1
Autonomic Testing: Testing with Scenarios from Production
Qiu, K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3639802 · Cited by 1
“As Bertolino et al. [4] well summarize in their survey, the approaches proposed so far to test software in production mostly focus on the problem of executing test cases without interfering with the running systems, and rely on existing test cases and simple templates to generate test cases with…”
BackgroundAutomatic Test Case Generation for Smart Human-Centric Ecosystems
Xhyra, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00040 · Cited by 1
“Field Testing: Field testing approaches test software systems either in-house with data from the field ( ex-vivo testing) or directly in the field ( in-vivo testing) [10].”
Flakiness Goes Live: Insights from Running Flaky Tests into the FieldPreprint
Barboni, M., Bertolino, A., & Angelis, G. D. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4095579 · Cited by 1
Self-citationReviewing and Proposing Approaches for Self-adaptive Testing in the Field
da Silva, S. S. (2023) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew60843.2023.00035
“Furthermore, in their summary of open challenges in field-based testing, they state that test cases “shall adapt to the production environment” [1], and that, the oracle “need to adapt to the unknown execution conditions that can emerge in the field” [1].”
BackgroundMethodInfluentialTowards Life-long Software Self-validation in Production
Qu, D., Zhao, C., Jiang, Y., & Xu, C. (2024) · Asia-Pacific Symposium on Internetware · doi:10.1145/3671016.3671382
Field-Based Security Testing of SDN Configuration Updates
Malik, J., & Pastore, F. (2025) · IEEE Transactions on Reliability · doi:10.1109/tr.2025.3531654
“Both these two cases are expensive, indeed the first implies that engineers manually produce a model, which is time consuming, the second implies the development of a tool that is tightly integrated with the SDN management software (e.g., OpenDaylight [19]), which is hard to achieve because, in industry, SDN management…”
InfluentialEver-Improving Test Suite by Leveraging Large Language Models
Qiu, K. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728614
“Field testing approaches leverage execution traces to test the actual behavior of software systems and enrich the test suites with scenarios from production [2, 18].”
Automatic Curtain Slider Based on Light Intensity to Improve Human Health and Circadian Rhythm
Wibisana, E. F. S., Pranoto, T. C., Putri, N. A., & Widianto, M. H. (2025) · International Conference on Electronic and Electrical Engineering and Intelligent System (ICE3IS) · doi:10.1109/ice3is66769.2025.11281328
Live Testing of Cloud Services: A Framework and Lessons Learned
Jebbar, O., & Khendek, F. (2025) · IEEE Future Networks World Forum (FNWF) · doi:10.1109/fnwf66845.2025.11317173
“Such a choice is guided by what the authors referred to in [5] as test isolation policy.”
InfluentialBias, Fairness, and Accountability in Forensic AI Systems
Jain, A. K., & Vijay, P. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-2278-8.ch005
AdapTSoS: Adaptive Testing of Systems of Systems
da Silva, S. S., Lonetti, F., & Bertolino, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-20631-2_23
Self-citation
Metadata
- Authors
- Bertolino, A., Braione, P., Angelis, G. D., Gazzola, L., Kifetew, F., Mariani, L., Orrù, M., Pezzè, M., Pietrantuono, R., Russo, S., & Tonella, P.
- Affiliations
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- University of Milano-Bicocca (IT)
- Istituto di Analisi dei Sistemi ed Informatica Antonio Ruberti (IT)
- Fondazione Bruno Kessler (IT)
- Constructor Institute (CH)
- University of Naples Federico II (IT)
- Università della Svizzera italiana (CH)
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2021-05-25
- Volume
- 54
- Issue
- 5
- Pages
- 1-39
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3165293486
- Cited by
- 37
- References
- 106
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 5 · 2023: 10 · 2022: 7 · 2021: 1
- FWCI
- 9.54
- Type
- journal article
- Language
- en
- Open access
- Locations
- ACM Computing Surveys (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- ISTI Open Portal (submittedVersion, other-oa)
- Funders
- Italian MIUR PRIN 2015 (GAUSS - Prot. 2015KWREMX)
- European Research Council (PRECRIME - Advanced Grant 2017 Agreement n. 787703)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- field testing, software testing, software defects, fault detection, software quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.