Construct Validity in Software Engineering Research and Software Metrics
Abstract
Construct validity is essentially the degree to which our scales, metrics and instruments actually measure the properties they are supposed to measure. Although construct validity is widely considered an important quality criterion for most empirical research, many software engineering studies simply assume that proposed measures are valid and make no attempt to assess construct validity. Researchers may ignore construct validity because evaluating it is intrinsically difficult, or due to lack of specific guidance for addressing it. In any case, some research inevitably produces erroneous conclusions, because due to invalid measures. This article therefore attempts to address these problems by explaining the theoretical basis of construct validity, presenting a framework for understanding it, and developing specific guidelines for assessing it. The paper draws on a detailed example involving 15 software metrics, which ostensibly measure the size, coupling and cohesion of Java classes.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditional metrics such as lines of code, number of tasks, or bugs fixed have been criticized for lacking validity and context.
“traditional metrics such as lines of code, number of tasks, or bugs fixed have been criticized for lacking validity and context (Petersen, 2011; Ralph & Tempero, 2018).”
From Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study (2026) · cites this paper as Ralph & Tempero, 2018
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · see citing works
An Index of Factorial Simplicity
Kaiser, H. F. (1974) · Psychometrika · doi:10.1007/bf02291575 · Cited by 14,898 · see citing works
Jolliffe, I. T. (2005) · Wiley eBooks · doi:10.1002/0470013192.bsa501 · Cited by 14,522 · see citing works
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · see citing works
Moore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150 · see citing works
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,628 · see citing works
A metrics suite for object oriented design
Chidamber, S. R., & Kemerer, C. F. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.295895 · Cited by 5,698 · see citing works
Validating Instruments in MIS Research1
Straub, D. W. (1989) · MIS Quarterly · doi:10.2307/248922 · Cited by 2,549 · see citing works
Software Metrics: A Rigorous and Practical Approach
Fenton, N., & Pfleeger, S. L. (2013) · Cited by 2,517 · see citing works
The Logic of Scientific Discovery.
Rescher, N., Popper, K. R., & Hanson, N. R. (1960) · Philosophy and Phenomenological Research · doi:10.2307/2104331 · Cited by 2,455 · see citing works
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · see citing works
Object-oriented metrics that predict maintainability
Li, W., & Henry, S. M. (1993) · Journal of Systems and Software · doi:10.1016/0164-1212(93)90077-b · Cited by 1,080 · see citing works
Carron, A. V., Widmeyer, W. N., & Brawley, L. R. (1985) · Journal of Sport Psychology · doi:10.1123/jsp.7.3.244 · Cited by 1,050 · cited 2× · see citing works
“For example, team cohesion refers to a group’s tendency to stick together in pursuing its goals [6].”
BackgroundStanford encyclopedia of philosophy
(2000) · Choice Reviews Online · doi:10.5860/choice.37sup-195 · Cited by 976 · see citing works
Towards a metrics suite for object oriented design
Chidamber, S. R., & Kemerer, C. F. (1991) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/117954.117970 · Cited by 841 · see citing works
A unified framework for coupling measurement in object-oriented systems
Briand, L., Daly, J. W., & Wüst, J. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.748920 · Cited by 816 · see citing works
Software Structure Metrics Based on Information Flow
Henry, S. M., & Kafura, D. G. (1981) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1981.231113 · Cited by 773 · see citing works
Stevens, W. P., Myers, G. J., & Constantine, L. L. (1979) · Cited by 610 · see citing works
(1968) · PMLA/Publications of the Modern Language Association of America · doi:10.1632/s0030812900035720 · Cited by 590 · see citing works
Exploratory Factor Analysis in Behavior Genetics Research: Factor Recovery with Small Sample Sizes
Preacher, K. J., & MacCallum, R. C. (2002) · Behavior Genetics · doi:10.1023/a:1015210025234 · Cited by 423 · see citing works
Learning a Metric for Code Readability
Buse, R. P. L., & Weimer, W. R. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.70 · Cited by 396 · see citing works
The Future of Empirical Methods in Software Engineering Research
Sjøberg, D. I. K., Dybå, T., & Jørgensen, M. (2007) · Future of Software Engineering (FOSE) · doi:10.1109/fose.2007.30 · Cited by 390 · see citing works
Software Engineering Metrics: What Do They Measure and How Do We Know?
Kaner, C., & Bond, W. P. (2004) · Cited by 256 · see citing works
Validity threats in empirical software engineering research - An initial survey
FELDT, R. H., & Magazinius, A. (2010) · Chalmers Publication Library (Chalmers University of Technology) · Cited by 188 · see citing works
Measuring the Coupling and Cohesion of an Object-Oriented Program Based on Information Flow
Lee, Y. S. (1995) · Cited by 187 · see citing works
Using Cognitive Principles to Guide Classification in Information Systems Modeling1
Parsons, J., & Wand, Y. (2008) · MIS Quarterly · doi:10.2307/25148874 · Cited by 130 · see citing works
Sedano, T., Ralph, P., & Péraire, C. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.20 · Cited by 126 · see citing works
Self-citationRoberts, G. C. K. (2012) · Springer eBooks · doi:10.1007/978-3-642-16712-6_100780 · Cited by 106 · see citing works
Toward Methodological Guidelines for Process Theories and Taxonomies in Software Engineering
Ralph, P. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2796554 · Cited by 94 · see citing works
Self-citationThe sensemaking-coevolution-implementation theory of software design
Ralph, P. (2015) · Science of Computer Programming · doi:10.1016/j.scico.2014.11.007 · Cited by 60 · see citing works
Self-citationProceedings of the 1997 4th International Software Metrics Symposium
anon, A. (1999) · Computer Standards & Interfaces · doi:10.1016/s0920-5489(99)92176-1 · Cited by 45 · see citing works
Ralph, P. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.06.010 · Cited by 33 · see citing works
Self-citationDeveloping and evaluating software engineering process theories
Ralph, P. (2015) · doi:10.5555/2818754.2818760 · Cited by 10 · see citing works
Self-citationUsing cognitive principles to guide classification in information systems modeling
ParsonsJeffrey, & WandYair (2008) · MIS Quarterly · doi:10.5555/2017399.2017409 · Cited by 8 · see citing works
A Model for Defining Coupling Metrics
Tempero, E. D., & Ralph, P. (2016) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2016.030 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
1 of the 36 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Ralph, P., Baltes, S., Adisaputri, G., Torkar, R., Kovalenko, V., Kalinowski, M., Novielli, N., Yoo, S., Devroey, X., Tan, X., et al. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09875-y · Cited by 193 · see citing works
Self-citationKosar, T., Ostojić, D., Liu, Y. D., & Mernik, M. (2024) · Mathematics · doi:10.3390/math12050629 · Cited by 98 · see citing works
Šmite, D., Tkalich, A., Moe, N. B., Papatheocharous, E., Klotins, E., & Buvik, M. P. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111197 · Cited by 87 · see citing works
Construct Validity in Software Engineering
Sjøberg, D. I. K., & Bergersen, G. R. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3176725 · Cited by 70 · see citing works
A grounded theory of coordination in remote-first and hybrid software teams
Santos, R. E. S., & Ralph, P. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510105 · Cited by 66 · see citing works
Self-citationOn the usage, co-usage and migration of CI/CD tools: A qualitative analysis
Mazrae, P. R., Mens, T., Golzadeh, M., & Decan, A. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10285-5 · Cited by 58 · see citing works
Alotaibi, S., Omer, S., & Su, Y. (2022) · Electricity · doi:10.3390/electricity3030020 · Cited by 57 · see citing works
Factors Affecting On-Time Delivery in Large-Scale Agile Software Development
Kula, E., Greuter, E., Arie Van Deursen, A., & Gousios, G. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3101192 · Cited by 48 · see citing works
Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines
Graziotin, D., Lenberg, P., FELDT, R. H., & Wagner, S. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3469888 · Cited by 48 · see citing works
Advancing Static Code Analysis With Language-Agnostic Component Identification
Schiewe, M., Curtis, J., Bushong, V., & Cerny, T. (2022) · IEEE Access · doi:10.1109/access.2022.3160485 · Cited by 36 · see citing works
Metamorphic Testing for Web System Security
Chaleshtari, N. B., Pastore, F., Göknil, A., & Briand, L. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3256322 · Cited by 29 · see citing works
Furia, C. A., Torkar, R., & FELDT, R. H. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3490953 · Cited by 29 · see citing works
Do I really need all this work to find vulnerabilities?
Elder, S., Zahan, N., Shu, R., Metro, M., Kozarev, V., Menzies, T., & Williams, L. A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10179-6 · Cited by 28 · see citing works
Not All Requirements Prioritization Criteria are Equal at All Times: A Quantitative Analysis
Svensson, R. B., & Torkar, R. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111909 · Cited by 33 · see citing works
Method-Level Bug Severity Prediction using Source Code Metrics and LLMs
Mashhadi, E., Ahmadvand, H., & Hemmati, H. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00055 · Cited by 23 · see citing works
Like, dislike, or just do it? How developers approach software development tasks
Masood, Z., Hoda, R., Blincoe, K., & Damian, D. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106963 · Cited by 22 · see citing works
A machine learning approach for hierarchical classification of software requirements
Binkhonain, M., & Zhao, L. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100457 · Cited by 22 · see citing works
An empirical study on maintainable method size in Java
Chowdhury, S., Uddin, G., & Holmes, R. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3527975 · Cited by 20 · see citing works
An empirical study on bug severity estimation using source code metrics and static analysis
Mashhadi, E., Chowdhury, S., Modaberi, S., Hemmati, H., & Uddin, G. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112179 · Cited by 15 · see citing works
A framework for defining coupling metrics
Tempero, E. D., & Ralph, P. (2018) · Science of Computer Programming · doi:10.1016/j.scico.2018.02.004 · Cited by 15 · see citing works
Self-citationLeveraging Intermediate Artifacts to Improve Automated Trace Link Retrieval
Rodriguez, A. D., Cleland‐Huang, J., & Falessi, D. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00014 · Cited by 14 · see citing works
State of the Practice in Software Testing Teaching in Four European Countries
Tramontana, P., Marín, B., Paiva, A. C. R., Mendes, A. S., Vos, T. E. J., Amalfitano, D., Cammaerts, F., Snoeck, M., & Fasolino, A. R. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00015 · Cited by 13 · see citing works
Architecture decisions in quantum software systems: An empirical study on Stack Exchange and GitHub
Aktar, M. S., Liang, P., Waseem, M., Tahir, A., Ahmad, A., Zhang, B., & Li, Z. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107587 · Cited by 12 · see citing works
Social science theories in software engineering research
Lorey, T., Ralph, P., & Felderer, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510076 · Cited by 12 · see citing works
Self-citationEmpirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods
ter Beek, M. H., & Ferrari, A. (2022) · Software · doi:10.3390/software1040017 · Cited by 12 · see citing works
Distributed state model inference for scriptless GUI testing
Ricós, F. P., Slomp, A., Marín, B., Aho, P., & Vos, T. E. J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111645 · Cited by 12 · see citing works
Seth, A., Bhattacharya, S., Elder, S., Zahan, N., & Williams, L. A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10621-5 · Cited by 11 · see citing works
Improving the Reporting of Threats to Construct Validity
Sjøberg, D. I. K., & Bergersen, G. R. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3593449 · Cited by 10 · see citing works
Automated, Cost-effective, and Update-driven App Testing
Ngo, C. D., Pastore, F., & Briand, L. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3502297 · Cited by 10 · see citing works
Understanding Fairness in Software Engineering: Insights from Stack Exchange Sites
Sesari, E., Sarro, F., & Rastogi, A. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686687 · Cited by 8 · see citing works
Potential implementation of EVs – Features, Challenges and User perspective
Alotaibi, S., Omer, S., Aljarallah, S., & Su, Y. (2021) · IEEE Green Energy and Smart Systems Conference (IGESSC) · doi:10.1109/igessc53124.2021.9618697 · Cited by 8 · see citing works
Exploring potential implications of intelligent tools for human aspects of software engineering
Melegati, J., Nascimento, N., Chanin, R. M., Sales, A., & Wiese, I. (2024) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3641822.3641877 · Cited by 7 · see citing works
Seth, A., Bhattacharya, S., Elder, S., Zahan, N., & Williams, L. A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4306114 · Cited by 7 · see citing works
Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study
Stray, V., Brandtzæg, E. G., Wivestad, V., Barbala, A., & Moe, N. B. (2026) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2026.880 · Cited by 9 · see citing works
“While productivity is often defined as output per unit of time (Meyer et al., 2014), traditional metrics such as lines of code, number of tasks, or bugs fixed have been criticized for lacking validity and context (Petersen, 2011; Ralph & Tempero, 2018).”
Do Test and Environmental Complexity Increase Flakiness? An Empirical Study of SAP HANA
Berndt, A., Bach, T., & Baltes, S. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3695407 · Cited by 6 · see citing works
On technical debt in mathematical programming: An exploratory study
Vidoni, M. C., & Cunico, M. L. (2022) · Mathematical Programming Computation · doi:10.1007/s12532-022-00225-1 · Cited by 6 · see citing works
Delta GUI change detection using inferred models
Ricós, F. P., Marín, B., Vos, T. E. J., Neeft, R., & Aho, P. (2024) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103925 · Cited by 6 · see citing works
Dynamic Prediction of Delays in Software Projects using Delay Patterns and Bayesian Modeling
Kula, E., Greuter, E., van Deursen, A., & Gousios, G. (2023) · ESEC/SIGSOFT FSE · doi:10.1145/3611643.3616328 · Cited by 8 · see citing works
On the comprehensibility of functional decomposition: An empirical study
Tempero, E. D., Denny, P. C., Finnie-Ansley, J., Luxton-Reilly, A., Kirk, D., Leinonen, J., Shakil, A., Sheehan, R., Tizard, J., Tu, Y., et al. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644432 · Cited by 5 · see citing works
Self-citationModeling Team Dynamics for the Characterization and Prediction of Delays in User Stories
Kula, E., Arie Van Deursen, A., & Gousios, G. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678939 · Cited by 6 · see citing works
Context-Aware Automated Sprint Plan Generation for Agile Software Development
Kula, E., Arie Van Deursen, A., & Gousios, G. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695540 · Cited by 5 · see citing works
Hyrynsalmi, S. M., Liebel, G., Santos, R. D. S., & Baltes, S. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1109/icse-seis66351.2025.00016 · Cited by 5 · see citing works
The Impact of Forced Working-From-Home on Code Technical Debt: An Industrial Case Study
Zabardast, E., González‐Huerta, J., & Palma, F. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00054 · Cited by 5 · see citing works
Understanding Remote Work Experience: Insights Into Well‐Being
Vizcaíno, A., Suárez-Albanchez, J., Šmite, D., & García, F. O. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.2757 · Cited by 4 · see citing works
Information-theoretic detection of unusual source code changes
Torres, A., Baltes, S., Treude, C., & Wagner, M. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10644-y · Cited by 4 · see citing works
Examining ownership models in software teams
Koana, U. A., Le, Q. H., Rahman, S., Carlson, C., Chew, F., & Nayebi, M. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10538-5 · Cited by 5 · see citing works
Are cloud providers exploiting open-source? An exploratory study of Redis license change
Han, F., Zhao, S., Xia, X., & Wang, W. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107951 · Cited by 3 · see citing works
An Industrial Experience Leveraging the iv4XR Framework for BDD Testing of a 3D Sandbox Game
Ricós, F. P., Marín, B., Prasetya, I. S. W. B., Vos, T. E. J., Davidson, J., & Hovorka, K. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-59465-6_24 · Cited by 3 · see citing works
Taming Timeout Flakiness: An Empirical Study of SAP HANA
Berndt, A., Baltes, S., & Bach, T. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3639741 · Cited by 3 · see citing works
Evidence is All We Need: Do Self-Admitted Technical Debts Impact Method-Level Maintenance?
Chowdhury, S., Kidwai, H., & Asaduzzaman, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00118 · Cited by 3 · see citing works
Philip, M. M., Natarajan, K., Ramanathan, A., & Balakrishnan, V. (2019) · IET Software · doi:10.1049/iet-sen.2019.0006 · Cited by 3 · see citing works
Challenges of Working From Home in Persistent Covid-19 Environment
Chinnaiah, P. M., & Smt. Chythra, P. (2021) · International Research Journal of Business Studies · doi:10.21632/irjbs.14.1.85-97 · Cited by 3 · see citing works
Stochel, M. G., Wawrowski, M. R., & Chołda, P. (2022) · Applied Sciences · doi:10.3390/app122211347 · Cited by 3 · see citing works
Improving hardware/software interface management in systems of systems through documentation as code
Cadavid, H., Andrikopoulos, V., & Avgeriou, P. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10350-7 · Cited by 3 · see citing works
Broken Agreement: The Evolution of Solidity Error Handling
Mitropoulos, C., Kechagia, M., Maschas, C., Ioannidis, S., Sarro, F., & Mitropoulos, D. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686686 · Cited by 3 · see citing works
Teaching Software Metrology: The Science of Measurement for Software Engineering
Ralph, P., Kuutila, M., Arif, H., & Ayoola, B. (2024) · Springer eBooks · doi:10.1007/978-3-031-71769-7_5 · Cited by 4 · see citing works
Self-citationEmpowering software startups with agile methods and practices: A design science research
Gandomani, T. J., Zulzalil, H. B., & Rami Bahsoon, A. (2024) · Software Practice and Experience · doi:10.1002/spe.3371 · Cited by 2 · see citing works
Towards an Anatomy of Software Craftsmanship
Sundelin, A., González‐Huerta, J., Wnuk, K., & Gorschek, T. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3468504 · Cited by 2 · see citing works
Evolution of internal dimensions in object‐oriented software–A time series based approach
de Sousa, B. L., Bigonha, M. A. S., Ferreira, K. A. M., & Franco, G. C. (2024) · Software Practice and Experience · doi:10.1002/spe.3310 · Cited by 2 · see citing works
Grammar-Based Action Selection Rules for Scriptless Testing
Hufkens, L. V., Ricós, F. P., Marín, B., & Vos, T. E. J. (2024) · doi:10.1145/3644032.3644446 · Cited by 2 · see citing works
Chowdhury, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715006 · Cited by 2 · see citing works
Behavior Driven Development for 3D games
Ricós, F. P., Marín, B., Prasetya, I. S. W. B., Vos, T. E. J., Davidson, J., & Hovorka, K. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2025.102486 · Cited by 2 · see citing works
Mohd, F., Ismail, S., Jalil, M. A., Zulkarnain, F. I. N. M., & Zamin, N. B. (2021) · International Conference on Information Retrieval & Knowledge Management (CAMP) · doi:10.1109/camp51653.2021.9497978 · Cited by 2 · see citing works
LLM-Empowered Scriptless Functional Testing
Hooren, C. V., Ricós, F. P., Bromuri, S., Vos, T. E. J., & Marín, B. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00012 · Cited by 1 · see citing works
Kosar, T., Kovačević, Ž., Mernik, M., & Slivnik, B. (2023) · Mathematics · doi:10.3390/math11173744 · Cited by 1 · see citing works
Navigating Remote Work: Challenges and Adaptations of Agile Teams Amidst Covid-19
Canedo, E. D., Calazans, A. T. S., Silva, G. R. S., Brito, I. S., & Masson, E. T. S. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-64748-2_4 · Cited by 1 · see citing works
Teaching Theorizing in Software Engineering Research
Stol, K.-J. (2024) · Springer eBooks · doi:10.1007/978-3-031-71769-7_3 · Cited by 2 · see citing works
On Comprehension of Genetic Programming Solutions: A Controlled Experiment on Semantic Inference
Slivnik, B., Kovačević, Ž., Mernik, M., & Kosar, T. (2022) · Mathematics · doi:10.3390/math10183386 · Cited by 1 · see citing works
Leveraging Intermediate Artifacts to Improve Automated Trace Link Retrieval
Rodriguez, A. D. (2021) · doi:10.15368/theses.2021.114 · Cited by 1 · see citing works
Understanding Fairness in Software Engineering: Insights from Stack Exchange Sites
Sesari, E., Sarro, F., & Rastogi, A. (2024) · UCL Discovery (University College London)
Fuzzing-based mutation testing of C/C++ software in cyber-physical systems
Lee, J., Pastore, F., & Briand, L. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10700-7 · Cited by 1 · see citing works
Scriptless Test Exploration of 3D Computer Games
Ricós, F. P., Marín, B., Vos, T. E. J., Davidson, J., & Hovorka, K. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19050-5_21
Automated and configurable undo for CRDT-based distributed systems via metaprogramming
Mondal, P., & Tilevich, E. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112986
Can We Classify Flaky Tests Using Only Test Code? an LLM-Based Empirical Study
Berndt, A., Bekmyradov, V., Gemulla, R., Kessel, M., Bach, T., & Baltes, S. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00016
Udrescu, E. S., Udrescu, A., Suduc, A.-M., & Bîzoi, M. (2026) · Research Square · doi:10.21203/rs.3.rs-10720104/v1
Views on Internal and External Validity in Empirical Software Engineering: 10 Years Later and Beyond
Mailach, A., Siegmund, J., Apel, S., & Siegmund, N. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764520
Classifying Industrial Bugs to Advance Scriptless Test Oracles
Bouwmeester, R. R., Hufkens, L. V., Ricós, F. P., Marín, B., & Vos, T. E. J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21373-0_20
Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review
Rahman, M. S., Alam, K., Roy, B., & Roy, C. K. (2026) · arXiv · doi:10.48550/arxiv.2610.21144
- Authors
- Ralph, P., & Tempero, E.
- Affiliations
- University of Auckland (NZ)
- Venue
- Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018
- Publisher
- ACM
- Published
- 2018-06-28
- Pages
- 13-23
- OpenAlex
- W2810607235
- Cited by
- 78
- References
- 36
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 22 · 2023: 10 · 2022: 17 · 2021: 10 · 2020: 1 · 2019: 1 · 2018: 1
- FWCI
- 2.55
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- construct validity, software metrics, measurement validity, scale validation, program size, coupling, cohesion, Java classes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
