What Is the State of theArt of Systematic Reviewin Engineering Education?: Review of Systematic Reviews in Engineering Education
Abstract
Background Systematic review refers to a growing set of research methodologies intended to critically appraise and synthesize research to inform policy and practice. Systematic review methods, widely used in fields such as medicine, psychology, and education, are being applied in engineering education. Analysis of current systematic review practices in engineering education can contribute to continuous improvement of an important set of synthesis methodologies. Purpose To promote continuous improvement in systematic reviews, researchers are currently developing criteria to evaluate the quality of systematic review methodologies. The purpose of this article is to identify prior systematic reviews in areas of interest to the engineering education community, use these criteria to critique these systematic reviews, and make recommendations for improving use and quality of systematic review methods in engineering education research. Scope/Method Using systematic review search methodologies, we identified 12 conference papers and 37 journal articles that are systematic reviews published since 1990. Two reviewers coded each by topic, education level, quantitative or qualitative sources and synthesis methods, and 18 quantitative measures of review quality. Interrater reliability was calculated using kappa as described by Landis and Koch, which ranged from 0.43 (moderate) to 0.80 (almost perfect) with overall agreement of 0.64 (substantial). Conclusions We found that the 49 identified…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The review used Borrego, Foster, and Froyd's (2015) four complementary methods: search, selection, coding, and synthesis.
“To conduct the systematic literature review, we used Borrego, Foster, and Froyd's (2015) four complementary methods: search, selection, coding, and synthesis.”
From A Systematic Review of Studies on Educational Robotics (2019) · cites this paper as Borrego, Foster, and Froyd's (2015)
Like all systematic reviews, this study is limited by publication bias.
“Third, like all systematic reviews, this study is limited by publication bias (Borrego, Foster, & Froyd, 2015).”
From A Systematic Review of Studies on Educational Robotics (2019) · cites this paper as Borrego, Foster, & Froyd, 2015
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
Sloane-Seale, A. (2009) · Canadian Journal of University Continuing Education · doi:10.21225/d54s3d · Cited by 14,328 · see citing works
Practical Research: Planning and Design
Brent, E. E., & Leedy, P. D. (1990) · Teaching Sociology · doi:10.2307/1318509 · Cited by 9,364 · see citing works
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312 · see citing works
Does Active Learning Work? A Review of the Research
Prince, M. (2004) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2004.tb00809.x · Cited by 7,334 · cited 1× · see citing works
“Although other types of reviews are also highly cited, such as Prince (2004), few narrative reviews have had significant impact on engineering education, and Prince relied on systematic reviews to provide the majority of evidence supporting active learning (Johnson & Johnson, 1989; Johnson et al.,…”
BackgroundEducational Research: An Introduction
Dale, R. R., & Borg, W. R. (1965) · British Journal of Educational Studies · doi:10.2307/3119062 · Cited by 7,218 · see citing works
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · see citing works
Systematic Reviews in the Social Sciences: A Practical Guide
Petticrew, M., & Roberts, H. (2006) · doi:10.1002/9780470754887 · Cited by 3,586 · see citing works
Cooperation and Competition: Theory and Research
Johnson, D. W., & Johnson, R. T. (1989) · Medical Entomology and Zoology · Cited by 3,028 · see citing works
A systematic literature review of empirical evidence on computer games and serious games
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012) · Computers & Education · doi:10.1016/j.compedu.2012.03.004 · Cited by 2,890 · see citing works
Noblit, G., & Hare, R. (1988) · doi:10.4135/9781412985000 · Cited by 2,723 · see citing works
Methods for the synthesis of qualitative research: a critical review
Barnett-Page, E., & Thomas, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-59 · Cited by 2,087 · see citing works
The malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C. M., & Newcombe, N. S. (2012) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,986 · see citing works
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · see citing works
Online formative assessment in higher education: A review of the literature
Gikandi, J., Morrow, D., & Davis, N. E. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.06.004 · Cited by 1,383 · see citing works
Introduction to Systematic Reviews
(2013) · Active Learning Exercises · doi:10.21019/ale.2000.252 · Cited by 1,239 · see citing works
Exploring the educational potential of robotics in schools: A systematic review
Benitti, F. B. V. (2012) · Computers & Education · doi:10.1016/j.compedu.2011.10.006 · Cited by 1,200 · see citing works
Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks
Falchikov, N., & Goldfinch, J. (2000) · Review of Educational Research · doi:10.3102/00346543070003287 · Cited by 1,170 · see citing works
Tondeur, J., van Braak, J., Sang, G., Voogt, J. M., Fisser, P., & Ottenbreit-Leftwich, A. T. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.10.009 · Cited by 1,067 · see citing works
Henderson, C., Beach, A., & Finkelstein, N. (2011) · Journal of Research in Science Teaching · doi:10.1002/tea.20439 · Cited by 1,055 · cited 1× · see citing works
“Henderson, Beach, and Finkelstein (2011) created a framework for categorizing studies that described undergraduate STEM change strategies.”
BackgroundTamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011) · Review of Educational Research · doi:10.3102/0034654310393361 · Cited by 1,029 · see citing works
Cooperative Learning Returns To College What Evidence Is There That It Works?
Johnson, D. W., Johnson, R. T., & Smith, K. A. (1998) · Change: The Magazine of Higher Learning · doi:10.1080/00091389809602629 · Cited by 1,011 · see citing works
Hands-on, simulated, and remote laboratories
Ma, J., & Nickerson, J. V. (2006) · ACM Computing Surveys · doi:10.1145/1132960.1132961 · Cited by 986 · see citing works
Entrepreneurship education: a review of its objectives, teaching methods, and impact indicators
Mwasalwiba, E. S. (2010) · Education + Training · doi:10.1108/00400911011017663 · Cited by 838 · see citing works
Knobloch, K., Yoon, U., & Vogt, P. M. (2011) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2010.11.001 · Cited by 802 · see citing works
The Informed Design Teaching and Learning Matrix
Crismond, D., & Adams, R. S. (2012) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2012.tb01127.x · Cited by 667 · see citing works
Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields
Borrego, M. J., Foster, M., & Froyd, J. E. (2014) · Journal of Engineering Education · doi:10.1002/jee.20038 · Cited by 641 · see citing works
Self-citationŠumak, B., Heričko, M., & Pušnik, M. (2011) · Computers in Human Behavior · doi:10.1016/j.chb.2011.08.005 · Cited by 602 · see citing works
Metasynthesis: The State of the Art—So Far
Finfgeld, D. L. (2003) · Qualitative Health Research · doi:10.1177/1049732303253462 · Cited by 581 · see citing works
A Meta-analysis of the Effects of Computer Technology on School Students’ Mathematics Learning
Li, Q., & Ma, X. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9125-8 · Cited by 558 · see citing works
Southern, D. A., Norris, C. M., Quan, H., Shrive, F. M., Galbraith, P. D., Humphries, K. H., Gao, M., Knudtson, M. L., Ghali, W. A., & the APPROACH Investigators (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-1 · Cited by 503 · see citing works
Quality in Interpretive Engineering Education Research: Reflections on an Example Study
Walther, J., Sochacka, N. W., & Kellam, N. N. (2013) · Journal of Engineering Education · doi:10.1002/jee.20029 · Cited by 470 · see citing works
Krogsbøll, L. T., Hróbjartsson, A., & Gøtzsche, P. C. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-1 · Cited by 435 · see citing works
Salleh, N., Mendes, E., & Grundy, J. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.59 · Cited by 361 · see citing works
A systematic review of systematic reviews of homeopathy
Ernst, E. (2002) · British Journal of Clinical Pharmacology · doi:10.1046/j.1365-2125.2002.01699.x · Cited by 324 · see citing works
An analysis of research in computing disciplines
Glass, R. L., Ramesh, V., & Vessey, I. (2004) · Communications of the ACM · doi:10.1145/990680.990686 · Cited by 284 · see citing works
Devolder, A., van Braak, J., & Tondeur, J. (2012) · Journal of Computer Assisted Learning · doi:10.1111/j.1365-2729.2011.00476.x · Cited by 258 · see citing works
Adaptive and Intelligent Systems for Collaborative Learning Support: A Review of the Field
Magnisalis, I., Demetriadis, S., & Karakostas, A. (2011) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2011.2 · Cited by 231 · see citing works
Bowen, C. W. (2000) · Journal of Chemical Education · doi:10.1021/ed077p116 · Cited by 230 · see citing works
Fensham, P. J. (2004) · Contemporary trends and issues in science education · doi:10.1007/978-94-010-0175-5 · Cited by 220 · see citing works
Ethics Instruction in Engineering Education: A (Mini) Meta‐Analysis
Haws, D. R. (2001) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2001.tb00596.x · Cited by 214 · see citing works
Becker, K. H., & Park, K. (2011) · Digital Commons - USU (Utah State University) · Cited by 209 · see citing works
The art and science of knowledge synthesis
Tricco, A. C., Tetzlaff, J., & Moher, D. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.007 · Cited by 200 · see citing works
Teaching science problem solving: An overview of experimental work
Taconis, R., Ferguson‐Hessler, M. G. M., & Broekkamp, H. (2001) · Journal of Research in Science Teaching · doi:10.1002/tea.1013 · Cited by 193 · see citing works
Borrego, M. J. (2007) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2007.tb00911.x · Cited by 189 · see citing works
Self-citationConducting systematic reviews in medical education: a stepwise approach
Cook, D. A., & West, C. P. (2012) · Medical Education · doi:10.1111/j.1365-2923.2012.04328.x · Cited by 186 · see citing works
A review on systematic reviews of health information system studies
Lau, F., Kuziemsky, C. E., Price, M., & Gardner, J. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004838 · Cited by 182 · see citing works
State of Qualitative Research in Engineering Education: Meta‐Analysis of JEE Articles, 2005–2006
Koro‐Ljungberg, M., & Douglas, E. P. (2008) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2008.tb00965.x · Cited by 151 · see citing works
A Model for Freshman engineering r etention
Veenstra, C. P., & eriC, L. (2009) · AEE Journal · Cited by 149 · see citing works
Feminist Theory in Three Engineering Education Journals: 1995–2008
Beddoes, K., & Borrego, M. J. (2011) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2011.tb00014.x · Cited by 133 · see citing works
Self-citationSchmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Wade, C. A., Surkes, M. A., & Lowerison, G. (2009) · Journal of Computing in Higher Education · doi:10.1007/s12528-009-9021-8 · Cited by 120 · see citing works
A Review of Literature on Effectiveness of Use of Information Technology in Education
Kadiyala, M., & Crynes, B. L. (2000) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2000.tb00512.x · Cited by 118 · see citing works
Beyond Dissemination in College Science Teaching: An Introduction to Four Core Change Strategies
Henderson, C., Finkelstein, N. D., & Beach, A. L. (2010) · The journal of college science teaching · Cited by 118 · see citing works
Rosen, Y., & Salomon, G. (2007) · Journal of Educational Computing Research · doi:10.2190/r8m4-7762-282u-554j · Cited by 114 · see citing works
Beddoes, K. D., Jesiek, B. K., & Borrego, M. (2010) · Interdisciplinary Journal of Problem-Based Learning · doi:10.7771/1541-5015.1142 · Cited by 91 · see citing works
Self-citationCharacterizing research in computing education
Malmi, L., Sheard, J., Simon, Bednarik, R., Helminen, J., Korhonen, A., Myller, N., Sorva, J., & Taherkhani, A. (2010) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1839594.1839597 · Cited by 89 · see citing works
Cordray, D. S., Harris, T. R., & Klein, S. (2009) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2009.tb01031.x · Cited by 89 · see citing works
Towards characterising design-based learning in engineering education: a review of the literature
Puente, S. M. G., van Eijck, M., & Jochems, W. M. G. (2011) · European Journal of Engineering Education · doi:10.1080/03043797.2011.565116 · Cited by 87 · see citing works
A Systematic Survey of Games Used for Software Engineering Education
Caulfield, C., Xia, J. C., Veal, D., & Stanislaw Paul, M. A. J. (2011) · Modern Applied Science · doi:10.5539/mas.v5n6p28 · Cited by 83 · see citing works
Analysis of the First Ten Years of the Journal of Engineering Education
Wankat, P. C. (2004) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2004.tb00784.x · Cited by 79 · see citing works
What Constitutes Good Design? A Review of Empirical Studies of Design Processes*
Mehalik, M. M., & Schunn, C. D. (2006) · Cited by 72 · see citing works
Mapping global trends in engineering education research
Jesick, B. K., Borrego, M. J., Beddoes, K., Hurtado, M., & Rajendran, P. (2011) · International journal of engineering education · Cited by 61 · see citing works
Self-citationPreparing Students and Engineers for Global Software Development: A Systematic Review
Monasor, M. J., Vizcaíno, A., Piattini, M. G., & Caballero, I. (2010) · IEEE International Conference on Global Software Engineering (ICGSE) · doi:10.1109/icgse.2010.28 · Cited by 60 · see citing works
Anatomy of Scholarly Collaboration in Engineering Education: A Big‐Data Bibliometric Analysis
Xian, H., & Madhavan, K. P. C. (2014) · Journal of Engineering Education · doi:10.1002/jee.20052 · Cited by 54 · see citing works
STEM Education Related Dissertation Abstracts: A Bounded Qualitative Meta-study
Banning, J. H., & Folkestad, J. (2012) · Journal of Science Education and Technology · doi:10.1007/s10956-011-9361-9 · Cited by 43 · see citing works
An Evaluation of HigherEd 2.0 Technologies in Undergraduate Mechanical Engineering Courses.
Orange, A., Heinecke, W. F., Berger, E., Krousgrill, C. M., & Mikic, B. (2012) · AEE Journal · Cited by 43 · see citing works
Maeda, Y., & Yoon, S. Y. (2012) · Educational Psychology Review · doi:10.1007/s10648-012-9215-x · Cited by 39 · see citing works
A Systematic Review of Developing Team Competencies in Information Systems Education
Figl, K. (2010) · Journal of the Association for Information Systems · Cited by 38 · see citing works
WHITIN, K., & Sheppard, S. (2004) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2004.tb00783.x · Cited by 38 · see citing works
Lou, Y. (2004) · Journal of Educational Computing Research · doi:10.2190/xf5a-t29g-x170-7950 · Cited by 34 · see citing works
An Analysis of the Articles in the Journal of Engineering Education
Wankat, P. C. (1999) · Journal of Engineering Education · doi:10.1002/j.2168-9830.1999.tb00409.x · Cited by 33 · see citing works
Designing the Best: A Review of Effective Teaching and Learning of Design and Technology
Wilson, V., & Harris, M. (2003) · International Journal of Technology and Design Education · doi:10.1023/a:1026133808945 · Cited by 32 · see citing works
An Overview of the Literature: Research in P-12 Engineering Education
Díaz, N. V. M., & Cox, M. (2012) · AEE Journal · Cited by 30 · see citing works
Where's the proof? A review of literature on effectiveness of information technology in education
Kadiyala, M., & Crynes, B. L. (2002) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.1998.736797 · Cited by 29 · see citing works
da Silva, F. Q. B., Santos, A. L. M., Soares, S., França, A. C. C., & Monteiro, C. V. F. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852830 · Cited by 28 · see citing works
Why university lecturers enhance their teaching through the use of technology: a systematic review
Jump, L. (2011) · Learning Media and Technology · doi:10.1080/17439884.2010.521509 · Cited by 28 · see citing works
Knowledge sharing for sustainable development in civil engineering: a systematic review
Meese, N., & McMahon, C. A. (2011) · AI & Society · doi:10.1007/s00146-011-0369-8 · Cited by 27 · see citing works
Creating Theoretical Insights in Engineering Education
Johri, A. (2010) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2010.tb01055.x · Cited by 23 · see citing works
Systematic literature review: teaching novices programming using robots
Major, L., Kyriacou, T., & Brereton, O. P. (2011) · doi:10.1049/ic.2011.0003 · Cited by 22 · see citing works
Papadopoulos, C., & REISEL, J. R. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--3956 · Cited by 21 · see citing works
Dahm, K. D., Riddell, W. T., Constans, E. W., Courtney, J., Harvey, R., & von Lockette, P. (2009) · Cited by 19 · see citing works
Henderson, C., Beach, A. L., Finkelstein, N. D., Larson, R. S., Henderson, C., Sabella, M. S., & Hsu, L. (2008) · AIP conference proceedings · doi:10.1063/1.3021235 · Cited by 17 · see citing works
(2012) · Cited by 15 · see citing works
Conceptualizing Authenticity in Engineering Education: A Systematic Literature Review
Wang, J., Dyehouse, M. A., Weber, N., & Ströbel, J. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--21098 · Cited by 14 · see citing works
Power, J. F., Whelan, T. J., & Bergin, S. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953247 · Cited by 13 · see citing works
Crockett, E. R., & Ohland, M. (2020) · IEEE Lasers and Electro-Optics Society Annual Meeting (LEOS) · doi:10.18260/1-2--10104 · Cited by 12 · see citing works
Mobile Learning Content Authoring Tools (MLCATs): A Systematic Review
Mugwanya, R., & Marsden, G. (2010) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-642-12701-4_3 · Cited by 11 · see citing works
Meta-analysis of effectiveness of technology use in undergraduate engineering education
Michko, G. (2008) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2008.4720455 · Cited by 9 · see citing works
Collins, R., & Ohland, M. (2020) · IEEE Lasers and Electro-Optics Society Annual Meeting (LEOS) · doi:10.18260/1-2--10110 · Cited by 8 · see citing works
Steel, C. H., & Gunn, C. (2011) · ASCILITE - Australian Society for Computers in Learning in Tertiary Education Annual Conference · Cited by 3 · see citing works
Effective Faculty Mentoring For Diversity: An Assessment Of Mentoring Paradigms
Surovek, A., Karlin, J., & Groen, C. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--16133 · Cited by 2 · see citing works
A Meta Analysis Of Studies Of Cognition In Engineering Education
Brumback, T., Schumacker, R., & Fonseca, D. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--16052 · Cited by 1 · see citing works
A portrait of evaluation studies of learning technology innovations 2005-2010
Steel, C. H., & Gunn, C. (2011) · ASCILITE Publications · doi:10.14742/apubs.2011.1802 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Research and trends in STEM education: a systematic review of journal publications
Li, Y., Wang, K., Xiao, Y., & Froyd, J. E. (2020) · International Journal of STEM Education · doi:10.1186/s40594-020-00207-6 · Cited by 580 · see citing works
Self-citationA Systematic Review of Studies on Educational Robotics
Anwar, S., Bascou, N. A., Menekse, M., & Kardgar, A. (2019) · Journal of Pre-College Engineering Education Research (J-PEER) · doi:10.7771/2157-9288.1223 · Cited by 419 · see citing works
A Review of Competency‐Based Learning: Tools, Assessments, and Recommendations
Henri, M., Johnson, M., & Nepal, B. P. (2017) · Journal of Engineering Education · doi:10.1002/jee.20180 · Cited by 242 · see citing works
Engineers at Play: Games as Teaching Tools for Undergraduate Engineering Students
Bodnar, C. A., Anastasio, D. D., Enszer, J., & Burkey, D. D. (2015) · Journal of Engineering Education · doi:10.1002/jee.20106 · Cited by 236 · see citing works
Novice Programmers and Introductory Programming
Robins, A. V. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.013 · Cited by 183 · see citing works
Ong, M., Jaumot‐Pascual, N., & Ko, L. T. (2020) · Journal of Engineering Education · doi:10.1002/jee.20345 · Cited by 164 · see citing works
Denton, M., Borrego, M. J., & Boklage, A. (2020) · Journal of Engineering Education · doi:10.1002/jee.20322 · Cited by 142 · see citing works
Self-citationSafapour, E., Kermanshachi, S., & Taneja, P. (2019) · Education Sciences · doi:10.3390/educsci9040273 · Cited by 139 · see citing works
The Cambridge Handbook of Computing Education Research
(2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555 · Cited by 130 · see citing works
A Literature Review of Critical Thinking in Engineering Education
Ahern, A., Dominguez, C., McNally, C., O’Sullivan, J. J., & Pedrosa, D. (2019) · Studies in Higher Education · doi:10.1080/03075079.2019.1586325 · Cited by 127 · see citing works
Community engagement in engineering education: A systematic literature review
Natarajarathinam, M., Qiu, S., & Lu, W. (2021) · Journal of Engineering Education · doi:10.1002/jee.20424 · Cited by 115 · see citing works
Characteristics of systematic reviews in the social sciences
Chapman, K. (2021) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2021.102396 · Cited by 93 · see citing works
Curzon, P., Bell, T., Waite, J., & Dorling, M. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.018 · Cited by 80 · see citing works
Funds of Knowledge in STEM Education: A Scoping Review
Denton, M., & Borrego, M. (2021) · Studies in Engineering Education · doi:10.21061/see.19 · Cited by 72 · see citing works
Self-citationEngineering Thinking in Prekindergarten Children: A Systematic Literature Review
Lippard, C. N., Lamm, M. H., & Riley, K. L. (2017) · Journal of Engineering Education · doi:10.1002/jee.20174 · Cited by 70 · see citing works
Developing growth mindsets in engineering students: a systematic literature review of interventions
Campbell, A., Direito, I., & Mokhithi, M. (2021) · European Journal of Engineering Education · doi:10.1080/03043797.2021.1903835 · Cited by 68 · see citing works
Systems thinking assessments in engineering: A systematic literature review
Dugan, K. E., Mosyjowski, E. A., Daly, S. R., & Lattuca, L. R. (2021) · Systems Research and Behavioral Science · doi:10.1002/sres.2808 · Cited by 66 · see citing works
Mental health and wellbeing of undergraduate students in engineering: A systematic literature review
Asghar, M., Minichiello, A., & Ahmed, S. (2023) · Journal of Engineering Education · doi:10.1002/jee.20574 · Cited by 66 · see citing works
Programming Paradigms and Beyond
Krishnamurthi, S., & Fisler, K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.014 · Cited by 65 · see citing works
Horn, M., & Bers, M. U. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.023 · Cited by 62 · see citing works
Boundary Spanning and Engineering: A Qualitative Systematic Review
Jesiek, B. K., Mazzurco, A., Buswell, N. T., & Thompson, J. D. (2018) · Journal of Engineering Education · doi:10.1002/jee.20219 · Cited by 57 · see citing works
The History of Computing Education Research
Guzdial, M., & du Boulay, B. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.002 · Cited by 57 · see citing works
A systematic review of observation protocols used in postsecondary STEM classrooms
Anwar, S., & Menekşe, M. (2020) · Review of Education · doi:10.1002/rev3.3235 · Cited by 46 · see citing works
A scoping review of engineering education systematic reviews
Phillips, M., Reed, J. B., Zwicky, D., & Epps, A. V. (2023) · Journal of Engineering Education · doi:10.1002/jee.20549 · Cited by 46 · see citing works
Industry 5 and the Human in Human-Centric Manufacturing
Briken, K., Moore, J., Scholarios, D., Rose, E., & Sherlock, A. (2023) · Sensors · doi:10.3390/s23146416 · Cited by 45 · see citing works
Gender in Engineering Education Research: A Content Analysis of Research in JEE , 1998-2012
Pawley, A. L., Schimpf, C., & Nelson, L. A. (2016) · Journal of Engineering Education · doi:10.1002/jee.20128 · Cited by 45 · see citing works
Sundman, J., Feng, X., Shrestha, A., Johri, A., Varis, O., & Taka, M. (2025) · European Journal of Engineering Education · doi:10.1080/03043797.2025.2532591 · Cited by 44 · see citing works
Jackson, A., Barrella, E. M., & Bodnar, C. A. (2023) · Journal of Engineering Education · doi:10.1002/jee.20548 · Cited by 43 · see citing works
Systematic Review of the Funds of Knowledge Framework in STEM Education
Verdín, D., Godwin, A., & Capobianco, B. (2016) · ASEE Annual Conference & Exposition · doi:10.18260/p.25999 · Cited by 40 · see citing works
Cognitive Sciences for Computing Education
Robins, A. V., Margulieux, L. E., & Morrison, B. B. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.010 · Cited by 39 · see citing works
Educating the Engineer Entrepreneur of the Future: A Team Competency Perspective
Neumeyer, X., & Santos, S. C. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3086778 · Cited by 38 · see citing works
Empirical research studies of practicing engineers: a mapping review of journal articles 2000–2018
Mazzurco, A., Crossin, E., Chandrasekaran, S., Daniel, S., & Sadewo, G. R. P. (2020) · European Journal of Engineering Education · doi:10.1080/03043797.2020.1818693 · Cited by 37 · see citing works
Computing EducationLiterature Review and Voices from the Field
Blikstein, P., & Moghadam, S. H. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.004 · Cited by 36 · see citing works
Supporting inclusivity in STEM makerspaces through critical theory: A systematic review
Andrews, M., & Boklage, A. (2023) · Journal of Engineering Education · doi:10.1002/jee.20546 · Cited by 36 · see citing works
Sigahi, T. F. A. C., & Sznelwar, L. I. (2021) · Journal of Engineering Education · doi:10.1002/jee.20438 · Cited by 35 · see citing works
A multidisciplinary engineering capstone design course: A case study for design-based approach
Qattawi, A., Alafaghani, A., Ablat, M. A., & Jaman, M. S. (2019) · International Journal of Mechanical Engineering Education · doi:10.1177/0306419019882622 · Cited by 34 · see citing works
Lewis, C. M., Shah, N., & Falkner, K. E. K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.017 · Cited by 33 · see citing works
Collaborative Learning in Engineering Design Education: A Systematic Literature Review
van Helden, G., Zandbergen, B. T. C., Specht, M. M., & Gill, E. K. A. (2023) · IEEE Transactions on Education · doi:10.1109/te.2023.3283609 · Cited by 32 · see citing works
Learning Sciences for Computing Education
Margulieux, L. E., Dorn, B., & Searle, K. A. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.009 · Cited by 32 · see citing works
Teacher Learning and Professional Development
Fincher, S. A., Kolikant, Y. B., & Falkner, K. E. K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.026 · Cited by 28 · see citing works
Systematic Reviews in the Engineering Literature: A Scoping Review
Phillips, M., Reed, J. B., Zwicky, D., Epps, A. V., Buhler, A., Rowley, E. M., Zhang, Q., Cox, J. M., & Zakharov, W. (2024) · IEEE Access · doi:10.1109/access.2024.3394755 · Cited by 27 · see citing works
Motivation, Attitudes, and Dispositions
Lishinski, A., & Yadav, A. K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.029 · Cited by 27 · see citing works
A Narrative Review of Design-Based Research in Engineering Education: Opportunities and Challenges
Minichiello, A., & Caldwell, L. (2021) · Studies in Engineering Education · doi:10.21061/see.15 · Cited by 27 · see citing works
Falkner, K. E. K., & Sheard, J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.016 · Cited by 27 · see citing works
Haden, P. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.006 · Cited by 26 · see citing works
Kimpton, C. D., & Maynard, N. E. (2024) · European Journal of Engineering Education · doi:10.1080/03043797.2024.2357343 · Cited by 24 · see citing works
Qualitative Methods for Computing Education
Tenenberg, J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.008 · Cited by 24 · see citing works
Teaching occupational health and safety in engineering using active learning: A systematic review
Neto, I. R., & Amaral, F. G. (2023) · Safety Science · doi:10.1016/j.ssci.2023.106391 · Cited by 23 · see citing works
Emerging Learning Environments and Technologies Post Covid-19 Pandemic: What’s Next?
Mahmud, M. M., Wong, S. F., & Ismail, O. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-030-91738-8_29 · Cited by 21 · see citing works
Magarian, J. N., & Seering, W. P. (2021) · Journal of Engineering Education · doi:10.1002/jee.20382 · Cited by 20 · see citing works
Cost estimation model of prefabricated construction for general contractors based on system dynamics
Liu, M., & Luo, M. (2023) · Engineering Construction & Architectural Management · doi:10.1108/ecam-03-2023-0304 · Cited by 19 · see citing works
Malmi, L., Utting, I., & Ko, A. J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.022 · Cited by 17 · see citing works
Computing for Other Disciplines
Guzdial, M. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.020 · Cited by 17 · see citing works
Learning Outside the Classroom
Begel, A., & Ko, A. J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.027 · Cited by 16 · see citing works
A systematic approach to develop risk registry frameworks for complex projects
Khallaf, R., Naderpajouh, N., & Hastak, M. (2018) · Built Environment Project and Asset Management · doi:10.1108/bepam-08-2017-0051 · Cited by 15 · see citing works
Engineering Education Research
Loui, M. C., & Borrego, M. J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.012 · Cited by 14 · see citing works
Self-citationEducational robotics studies in Italian scientific journals: A systematic review
Bonaiuti, G., Campitiello, L., Tore, S. D., & Marras, A. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.1005669 · Cited by 13 · see citing works
Reporting characteristics of systematic reviews in Psychology: A scoping review
Steil, A. V., Dias, N. M., Lopes, F. M., da Silva, M. L. B., Bousfield, A. B. D. S., & Canto, G. D. L. (2022) · Journal of Health Psychology · doi:10.1177/13591053221074592 · Cited by 13 · see citing works
Leveraging the Integrated Development Environment for Learning Analytics
Carter, A., Hundhausen, C. D., & Olivares, D. M. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.024 · Cited by 13 · see citing works
College-Level Sex Education Courses: A Systematic Literature Review
Manning‐Ouellette, A., & Shikongo, J. (2022) · American Journal of Sexuality Education · doi:10.1080/15546128.2022.2026850 · Cited by 12 · see citing works
Teacher Knowledge for Inclusive Computing Learning
Goode, J., & Ryoo, J. J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.025 · Cited by 12 · see citing works
A Systematic Map for Improving Teaching and Learning in Undergraduate Operating Systems Courses
Pamplona, S., Medinilla, N., & Flores, P. (2018) · IEEE Access · doi:10.1109/access.2018.2871768 · Cited by 10 · see citing works
Design for Six Sigma: A Review of the Definitions, Objectives, Activities, and Tools
Dzulinski, A. C., Braghini, A., & Chiroli, D. M. D. G. (2022) · Engineering Management Journal · doi:10.1080/10429247.2022.2041964 · Cited by 10 · see citing works
Lancaster, T., Robins, A. V., & Fincher, S. A. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.015 · Cited by 10 · see citing works
Vahrenhold, J., Cutts, Q. I., & Falkner, K. E. K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.019 · Cited by 9 · see citing works
A Case Study of Qualitative Methods
Lewis, C. M. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.032 · Cited by 9 · see citing works
A comprheensive review of Visual Programming Tools for Arduino
Melo, J. C. P., Fidelis, M., Alves, S., Freitas, U., & Dantas, R. R. (2020) · Latin American Robotics Symposium, Brazilian Symposium on Robotics and Workshop on Robotics in Education (LARS/SBR/WRE) · doi:10.1109/lars/sbr/wre51543.2020.9307023 · Cited by 8 · see citing works
Stephensen, R., & Cunningham, S. (2023) · Australasian journal of engineering education · doi:10.1080/22054952.2023.2282848 · Cited by 8 · see citing works
Hybrid Cloud Service Selection Strategies: A Qualitative Meta-Analysis
Azumah, K. K., Sørensen, L. T., & Tadayoni, R. (2018) · IEEE International Conference on Adaptive Science & Technology · doi:10.1109/icastech.2018.8506887 · Cited by 8 · see citing works
Shapiro, R. B., & Tissenbaum, M. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.021 · Cited by 8 · see citing works
Haden, P. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.007 · Cited by 8 · see citing works
Computing Education Research Today
Fincher, S. A., Tenenberg, J., Dorn, B., Hundhausen, C., McCartney, R., & Murphy, L. (2019) · The Cambridge Handbook of Computing Education Research · doi:10.1017/9781108654555.003 · Cited by 7 · see citing works
Ntunka, M. G., Maladzhi, R. W., Mthombeni, N. H., & Nemavhola, F. J. (2024) · World Engineering Education Forum - Global Engineering Deans Council (WEEF-GEDC) · doi:10.1109/weef-gedc63419.2024.10854923 · Cited by 7 · see citing works
Systematic Review of Rigorous Research in Teaching Introductory Circuits
Reagan, T., Claussen, S. A., & Lyne, E. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--35265 · Cited by 7 · see citing works
Sustainable systematic literature reviews
dos Santos, V., Iwazaki, A. Y., Felizardo, K. R., de Souza, É. F., & Nakagawa, E. Y. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107551 · Cited by 7 · see citing works
Student Knowledge and Misconceptions
Lewis, C. M., Clancy, M. J., & Vahrenhold, J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.028 · Cited by 7 · see citing works
Shephard, K. L. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.011 · Cited by 7 · see citing works
ChatGPT and Personalised Learning: Reshaping Pedagogical Approaches in the VUCA Age
Baskara, F. R. (2024) · Jurnal Educative Journal of Educational Studies · doi:10.30983/educative.v8i2.7620 · Cited by 6 · see citing works
Ko, A. J., & Fincher, S. A. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.005 · Cited by 6 · see citing works
Munn, K., Goh, S. C.-N., Basson, M., & Thorpe, D. (2021) · Australasian journal of engineering education · doi:10.1080/22054952.2021.1934262 · Cited by 6 · see citing works
Students As Teachers and Communicators
Simon, B., Hundhausen, C. D., McDowell, C., Werner, L., Hu, H. H., & Kussmaul, C. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.030 · Cited by 6 · see citing works
Fincher, S. A., & Robins, A. V. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.001 · Cited by 6 · see citing works
Understanding faculty decisions about the integration of laboratories into engineering education
Coutinho, G. S., Stites, N. A., & Magana, A. J. (2017) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2017.8190605 · Cited by 5 · see citing works
Benefits and Challenges of a Graduate Course: An Experience Teaching Systematic Literature Review
Iwazaki, A. Y., dos Santos, V., Felizardo, K. R., de Souza, É. F., Valentim, N., & Nakagawa, E. Y. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962446 · Cited by 5 · see citing works
Special session: Introduction to systematic reviews in engineering education research
Froyd, J. E., Foster, M., Martin, J. P., Borrego, M. J., Choe, H. S., & Chen, X. (2015) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2015.7344090 · Cited by 5 · see citing works
Self-citationA Case Study of Peer Instruction
Porter, L., & Simon, B. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.031 · Cited by 4 · see citing works
Perez, A., & Anwar, S. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893551 · Cited by 3 · see citing works
Dart, S., Young, A. E., & Steindl, E.-L. (2025) · Journal of Engineering Education · doi:10.1002/jee.70030 · Cited by 3 · see citing works
Haddadian, G., Kim, M. K., & Haddadian, N. (2025) · Research Synthesis in Applied Linguistics · doi:10.1080/29984475.2025.2598266 · Cited by 3 · see citing works
Zhang, L., wang, X., Wang, H., & Lin, C. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.106537 · Cited by 3 · see citing works
The Dimensions of Abundance in AI-Generated Feedback
Lindsay, E. D., Rodda, A. E., Lindqvist, A. L., Quince, Z., Lim, M., & Jiang, D. (2026) · Education Sciences · doi:10.3390/educsci16030465 · Cited by 3 · see citing works
The Graduate Student Role in Undergraduate Research Mentoring: A Systematic Literature Review
Asbill, H., Letchworth, M., Rynearson, A. M., & Pantoja, C. A. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--48101 · Cited by 3 · see citing works
What’s Going On – Mapping the Evolution of Engineering Education Research in Europe
Williams, B., & Neto, P. J. C. (2018) · International Conference of the Portuguese Society for Engineering Education (CISPEE) · doi:10.1109/cispee.2018.8593468 · Cited by 3 · see citing works
Mastery Learning in Undergraduate Engineering Courses: A Systematic Review
Perez, C., & Verdín, D. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--40522 · Cited by 3 · see citing works
Bhaduri, S. (2018) · VTechWorks (Virginia Tech) · Cited by 3 · see citing works
Methodological Commitments in Grounded Theory: Experiences of a Novice Researcher
Clement, E., Huff, J. L., & Brown, S. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342904 · Cited by 2 · see citing works
Wellbeing of Graduate Engineering Students: A Systematic Review
Kamal, S. A., Ali, S. F., & Sánchez‐Peña, M. L. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--48265 · Cited by 2 · see citing works
Phillips, M., Reed, J. B., & Zwicky, D. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892811 · Cited by 2 · see citing works
A Team Competency Framework for Engineering Entrepreneurship Education
Neumeyer, X., & Santos, S. C. (2020) · IEEE Technology & Engineering Management Conference (TEMSCON) · doi:10.1109/temscon47658.2020.9140099 · Cited by 2 · see citing works
The Principal’s Leadership Role to Minimize Bullying in Schools
Sholeh, M., Windasari, Putra, B. A. W., & Stiawan, A. (2023) · Advances in Social Science, Education and Humanities Research/Advances in social science, education and humanities research · doi:10.2991/978-2-38476-152-4_127 · Cited by 2 · see citing works
- Authors
- Borrego, M., Foster, M. J., & Froyd, J. E.
- Affiliations
- The University of Texas at Austin (US)
- Texas A&M University (US)
- Venue
- Journal of Engineering Education
- Publisher
- Wiley
- Published
- 2015-04
- Volume
- 104
- Issue
- 2
- Pages
- 212-242
- ISSN
- 1069-4730 (ISSN-L)
- OpenAlex
- W1720540152
- Cited by
- 124
- References
- 94
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 20 · 2023: 10 · 2022: 7 · 2021: 11 · 2020: 11 · 2019: 38 · 2018: 6 · 2017: 3 · 2016: 3 · 2015: 2
- FWCI
- 0.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Engineering Education (publishedVersion)
- Funders
- National Science Foundation (1446369, 1446319)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Engineering › Media Technology › Engineering Education and Curriculum Development
- Social Sciences › Education › Educational Research and Methods
- Keywords
- engineering education research, meta-analysis, synthesis methods, inter-rater reliability, study appraisal, evidence synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
