Comparability of Computer-Based and Paper-and-Pencil Testing in K–12 Reading Assessments: A Meta-Analysis of Testing Mode Effects
The DOI matches its Crossref record, with a small difference in title or year; 53 of the 55 references it cites were found in a registry or an index.
Abstract
In recent years, computer-based testing (CBT) has grown in popularity, is increasingly being implemented across the United States, and will likely become the primary mode for delivering tests in the future. Although CBT offers many advantages over traditional paper-and-pencil testing, assessment experts, researchers, practitioners, and users have expressed concern about the comparability of scores between the two test administration modes. To help provide an answer to this issue, a meta-analysis was conducted to synthesize the administration mode effects of CBTs and paper-and-pencil tests on K—12 student reading assessments. Findings indicate that the administration mode had no statistically significant effect on K—12 student reading achievement scores. Four moderator variables—study design, sample size, computer delivery algorithm, and computer practice—made statistically significant contributions to predicting effect size. Three moderator variables—grade level, type of test, and computer delivery method—did not affect the differences in reading scores between test modes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this difference between reading media was larger in studies that used fixed linear computerized tests ( n =37) than in those that used adaptive computerized tests ( n =5).”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Wang et al. (2007)
In the citing paper
“They found better reading outcomes in paper-based testing than in digital-based testing. The mean effect size (0.08) was significant, but small”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Wang et al. (2007)
In the citing paper
“The results of the two meta-analyses in the present study yield a clear picture of screen inferiority, with lower reading comprehension outcomes for digital texts compared to printed texts, which corroborates and extends previous research (Kong et al., 2018; Singer & Alexander, 2017b; Wang et al., 2007).”
From Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension (2018) · cites this paper as Wang et al., 2007
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
(2000) · Cited by 9,198
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · cited 1×
“The standardized mean difference or ES g for any individual study is defined (Hedges, 1981; Hedges & Olkin, 1985) as follows: where M CBT and M PPT are the means of CBT and PPT scores and SD p is the pooled estimate of the standard deviation, where n CBT and n PPT are the sample sizes of CBT and PPT…”
Method(1904) · The American Journal of Psychology · doi:10.2307/1412635 · Cited by 4,187
A nation at risk: the imperative for educational reform
Denning, P. J. (1983) · Communications of the ACM · doi:10.1145/358150.358154 · Cited by 3,925
Food, M.-D. (2011) · In Search of Research Excellence · doi:10.4337/9781849807630.00020 · Cited by 3,062
Statistical Methods for Meta-Analysis
(1985) · Elsevier eBooks · doi:10.1016/c2009-0-03396-0 · Cited by 2,521
Morris, S. B., & DeShon, R. P. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.105 · Cited by 2,502
A Nation at Risk: The Imperative for Educational Reform
(1983) · The Elementary School Journal · doi:10.1086/461348 · Cited by 2,341
Russell, W. F. (1930) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146813003100748 · Cited by 1,972
Meta-analysis of experiments with matched groups or repeated measures designs
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.170 · Cited by 1,693
Standards for educational and psychological testing: (500832009-001)
(1985) · PsycEXTRA Dataset · doi:10.1037/e500832009-001 · Cited by 1,480
Synthesizing standardized mean‐change measures
Becker, B. J. (1988) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1988.tb00901.x · Cited by 889
How hard is hard science, how soft is soft science? The empirical cumulativeness of research.
Hedges, L. V. (1987) · American Psychologist · doi:10.1037//0003-066x.42.5.443 · Cited by 435
Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis.
Mead, A. D., & Drasgow, F. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.114.3.449 · Cited by 432
The Effect of Computers on Student Writing: A Meta-analysis of Studies from 1992 to 2002
Goldberg, A., Russell, M., & Cook, A. (2003) · Open Access Journals at BC (Boston College) · Cited by 387
TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS
GREEN, B. F., BOCK, R. D., HUMPHREYS, L. G., LINN, R. L., & RECKASE, M. D. (1984) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1984.tb01039.x · Cited by 383
Practical Considerations in Computer-Based Testing
Parshall, C. G., Spray, J. A., Kalohn, J. C., & Davey, T. (2002) · Springer eBooks · doi:10.1007/978-1-4613-0083-0 · Cited by 272
Kolen, M. J., & Brennan, R. L. (1995) · Springer Series in Statistics · doi:10.1007/978-1-4757-2412-7 · Cited by 263
International Guidelines on Computer-Based and Internet-Delivered Testing
The International Test Commission (2006) · International Journal of Testing · doi:10.1207/s15327574ijt0602_4 · Cited by 197
Guidelines for computer-based tests and interpretations: (502052009-001)
(1986) · PsycEXTRA Dataset · doi:10.1037/e502052009-001 · Cited by 190
Inexorable and Inevitable: The Continuing Story of Technology and Assessment
Bennett, R. E. (2005) · Computer‐Based Testing and the Internet · doi:10.1002/9780470712993.ch11 · Cited by 180
The Equivalence of Paper-and-Pencil and Computer-Based Testing
Bugbee, A. C. (1996) · Journal of Research on Computing in Education · doi:10.1080/08886504.1996.10782166 · Cited by 169
Measuring Reading Comprehension with the Lexile Framework
Stenner, A. J. (2023) · Explanatory Models, Unit Standards, and Personalized Learning in Educational Measurement · doi:10.1007/978-981-19-3747-7_6 · Cited by 133
Test-taking dispositions: A missing link?
Schmit, M. J., & Ryan, A. M. (1992) · Journal of Applied Psychology · doi:10.1037//0021-9010.77.5.629 · Cited by 131
Russell, M., & Haney, W. (1997) · Digital Commons - University of South Florida (University of South Florida) · Cited by 127
A Meta-Analysis of Testing Mode Effects in Grade K-12 Mathematics Tests
Wang, S., Jiao, H., Young, M. J., Brooks, T., & Olson, J. (2007) · Educational and Psychological Measurement · doi:10.1177/0013164406288166 · Cited by 127
Self-citationTHE EQUIVALENCE OF SCORES FROM AUTOMATED AND CONVENTIONAL EDUCATIONAL AND PSYCHOLOGICAL TESTS
Mazzeo, J., & Harvey, A. L. (1988) · ETS Research Report Series · doi:10.1002/j.2330-8516.1988.tb00277.x · Cited by 111
How the Internet Will Help Large-Scale Assessment Reinvent Itself
Bennett, R. E. (2001) · Education Policy Analysis Archives · doi:10.14507/epaa.v9n5.2001 · Cited by 101
(2004) · Open Access Journals at BC (Boston College) · Cited by 95
Research on the Effects of Administering Tests via Computers
Wise, S. L., & Plake, B. S. (1989) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1989.tb00324.x · Cited by 87
Computerization of Paper-and-Pencil Tests: When are They Equivalent?
Neuman, G., & Baydoun, R. (1998) · Applied Psychological Measurement · doi:10.1177/01466216980221006 · Cited by 77
Mazzeo, J., & Harvey, A. L. (1988) · Cited by 76
Pomplun, M., Frey, S., & Becker, D. F. (2002) · Educational and Psychological Measurement · doi:10.1177/0013164402062002009 · Cited by 65
IMPLICATIONS OF CHANGING ANSWERS ON OBJECTIVE TEST ITEMS
Mueller, D. J., & WASSER, V. (1977) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1977.tb00023.x · Cited by 46
Effects of Computer Versus Paper Administration of a State-Mandated Writing Assessment
Russell, M., & Plati, T. (2001) · Teachers College Record The Voice of Scholarship in Education · Cited by 41
Chin, C., Donn, J. S., & Conry, R. F. (1991) · Educational and Psychological Measurement · doi:10.1177/0013164491513025 · Cited by 31
A Review of the Computerized Version of the Self‐Directed Search
McKee, L. M., & Levinson, E. M. (1990) · The Career Development Quarterly · doi:10.1002/j.2161-0045.1990.tb00222.x · Cited by 27
Computer Administration of the California Psychological Inventory
Scissons, E. H. (1976) · Measurement and Evaluation in Guidance · doi:10.1080/00256307.1976.12022707 · Cited by 25
How Review Options and Administration Modes Influence Scores on Computerized Vocabulary Tests.
Vispoel, W. P. (1992) · Cited by 21
Large-Scale Testing: Current Practices and New Directions
Klein, S. P., & Hamilton, L. (1999) · RAND Corporation eBooks · doi:10.7249/ip182 · Cited by 19
Mode of Administration Effects on MCAS Composition Performance for Grades Four, Eight, and Ten
Russell, M., & Plati, T. (2000) · Cited by 19
From Simulation to Application: Examinees React to Computerized Testing
Pommerich, M., & Burden, T. (2000) · Cited by 18
Ability Measure Equivalence of Computer Adaptive and Pencil and Paper Tests: A Research Synthesis.
Bergstrom, B. A. (1992) · Cited by 16
DERIVING COMPARABLE SCORES FOR COMPUTER ADAPTIVE AND CONVENTIONAL TESTS: AN EXAMPLE USING THE SAT1,2
Eignor, D. R. (1993) · ETS Research Report Series · doi:10.1002/j.2333-8504.1993.tb01566.x · Cited by 16
Children’s reading skills: A comparison of traditional and computerized assessment
Evans, L. D., Tannehill, R. L., & Martin, S. (1995) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204723 · Cited by 15
Meta-Analysis of Equivalence of Computerized and P&P Tests on Ability Measures
Kim, J. (1999) · Cited by 12
A DIF Analysis of Item-Level Mode Effects for Computerized and Paper-and-Pencil Tests.
Schwarz, R. D., Rich, C. S., & Podrabsky, T. (2004) · Cited by 11
Kingsbury, G. G., & Houser, R. L. (1988) · Cited by 9
Administration Mode Comparability Study for Stanford Diagnostic Reading and Mathematics Tests
Wang, S. (2004) · Cited by 6
Self-citationThe Validity of a Computerized Measure of Reading Rate
Pomplun, M. R., Frey, S., Becker, D. F., & Hughes, K. J. (2000) · American Educational Research Association Annual Meeting · Cited by 6
Research works printed but not linked: 34
- American Council on Education., 1995, Guidelines for computerized adaptive test development and use in education
- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education., 1999, Standards for educational and psychological testing
- American Psychological Association Committee on Professional Standards and Committee on Psychological Tests and Assessment., 1986, Guidelines for computer-based tests and interpretations
- Arce-Ferrer, A., 2004, Comparison of paper-and-pencil and online versions of Grade 5 mathematics and Grade 8 reading and writing tests
- Association of Test Publishers., 2000, Guidelines for computer-based testing
- Bennett, R.E., 2001, Education Policy Analysis Archive,, 9
- Bennett, R.E., 2002, Journal of Technology, Learning, and Assessment,, 1
- Bergstrom, B., Paper presented at the annual meeting of the American Educational Research Association
- Boo, J., Paper presented at the annual meeting of the National Council on Measurement in Education
- Choi, S.W., Paper presented at the annual meeting of the National Council on Measurement in Education
- Folk, V.G., Paper presented at the ETS-sponsored colloquium on Computer-Based Tests: Building the Foundation for Future Assessments
- Godwin, J., Paper presented at the annual meeting of the National Council on Measurement in Education
- Goldberg, A., 2003, Journal of Technology, Learning, and Assessment,, 2
- 10.1037/0003-066X.42.5.443 · doi:10.1037/0003-066x.42.5.443 · Paper page
- Ito, K., Paper presented at the annual meeting of the American Educational Research Association
- Johnson, M., Paper presented at the British Educational Research Association Annual Conference
- Kim, J., Paper presented at the annual meeting of the Mid-Western Educational Research Association
- Kingsbury, G.G., Paper presented at the annual meeting of the American Educational Research Association
- Kingsbury, G.G., Paper presented at the annual meeting of the American Educational Research Association
- Mead, A.D., 1993, Psychological Bulletin,, 9, 287
- National Association of State Boards of Education., 2001, Any time, any place, any path, any pace: Taking the lead on e-learning policy
- National Center for Education Statistics., 2000, Teacher use of computers and the Internet in public schools (NCES 2000-090)
- Park, J., 2003, Education Week, 22, 15
- Poggio, J., Paper presented at the annual meeting of the National Council on Measurement in Education
- Pommerich, M., 2004, Journal of Technology, Learning, and Assessment,, 2
- Pommerich, M., Paper presented at the annual meeting of the National Council on Measurement in Education
- Pomplun, M., Paper presented at the annual meeting of the American Educational Research Association
- Russell, M., 1999, Educational Policy Analysis Archives, 7
- Russell, M., 1997, Educational Policy Analysis Archives, 5
- Russell, M., 2001, Teachers College Record
- Schwarz, R.D., Paper presented at the annual meeting of the National Council on Measurement in Education
- Stenner, A.J., Paper presented at the California Comparability Symposium
- Vispoel, W.P., Paper presented at the annual meeting of the AERA
- Wang, S., 2000, AT&T aptitude test equivalence study: A comparison of computer and paper-and-pencil employment examinations
Cited by
Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018) · Educational Research Review · doi:10.1016/j.edurev.2018.09.003 · Cited by 664
Reading from paper compared to screens: A systematic review and meta‐analysis
Clinton, V. (2019) · Journal of Research in Reading · doi:10.1111/1467-9817.12269 · Cited by 436
Coiro, J. (2021) · Reading Research Quarterly · doi:10.1002/rrq.302 · Cited by 216
Technological Issues for Computer-Based Assessment
Csapó, B., Ainley, J., Bennett, R. E., Latour, T., & Law, N. W. Y. (2011) · Springer eBooks · doi:10.1007/978-94-007-2324-5_4 · Cited by 203
Reiss, K., Weis, M., Klieme, E., Köller, O., & OECD (2019) · Waxmann Verlag GmbH eBooks · doi:10.31244/9783830991007 · Cited by 209
Metadata
- Authors
- Wang, S., Jiao, H., Young, M. J., Brooks, T., & Olson, J.
- Affiliations
- Harcourt Assessment, Inc., shudong_wang@ harcourt.com
- Harcourt Assessment, Inc.
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2007-09-12
- Volume
- 68
- Issue
- 1
- Pages
- 5-24
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- OpenAlex
- W2079754662
- Cited by
- 186
- References
- 55
- Citations by year
- 2026: 9 · 2025: 9 · 2024: 10 · 2023: 11 · 2022: 15 · 2021: 15 · 2020: 14 · 2019: 13 · 2018: 13 · 2017: 7 · 2016: 5 · 2015: 20 · 2014: 6 · 2013: 9 · 2012: 11
- FWCI
- 5.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational and Psychological Measurement (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Keywords
- computer-based testing, paper-and-pencil tests, test comparability, reading assessment, K-12 students, reading achievement, computer practice, test format effects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.