Technological Innovations in Large-Scale Assessment
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Computers have had a tremendous impact on assessment practices over the past half century. Advances in computer technology have substantially influenced the ways in which tests are made, administered, scored, and reported to examinees. These changes are particularly evident in computer-based testing, where the use of computers has allowed test developers to re-envision what test items look like and how they are scored. By integrating technology into assessments, it is increasingly possible to create test items that can sample as broad or as narrow a range of behaviors as needed while preserving a great deal of fidelity to the construct of interest. In this article we review and illustrate some of the current technological developments in computer-based testing, focusing on novel item formats and automated scoring methodologies. Our review indicates that a number of technological innovations in performance assessment are increasingly being researched and implemented by testing programs. In some cases, complex psychometric and operational issues have successfully been dealt with, but a variety of substantial measurement concerns associated with novel item types and other technological aspects impede more widespread use. Given emerging research, however, there appears to be vast potential for expanding the use of more computerized constructed-response type items in a variety of testing contexts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An introduction to latent semantic analysis
Landauer, T. K., Foltz, P. W., & Laham, D. (1998) · Discourse Processes · doi:10.1080/01638539809545028 · Cited by 4,880 · cited 1×
“Latent semantic analysis (LSA) is a theory and method for extracting and representing the contextual usage of words by statistical computations applied to a large corpus of text (Foltz, Kintsch, & Landauer, 1998; Landauer, Foltz, & Laham, 1998; Rehder et al., 1998).”
MethodStandards of validity and the validity of standards in performance asessment
Messick, S. (1995) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1995.tb00881.x · Cited by 697
Performance‐Based Assessment: Implications of Task Specificity
Linn, R. L., & Burton, E. (1994) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.1994.tb00778.x · Cited by 152
Research works printed but not linked: 52
- Ackerman T. A., 1994, Applied Measurement in Education, 7, 255, 10.1207/s15324818ame0704_1 · DOI · Paper page
- Ackerman, T. A., Evans, J., Park, K., Tamassia, C. & Turner, R. (1999). Computer assessment using visual stimuli: A test of dermatological skin disorders. In F. Drasgow and J. B. Olson-Buchanan (Eds.), Innovations in computerized assessment (pp. 137-150). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
- Ashworth, S. D. & Joyce, T. M. (1994, April). Developing scoring protocols for a computerized multimedia in-basket exercise. Paper presented at the Ninth Annual Conference of the Society for Industrial and Organizational Psychology, Nashville, TN.
- 10.1037//0021-9010.76.4.522 · DOI
- 10.1177/01466210022031769 · DOI
- 10.1207/S15324818AME1303_5 · doi:10.1207/s15324818ame1303_5
- Bennett R. E., 1999, Journal of Educational Measurement, 36, 233, 10.1111/j.1745-3984.1999.tb00556.x · doi:10.1111/j.1745-3984.1999.tb00556.x
- Bennett R. E., 1995, Journal of Educational Measurement, 32, 19, 10.1111/j.1745-3984.1995.tb00454.x · doi:10.1111/j.1745-3984.1995.tb00454.x
- Bennett R. E., 1996, Applied Measurement in Education, 9, 133, 10.1207/s15324818ame0902_3 · doi:10.1207/s15324818ame0902_3
- Bennett R. E., 1997, Journal of Educational Measurement, 34, 64, 10.1111/j.1745-3984.1997.tb00507.x · DOI
- Bennett R. E., 1991, Applied Psychological Measurement, 15, 227, 10.1177/014662169101500302 · DOI
- Bennett R. E., 1997, Journal of Educational Measurement, 34, 162, 10.1111/j.1745-3984.1997.tb00512.x · DOI
- Braun H. I., 1990, Journal of Educational Measurement, 27, 93, 10.1111/j.1745-3984.1990.tb00736.x · doi:10.1111/j.1745-3984.1990.tb00736.x
- Brennan R. L., 1995, Educational Measurement: Issues and Practice, 14, 25, 10.1111/j.1745-3992.1995.tb00882.x · DOI
- Burstein, J. & Chodorow, M. (2002). Directions in automated essay scoring analysis. In R. Kaplan (Ed.), Oxford handbook of applied linguistics (pp. 487-497). Oxford, England: Oxford University Press.
- Burstein, J., Kukich, K., Wolff, S., Lu, C. & Chodorow, M. (1998, April). Computer analysis of essays. Paper presented at the NCME Symposium on Automated Scoring, San Diego, CA. · DOI
- 10.1016/S0747-5632(99)00032-1 · DOI
- 10.1177/01466210022031778 · DOI
- Clauser B. E., 2000, Journal of Educational Measurement, 37, 245, 10.1111/j.1745-3984.2000.tb01085.x · DOI
- Clauser B. E., 1997, Journal of Educational Measurement, 34, 141, 10.1111/j.1745-3984.1997.tb00511.x · DOI
- Crocker L., 1997, Applied Measurement in Education, 10, 83, 10.1207/s15324818ame1001_5 · DOI
- Davey T., 1997, Journal of Educational Measurement, 34, 21, 10.1111/j.1745-3984.1997.tb00505.x · doi:10.1111/j.1745-3984.1997.tb00505.x
- Desmarais, L. B., Dyer, P. J., Midkiff, K. R., Barbera, K. M., Curtis, J. R., Esrig, F. H. & Masi, D. L. (1992, May). Scientific uncertainties in the development of a multimedia test: Trade-offs and decisions. Paper presented at the Seventh Annual Conference of the Society for Industrial and Organizational Psychology, Montreal, Quebec, Canada.
- Drasgow, F., Olson-Buchanan, J. B. & Moberg, P. J. (1999). Development of an interactive video assessment: Trials and tribulations. In F. Drasgow & J. B. Olson-Buchanan (Eds.), Innovations in computerized assessment (pp. 197-219). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. · DOI
- Enright M. K., 1998, Journal of Educational Measurement, 35, 250, 10.1111/j.1745-3984.1998.tb00538.x · doi:10.1111/j.1745-3984.1998.tb00538.x
- Foltz P. W., 1998, Discourse Processes, 25, 285, 10.1080/01638539809545029 · DOI · Paper page
- Glaser, R. (1991). Expertise and assessment. In M. C. Wittrock & E. L. Baker (Eds.), Testing and cognition (pp. 17-30). Englewood Cliffs, NJ: Prentice Hall.
- Guion R. M., 1995, Educational Measurement: Issues and Practice, 14, 25, 10.1111/j.1745-3992.1995.tb00886.x · DOI
- Hambleton, R. K. (1997, October). Promising GMAT item formats for the 21st century. Invited presentation at the international workshop on the GMAT, Paris, France.
- 10.1016/S0747-5632(99)00026-6 · DOI
- Huff K. L., 2001, Educational Measurement: Issues and Practice, 20, 16, 10.1111/j.1745-3992.2001.tb00066.x · DOI
- 10.1016/S0346-251X(98)00034-7 · DOI
- Kane M. T., 1992, Evaluation and the Health Professions, 15, 163, 10.1177/016327879201500203 · DOI · Paper page
- Kane M., 1999, Educational Measurement: Issues and Practice, 18, 5, 10.1111/j.1745-3992.1999.tb00010.x · DOI · Paper page
- Landauer, T. K., Laham, D., Rehder, B. & Schreiner, M. E. (1997). How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. In G. Shafto & P. Langley (Eds.), Proceedings of the 19th annual meeting of the Cognitive Science Society (pp. 412-417). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
- Larkey, L. S. (1998). Automatic essay grading using text categorization techniques. In the Proceedings of the 21st Annual International Conference of the Association for Computing Machinery-Special Interest Group on Information Retrieval, Melbourne, Australia, 90-95. · DOI · Paper page
- Martinez M. E., 1991, Journal of Educational Measurement, 28, 131, 10.1111/j.1745-3984.1991.tb00349.x · DOI
- Martinez M. E., 1992, Applied Measurement in Education, 5, 151, 10.1207/s15324818ame0502_5 · doi:10.1207/s15324818ame0502_5
- Mills, C. (2000, February). Unlocking the promise of CBT. Keynote address presented at a conference of the Association of Test Publishers, Carmel Valley, CA.
- Mislevy, R. J., Steinberg, L. S., Breyer, F. J., Almond, R. G. & Johnson, L. (1999, September 16-17). Making sense of data from complex assessments. Paper presented at the 1999 CRESST Conference, Los Angeles, CA. · DOI
- Page E. B., 1995, Phi Delta Kappan, 76, 561
- Page E. B., 1994, Journal of Experimental Education, 62, 127, 10.1080/00220973.1994.9943835 · DOI · Paper page
- Parshall C. G., 2001, Educational Measurement: Issues and Practice, 20, 5, 10.1111/j.1745-3992.2001.tb00058.x · DOI
- Parshall, C. G., Davey, T. & Pashley, P. (2000). Innovative item types for computerized testing. In W. J. van der Linden & C. Glas (Eds.), Computer-adaptive testing: Theory and practice (pp. 129-148). Boston: Kluwer Academic. · DOI
- Rehder B., 1998, Discourse Processes, 25, 337, 10.1080/01638539809545031 · DOI · Paper page
- 10.1037//0021-9010.76.6.856 · doi:10.1037//0021-9010.76.6.856
- Shavelson R. J., 1991, Applied Measurement in Education, 4, 347, 10.1207/s15324818ame0404_7 · DOI · Paper page
- van der Linden, W. J. & Hambleton, R. K. (Eds.). (1997). Handbook of modern item response theory. New York: Springer. · DOI · Paper page
- Vispoel, W. P. (1999). Creating computerized adaptive tests of music aptitude: Problems, solutions, and future directions. In F. Drasgow & J. B. Olson-Buchanan (Eds.), Innovations in computerized assessment (pp. 151-176). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
- Williamson D. M., 1999, Journal of Educational Measurement, 36, 158, 10.1111/j.1745-3984.1999.tb00552.x · DOI · Paper page
- 10.1207/S15324818AME1504_04 · DOI · Paper page
- Zenisky, A. L. & Sireci, S. (2001). Feasibility review of selected performance assessment item types for the computerized Uniform CPA Exam (Laboratory of Psychometric and Evaluative Research Rep. No. 405). Amherst: School of Education, University of Massachusetts.
Cited by
Directrices para la traducción y adaptación de los tests: segunda edición
Muñiz, J., Elosua, P., & Hambleton, R. (2013) · Psicothema · doi:10.7334/psicothema2013.24 · Cited by 921
When Small Words Foretell Academic Success: The Case of College Admissions Essays
Pennebaker, J. W., Chung, C. K., Frazee, J., Lavergne, G. M., & Beaver, D. I. (2014) · PLoS ONE · doi:10.1371/journal.pone.0115844 · Cited by 568
The Eras and Trends of Automatic Short Answer Grading
Burrows, S., Gurevych, I., & Stein, B. (2015) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0026-8 · Cited by 461
Siddiq, F., Hatlevik, O. E., Olsen, R. V., Throndsen, I., & Scherer, R. (2016) · Educational Research Review · doi:10.1016/j.edurev.2016.05.002 · Cited by 269
Kuo, C.-Y., & Wu, H.-K. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.06.002 · Cited by 66
Metadata
- Authors
- Zenisky, A. L., & Sireci, S. G.
- Venue
- Applied Measurement in Education
- Publisher
- Informa UK Limited
- Published
- 2002-10
- Volume
- 15
- Issue
- 4
- Pages
- 337-362
- ISSN
- 0895-7347 (ISSN-L)
- 1532-4818
- OpenAlex
- W2041638702
- Cited by
- 138
- References
- 3
- FWCI
- 5.96
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Technological Innovations in Large-Scale Assessment. https://app.cochat.ai/papers/10.1207/s15324818ame1504_02
- The paper
- Zenisky, A. L., & Sireci, S. G. (2002). Technological Innovations in Large-Scale Assessment. Applied Measurement in Education. https://doi.org/10.1207/s15324818ame1504_02
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.