What We Know About Spreadsheet Errors
The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Although some spreadsheets are small “scratch pad” applications, many are large and complex, and many missioncritical decisions depend on spreadsheet analyses. In recent years, we have learned a good deal about the errors that people make when they develop spreadsheets. In general, errors seem to occur in a few percent of all cells, meaning that for large spreadsheets, the issue is how many errors there are, not whether an error exists. These error rates, although troubling, are in line with error rates in programming and other human cognitive domains. In programming, we have learned to follow strict development disciplines to eliminate most errors. Surveys of spreadsheet developers indicate that spreadsheet creation, in contrast, is informal, and that few organizations have comprehensive policies for spreadsheet development. Although prescriptive articles have focused on such disciplines as modularization and having assumptions sections, these may be far less important than postdevelopment testing.Request access from your librarian to read this article's full text.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Research shows that errors are surprisingly common in the business setting [4]”
From Gene name errors: Lessons not learned (2021) · cites this paper as [4]
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
(2013) · Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship · doi:10.1007/978-1-4614-3858-8_100431 · Cited by 4,799
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
(1998) · PDCA/Test · doi:10.1201/9781420048131.axh · Cited by 1,900
Identifying the organization of writing processes
Haves, J. (1980) · Medical Entomology and Zoology · Cited by 1,553
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Animal Intelligence: An Experimental Study of the Associative Processes in Animals
Kline, L. W., & Thorndike, E. L. (1898) · The American Journal of Psychology · doi:10.2307/1412682 · Cited by 1,092
Thinking, Problem Solving, Cognition
D., C. P., & Mayer, R. E. (1984) · The American Journal of Psychology · doi:10.2307/1422612 · Cited by 992
How We Know What Isn't So: The Fallibility of Human Reason in Everyday Life
Gilovich, T. D. (1991) · Medical Entomology and Zoology · Cited by 990
Edmondson, A. C. (1996) · The Journal of Applied Behavioral Science · doi:10.1177/0021886396321001 · Cited by 937
Cognitive Processes in Writing
Gregg, L. W. (2016) · Routledge eBooks · doi:10.4324/9781315630274 · Cited by 931
The Dynamics of Composing : Making Plans and Juggling Constraints
Flower (1980) · Medical Entomology and Zoology · Cited by 687
A small matter of programming: perspectives on end user computing
(1994) · Choice Reviews Online · doi:10.5860/choice.31-4963 · Cited by 600
Calibration of Probabilities: The State of the Art
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1977) · Decision Making and Change in Human Affairs · doi:10.1007/978-94-010-1276-8_19 · Cited by 483
The overconfidence effect in social prediction.
Dunning, D., Griffin, D. W., Milojkovic, J. D., & Ross, L. (1990) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.58.4.568 · Cited by 431
Lichtenstein, S., & Fischhoff, B. (1980) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(80)90052-5 · Cited by 375
Two Models of Group Behavior in the Solution of Eureka-Type Problems
Lorge, I., & Solomon, H. (1955) · Psychometrika · doi:10.1007/bf02288986 · Cited by 348
Two methods of reducing overconfidence
Arkes, H. R., Christensen, C., Lai, C., & Blumer, C. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90049-5 · Cited by 275
Twinkling lights and nested loops: distributed problem solving and spreadsheet development
Nardi, B. A., & Miller, J. R. (1991) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(91)90040-e · Cited by 260
A controlled experiment in program testing and code walkthroughs/inspections
Myers, G. J. (1978) · Communications of the ACM · doi:10.1145/359588.359602 · Cited by 257
An experimental study of people creating spreadsheets
Brown, P. S., & Gould, J. D. (1987) · ACM Transactions on Information Systems · doi:10.1145/27641.28058 · Cited by 179
Does The Expert Know? The Reliability of Predictions and Confidence Ratings of Experts
Wagenaar, W. A., & Keren, G. B. (1986) · NATO ASI Series · doi:10.1007/978-3-642-50329-0_6 · Cited by 159
Error detection processes in statistical problem solving
ALLWOOD, C. (1984) · Cognitive Science · doi:10.1016/s0364-0213(84)80009-9 · Cited by 154
Linger, R. C. (1994) · IEEE Software · doi:10.1109/52.268956 · Cited by 138
Spreadsheets on trial: a survey of research on spreadsheet risks
Panko, R. R., & Halverson, R. P. (1996) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1996.495416 · Cited by 133
Self-citationSpreadsheet analysis and design
Ronen, B., Palley, M. A., & Lucas, H. C. (1989) · Communications of the ACM · doi:10.1145/63238.63244 · Cited by 123
An elementary preliminary taxonomy for some errors in laboratory choice RT tasks
Rabbitt, P., & Vyas, S. M. (1970) · Acta Psychologica · doi:10.1016/0001-6918(70)90122-8 · Cited by 117
Analysis of the Cognition Involved in Spreadsheet Software Interaction
Olson, J. R., & Nilsen, E. (1987) · Human-Computer Interaction · doi:10.1207/s15327051hci0304_1 · Cited by 115
Hendry, D. G., & Green, T. R. G. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1047 · Cited by 114
Human Judgment and Decision Processes in Applied Settings
(1977) · Elsevier eBooks · doi:10.1016/c2013-0-10931-2 · Cited by 109
A Theory of Decision Support System Design for User Calibration
Kasper, G. M. (1996) · Information Systems Research · doi:10.1287/isre.7.2.215 · Cited by 108
Applying Code Inspection to Spreadsheet Testing
Panko, R. R. (1999) · Journal of Management Information Systems · doi:10.1080/07421222.1999.11518250 · Cited by 89
Self-citationHitting the wall: errors in developing and code inspecting a `simple' spreadsheet model
Panko, R. R., & Sprague, R. H. (1998) · Decision Support Systems · doi:10.1016/s0167-9236(97)00038-9 · Cited by 88
Self-citationHuman error detection processes
Rizzo, A., Bagnara, S., & Visciòla, M. (1987) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(87)80015-9 · Cited by 87
Spreadsheet Modelling Abuse: An Opportunity for OR?
Cragg, P. B., & King, M. (1993) · Journal of the Operational Research Society · doi:10.1057/jors.1993.133 · Cited by 84
SOME RESULTS ON OPERATOR PERFORMANCE IN EMERGENCY EVENTS
Woods, D. D. (1984) · Elsevier eBooks · doi:10.1016/b978-0-85295-172-9.50006-8 · Cited by 80
Spreadsheet Presentation and Error Detection: An Experimental Study
Galletta, D. F., Hartzel, K. S., Johnson, S. E., Joseph, J. L., & Rustagi, S. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518133 · Cited by 78
Using a structured design approach to reduce risks in end user spreadsheet development
Janvrin, D. J., & Morrison, J. (2000) · Information & Management · doi:10.1016/s0378-7206(99)00029-4 · Cited by 78
An empirical study of spreadsheet error-finding performance
Galletta, D. F., Abraham, D. M., Louadi, M. E., Lekse, W. J., Pollalis, Y., & Sampler, J. L. (1993) · Accounting Management and Information Technologies · doi:10.1016/0959-8022(93)90001-m · Cited by 77
Quality control in spreadsheets: a software engineering-based approach to spreadsheet development
Rajalingham, K., Chadwick, D. R., Knight, B., & Edwards, D. (2005) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2000.926736 · Cited by 73
A Study of Three Database Query Languages.
Greenblatt, D. J., & Waxman, J. (1978) · Cited by 70
End user computing: management applications and technology
Panko, R. R. (1988) · Medical Entomology and Zoology · Cited by 69
Self-citationA risk and control-oriented study of the practices of spreadsheet application developers
Hall, M. (1996) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1996.495420 · Cited by 64
Are Two Heads Better than One? (At Reducing Errors in Spreadsheet Modeling).
Panko, R. R., & Halverson, R. P. (1997) · Cited by 58
Self-citationToward a logical/physical theory of spreadsheet modeling
Isakowitz, T., Schocken, S., & Lucas, H. C. (1995) · ACM Transactions on Information Systems · doi:10.1145/195705.195708 · Cited by 58
Drivers' margins of safety considered as a focus for research on error
BROWN, I. D. (1990) · Ergonomics · doi:10.1080/00140139008925334 · Cited by 55
The relationship between objective risk, subjective risk and behaviour
Howarth, C. (1988) · Ergonomics · doi:10.1080/00140138808966697 · Cited by 54
A model of end-user computing policy
Galletta, D. F., & Hufnagel, E. M. (1992) · Information & Management · doi:10.1016/0378-7206(92)90002-w · Cited by 52
Confidence Ratings, Second-Choice Responses, and Confusion Matrices in Intelligibility Tests
Clarke, F. R. (1960) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907873 · Cited by 47
Quantitative and qualitative errors in spreadsheet development
Teo, T. S. H., & Tan, M. (2002) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1997.661583 · Cited by 43
Spreadsheets can be hazardous to your health
Ditlea, S. (1987) · Personal computing · Cited by 41
Factors influencing risks and outcomes in end-user development
Janvrin, D. J., & Morrison, J. (1996) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1996.495418 · Cited by 35
JUDGMENT AND SWINE: APPROACHES AND ISSUES IN APPLIED JUDGMENT ANALYSIS
Shanteau, J., & Phelps, R. H. (1977) · Elsevier eBooks · doi:10.1016/b978-0-12-397240-8.50018-5 · Cited by 35
Two corpuses of spreadsheet errors
Panko, R. R. (2005) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2000.926738 · Cited by 34
Self-citationSchultheis, R. A., & Sumner, M. R. (1994) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.1994040102 · Cited by 32
Goodell, H. (1997) · doi:10.1145/1120212.1120304 · Cited by 32
How to Make Spreadsheets Error-Proof
Freeman, D. M. (1996) · Journal of accountancy online/Journal of accountancy · Cited by 23
Four Applications of a Software Data Collection and Analysis Methodology
Basili, V. R., & Selby, R. W. (1986) · Springer eBooks · doi:10.1007/978-3-642-82846-1_1 · Cited by 18
Structured Techniques for Successful End User Spreadsheets
Benham, H. C., Delaney, M., & Luzi, A. (1993) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.1993040102 · Cited by 15
A New Ideomotor Theory of Voluntary Control
Baars, B. J. (1992) · Springer eBooks · doi:10.1007/978-1-4899-1164-3_4 · Cited by 15
Speier, C., & Brown, C. V. (1996) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1996.495421 · Cited by 7
A BEHAVIORAL STUDY OF THE COLLISION AVOIDANCE TASK IN BRIDGE WATCHKEEPING
Habberley, J. S., Taylor, D. H., & Shaddick, C. A. (1986) · Cited by 4
Systems, Software, and Quality Engineering: Applying Defect Behavior Theory to Programming
Ferdinand, A. E. (1993) · Medical Entomology and Zoology · Cited by 3
Cited by
Reconceptualizing System Usage: An Approach and Empirical Test
Burton-Jones, A., & Straub, D. W. (2006) · Information Systems Research · doi:10.1287/isre.1060.0096 · Cited by 1,040
The Ultimate qPCR Experiment: Producing Publication Quality, Reproducible Data the First Time
Taylor, S. C., Nadeau, K., Abbasi, M., Lachance, C., Nguyen, M., & Fenrich, J. (2019) · Trends in Biotechnology · doi:10.1016/j.tibtech.2018.12.002 · Cited by 830
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
User Developed Applications and Information Systems Success: A Test of DeLone and McLean's Model
McGill, T., Hobbs, V., & Klobas, J. (2003) · Information Resources Management Journal · doi:10.4018/irmj.2003010103 · Cited by 290
Probing the improbable: methodological challenges for risks with low probabilities and high stakes
Ord, T., Hillerbrand, R., & Sandberg, A. (2010) · Journal of Risk Research · doi:10.1080/13669870903126267 · Cited by 72
“8 find error rates on the order of 88% (Panko 1998).”
Background
Metadata
- Authors
- Panko, R. R.
- Affiliations
- University of Hawaiʻi at Mānoa (US)
- Corresponding author
- Panko, Raymond R.
- Venue
- Journal of Organizational and End User Computing
- Publisher
- IGI Global Scientific Publishing
- Published
- 1998-04
- Volume
- 10
- Issue
- 2
- Pages
- 15-21
- ISSN
- 1546-2234 (ISSN-L)
- 1546-5012
- OpenAlex
- W2112338634
- Cited by
- 464
- References
- 63
- Citations by year
- 2026: 12 · 2025: 9 · 2024: 6 · 2023: 7 · 2022: 7 · 2021: 8 · 2020: 17 · 2019: 14 · 2018: 16 · 2017: 27 · 2016: 21 · 2015: 22 · 2014: 17 · 2013: 19 · 2012: 25
- FWCI
- 33.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Organizational and End User Computing (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Social Sciences › Business, Management and Accounting › Management Information Systems › Decision Support System Applications
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- spreadsheet errors, error rate, spreadsheet development, spreadsheet testing, human error, modularization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.