Research Commentary —Too Big to Fail: Large Samples and the p -Value Problem
Abstract
The Internet has provided IS researchers with the opportunity to conduct studies with extremely large samples, frequently well over 10,000 observations. There are many advantages to large samples, but researchers using statistical inference must be aware of the p-value problem associated with them. In very large samples, p-values go quickly to zero, and solely relying on p-values can lead the researcher to claim support for results of no practical significance. In a survey of large sample IS research, we found that a significant number of papers rely on a low p-value and the sign of a regression coefficient alone to support their hypotheses. This research commentary recommends a series of actions the researcher can take to mitigate the p-value problem in large samples and illustrates them with an example of over 300,000 camera sales on eBay. We believe that addressing the p-value problem will increase the credibility of large sample IS research as well as provide more insights for readers.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Owing to the very large sample sizes, P values were not calculated since they can be very small even if differences between cohorts are of no clinical significance.
“Owing to the very large sample sizes in this study, P values were not calculated since they can be very small even if differences between cohorts are of no clinical significance.49”
From Real-world menstrual cycle characteristics of more than 600,000 menstrual cycles (2019) · cites this paper as [49]
Cites
Galit Shmueli, A. (2010) · Statistical Science · doi:10.1214/10-sts330 · Cited by 3,121 · see citing works
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781 · see citing works
Chris Forman, A., Ghose, A., & Wiesenfeld, B. M. (2008) · Information Systems Research · doi:10.1287/isre.1080.0193 · Cited by 1,698 · see citing works
Predictive Analytics in Information Systems Research1
Shmueli, G., & Koppius, O. R. (2011) · MIS Quarterly · doi:10.2307/23042796 · Cited by 1,328 · see citing works
Self-citationEntrepreneurship and bank credit availability
Black, S. E., & Strahan, P. E. (2002) · The Journal of Finance · doi:10.1111/1540-6261.00513 · Cited by 1,140 · see citing works
Specification Searches: Ad Hoc Inference with Non-Experimental Data.
Richard, J., & Leamer, E. E. (1981) · Economica · doi:10.2307/2553491 · Cited by 1,129 · see citing works
The Puzzling Persistence of the Distance Effect on Bilateral Trade
Studi, C., Head, K. C., D’agliano, C. S. L., Université De Paris, I., & Head, K. (2004) · Cited by 1,120 · see citing works
Self-Selection and Information Role of Online Product Reviews
Li, X., & Hitt, L. M. (2008) · Information Systems Research · doi:10.1287/isre.1070.0154 · Cited by 1,101 · see citing works
Limited Asset Market Participation and the Elasticity of Intertemporal Substitution
Vissing‐Jørgensen, A. (2002) · Journal of Political Economy · doi:10.1086/340782 · Cited by 1,084 · see citing works
The case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931 · see citing works
Pavlou, P. A., & Dimoka, A. (2006) · Information Systems Research · doi:10.1287/isre.1060.0106 · Cited by 930 · see citing works
Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models
Lyles, R. H. (2006) · Journal of the American Statistical Association · doi:10.1198/jasa.2006.s90 · Cited by 790 · see citing works
The Philosophy of Multiple Comparisons
Tukey, J. W. (1991) · Statistical Science · doi:10.1214/ss/1177011945 · Cited by 773 · see citing works
Jones, Q., Ravid, G., & Rafaeli, S. (2004) · Information Systems Research · doi:10.1287/isre.1040.0023 · Cited by 722 · see citing works
An Empirical Analysis of Search Engine Advertising: Sponsored Search in Electronic Markets
Ghose, A., & Yang, S. (2009) · Management Science · doi:10.1287/mnsc.1090.1054 · Cited by 684 · see citing works
Predictive Analytics in Information Systems Research
Shmueli, G., & Koppius, O. (2010) · doi:10.2139/ssrn.1606674 · Cited by 676 · see citing works
Regression Methods in Biostatistics
Vittinghoff, E., Glidden, D. V., Shiboski, S. C., & McCulloch, C. E. (2012) · Statistics in the health sciences · doi:10.1007/978-1-4614-1353-0 · Cited by 666 · see citing works
Things I have learned (so far).
Cohen, J. (1992) · American Psychological Association eBooks · doi:10.1037/10109-028 · Cited by 553 · see citing works
PENNIES FROM EBAY: THE DETERMINANTS OF PRICE IN ONLINE AUCTIONS*
Lucking‐Reiley, D., Bryan, D., Prasad, N., & Reeves, D. M. (2007) · Journal of Industrial Economics · doi:10.1111/j.1467-6451.2007.00309.x · Cited by 550 · see citing works
Chris Forman, A., Ghose, A., & Avi Goldfarb, B. (2008) · Management Science · doi:10.1287/mnsc.1080.0932 · Cited by 538 · see citing works
Battle of the Retail Channels: How Product Selection and Geography Drive Cross-Channel Competition
Brynjolfsson, E., Hu, Y. J., & Rahman, M. S. (2009) · Management Science · doi:10.1287/mnsc.1090.1062 · Cited by 437 · see citing works
The Sound of Silence in Online Feedback: Estimating Trading Risks in the Presence of Reporting Bias
Dellarocas, C. N., & Wood, C. A. (2007) · Management Science · doi:10.1287/mnsc.1070.0747 · Cited by 434 · see citing works
The Impact of Internet Subsidies in Public Schools
Goolsbee, A., & Guryan, J. (2006) · The Review of Economics and Statistics · doi:10.1162/rest.88.2.336 · Cited by 368 · see citing works
Learning from Experience in Software Development: A Multilevel Analysis
Boh, W. F., Slaughter, S. A., & Espinosa, J. A. (2007) · Management Science · doi:10.1287/mnsc.1060.0687 · Cited by 344 · see citing works
The Cult of Statistical Significance
McCLOSKEY, D. N., & Ziliak, S. T. (2008) · University of Michigan Press eBooks · doi:10.3998/mpub.186351 · Cited by 341 · see citing works
Market Value of Voluntary Disclosures Concerning Information Security1
Gordon, Loeb, M. P., & Sohail, T. (2010) · MIS Quarterly · doi:10.2307/25750692 · Cited by 320 · see citing works
What Happens When You Tax the Rich? Evidence from Executive Compensation
Goolsbee, A. (2000) · Journal of Political Economy · doi:10.1086/262122 · Cited by 313 · see citing works
Ghose, A., Smith, M. D., & Telang, R. (2006) · Information Systems Research · doi:10.1287/isre.1050.0072 · Cited by 288 · see citing works
The Demand for Housing: A Review of Cross-Section Evidence
de Leeuw, F. (1971) · The Review of Economics and Statistics · doi:10.2307/1925374 · Cited by 270 · see citing works
An Empirical Analysis of Network Externalities in Peer-to-Peer Music-Sharing Networks
Asvanund, A., Clay, K. B., Krishnan, R., & Smith, M. D. (2004) · Information Systems Research · doi:10.1287/isre.1040.0020 · Cited by 267 · see citing works
Is the World Really Flat? A Look at Offshoring at an Online Programming Marketplace1
Gefen, D., & Carmel, E. (2008) · MIS Quarterly · doi:10.2307/25148844 · Cited by 230 · see citing works
What Happens When You Tax the Rich? Evidence from Executive Compensation
Goolsbee, A. (1997) · National Bureau of Economic Research · doi:10.3386/w6333 · Cited by 230 · see citing works
Competing with Free: The Impact of Movie Broadcasts on DVD Sales and Internet Piracy
Smith, M. D., & Telang, R. (2008) · MIS Quarterly · doi:10.2307/20650294 · Cited by 229 · see citing works
Internet Exchanges for Used Goods: An Empirical Analysis of Trade Patterns and Adverse Selection1
Ghose, A. (2009) · MIS Quarterly · doi:10.2307/20650292 · Cited by 210 · see citing works
Market value of voluntary disclosures concerning information security
Gordon, L. A., Loeb, M. P., & Sohail, T. (2010) · MIS Quarterly · doi:10.5555/2017470.2017479 · Cited by 175 · see citing works
The Significance of Statistical Significance Tests in Marketing Research
Sawyer, A. G., & Peter, J. P. (1983) · Journal of Marketing Research · doi:10.1177/002224378302000203 · Cited by 175 · see citing works
The Role of Feedback in Managing the Internet-Based Volunteer Work Force
Moon, J. Y., & Sproull, L. S. (2008) · Information Systems Research · doi:10.1287/isre.1080.0208 · Cited by 156 · see citing works
Mithas, S., & Krishnan, M. S. (2008) · Management Science · doi:10.1287/mnsc.1070.0778 · Cited by 145 · see citing works
Problem Solving: A Statistician's Guide.
Kafandaris, S. P., & Chatfield, C. (1996) · Journal of the Operational Research Society · doi:10.2307/3010293 · Cited by 141 · see citing works
Mithas, S., & Lucas, H. C. (2010) · Management Science · doi:10.1287/mnsc.1100.1149 · Cited by 129 · see citing works
Self-citationUsing Transaction Prices to Re-Examine Price Dispersion in Electronic Markets
Ghose, A., & Yao, Y. (2010) · Information Systems Research · doi:10.1287/isre.1090.0252 · Cited by 111 · see citing works
Overby, E. M., & Jap, S. D. (2009) · Management Science · doi:10.1287/mnsc.1090.0998 · Cited by 106 · see citing works
Chatfield, C. (1995) · Chapman & Hall/CRC eBooks · doi:10.1201/b15238 · Cited by 98 · see citing works
Thompson, B. (1989) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.1990.12022905 · Cited by 89 · see citing works
The Significance of Statistical Significance Tests in Marketing Research
Sawyer, A. G., & Peter, J. P. (1983) · Journal of Marketing Research · doi:10.2307/3151679 · Cited by 80 · see citing works
Why We Don't Really Know What Statistical Significance Means: Implications for Educators
Hubbard, R., & Armstrong, J. S. (2006) · Journal of Marketing Education · doi:10.1177/0273475306288399 · Cited by 63 · see citing works
Euro-Illusion: A Natural Experiment
Cannon, E. S., & Cipriani, G. P. (2006) · Journal of money credit and banking · doi:10.1353/mcb.2006.0066 · Cited by 49 · see citing works
Private Network EDI vs. Internet Electronic Markets: A Direct Comparison of Fulfillment Performance
Yao, Y., Dresner, M., & Palmer, J. W. (2009) · Management Science · doi:10.1287/mnsc.1080.0990 · Cited by 41 · see citing works
Examining the Relationship Between Reviews and Sales
FormanChris, GhoseAnindya, & WiesenfeldBatia (2008) · Information Systems Research · doi:10.5555/3215995.3215996 · Cited by 31 · see citing works
Why We Don't Really Know What ‘Statistical Significance’ Means: A Major Educational FailurePreprint
Armstrong, J. S., & Hubbard, R. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1154386 · Cited by 25 · see citing works
Intergenerational Effects on Fertility Behavior and Earnings Mobility in Spain
Iglesias, F. H., & Riboud, M. (1988) · The Review of Economics and Statistics · doi:10.2307/1928309 · Cited by 12 · see citing works
Euro-Illusion: A Natural ExperimentPreprint
Cipriani, G. P., & Cannon, E. S. (2006) · SSRN Electronic Journal · Cited by 11 · see citing works
Household-Specific Regressions Using Clickstream Data
Goldfarb, A., & Lu, Q. (2006) · Statistical Science · doi:10.1214/088342306000000150 · Cited by 7 · see citing works
Stata Tip 53: Where did My P-Values Go?
Buis, M. L. (2007) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0800700408 · Cited by 7 · see citing works
SmithMichael, D., & TelangRahul (2009) · MIS Quarterly · doi:10.5555/2017424.2017429 · Cited by 5 · see citing works
Stata Tip 53: Where did My P-Values Go?
Buis, M. L. (2007) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0700700408 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Big Data Research in Information Systems: Toward an Inclusive Research Agenda
University of Virginia, Abbasi, A., Sarker, S., University of Virginia, Aalto University, Chiang, R., & University of Cincinnati (2016) · Journal of the Association for Information Systems · doi:10.17705/1jais.00423 · Cited by 688 · see citing works
Real-world menstrual cycle characteristics of more than 600,000 menstrual cycles
Bull, J. R., Rowland, S. P., Scherwitzl, E. B., Scherwitzl, R., Danielsson, K. G., & Harper, J. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0152-7 · Cited by 404 · see citing works
“Owing to the very large sample sizes in this study, P values were not calculated since they can be very small even if differences between cohorts are of no clinical significance.(49) Instead, effect size between two cohorts was estimated as a mean difference with a 95% confidence interval calculated from 200 bootstrap…”
MethodKhalilzadeh, J., & Tasci, A. D. A. (2017) · Tourism Management · doi:10.1016/j.tourman.2017.03.026 · Cited by 298 · see citing works
Mapping neuroplastic potential in brain-damaged patients
Herbet, G., Maheu, M., Costi, E., Lafargue, G., & Duffau, H. (2016) · Brain · doi:10.1093/brain/awv394 · Cited by 289 · see citing works
Fostering public trust in science: The role of social media
Huber, B., Barnidge, M., Gil de Zúñiga, H., & Liu, J. (2019) · Public Understanding of Science · doi:10.1177/0963662519869097 · Cited by 231 · see citing works
Dissociable effects of self-reported daily sleep duration on high-level cognitive abilities
Wild, C. J., Nichols, E. S., Battista, M. E., Stojanoski, B., & Owen, A. M. (2018) · Sleep · doi:10.1093/sleep/zsy182 · Cited by 123 · see citing works
Robillard, R., Daros, A. R., Phillips, J. L., Porteous, M., Saad, M., Pennestri, M.-H., Kendzerska, T., Edwards, J. D., Solomonova, E., Bhatla, R., et al. (2021) · The Canadian Journal of Psychiatry · doi:10.1177/0706743720986786 · Cited by 112 · see citing works
Pot, M., Paulussen, T. G., Ruiter, R. A., Eekhout, I., de Melker, H. E., Spoelstra, M. E., & van Keulen, H. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7449 · Cited by 62 · see citing works
Evers, N. F. G., Greenfield, P. M., & Evers, G. W. (2021) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.251 · Cited by 57 · see citing works
Esser, P., Mehnert, A., Johansen, C., Hornemann, B., Dietz, A., & Ernst, J. (2018) · Psycho-Oncology · doi:10.1002/pon.4494 · Cited by 53 · see citing works
Catarino, A., Bateup, S., Tablan, V., Innes, K., Freer, S., Richards, A., Stott, R., Hollon, S. D., Chamberlain, S. R., Hayes, A., et al. (2018) · BJPsych Open · doi:10.1192/bjo.2018.57 · Cited by 40 · see citing works
Drummond, A., & Sauer, J. D. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103704 · Cited by 42 · see citing works
Prevalence of 3 Healthy Lifestyle Behaviors Among US Adults With and Without History of Stroke
Bailey, R. R., Phad, A., McGrath, R., Tabak, R., & Haire-Joshu, D. (2019) · Preventing Chronic Disease · doi:10.5888/pcd16.180409 · Cited by 12 · see citing works
Childhood emotional dysregulation paths for suicide-related behaviour engagement in adolescence
de la Torre-Luque, A., Essau, C. A., Lara, E., Leal-Leturia, I., & Borges, G. (2023) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-022-02111-6 · Cited by 11 · see citing works
Souza, J., Caballero, I., Vasco Santos, J., Lobo, M., Pinto, A., Viana, J., Sáez, C., Lopes, F., & Freitas, A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104242 · Cited by 7 · see citing works
- Authors
- Lin, M., Lucas, H. C., & Shmueli, G.
- Affiliations
- University of Arizona (US)
- University of Maryland, College Park (US)
- Srini Raju Centre for IT & the Networked Economy, Indian School of Business, Hyderabad 500 032, India
- Venue
- Information Systems Research
- Published
- 2013-12
- Volume
- 24
- Issue
- 4
- Pages
- 906-917
- ISSN
- 1047-7047 (ISSN-L)
- 1526-5536
- OpenAlex
- W1991208598
- Cited by
- 972
- References
- 56
- Citations by year
- 2026: 63 · 2025: 86 · 2024: 78 · 2023: 93 · 2022: 103 · 2021: 104 · 2020: 108 · 2019: 79 · 2018: 71 · 2017: 55 · 2016: 55 · 2015: 44 · 2014: 27 · 2013: 5 · 2012: 1
- FWCI
- 48.10
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Information Systems Research (publishedVersion)
- Digital Eprints Services at ISB (DESI) (Indian School of Business) (acceptedVersion)
- Digital Eprints Services at ISB (DESI) (Indian School of Business) (submittedVersion)
- Funders
- University of South Florida
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- large sample size, statistical significance, practical significance, hypothesis testing, regression coefficients, research credibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
