The Relative Merits of Observational and Experimental Research: Four Key Principles for Optimising Observational Research Designs
Open accessAbstract
The main barrier to the publication of observational research is a perceived inferiority to randomised designs with regard to the reliability of their conclusions. This commentary addresses this issue and makes a set of recommendations. It analyses the issue of research reliability in detail and fully describes the three sources of research unreliability (certainty, risk and uncertainty). Two of these (certainty and uncertainty) are not adequately addressed in most research texts. It establishes that randomised designs are vulnerable as observation studies to these two sources of unreliability, and are therefore not automatically superior to observational research in all research situations. Two key principles for reducing research unreliability are taken from R.A. Fisher’s early work on agricultural research. These principles and their application are described in detail. The principles are then developed into four key principles that observational researchers should follow when they are designing observational research exercises in nutrition. It notes that there is an optimal sample size for any particular research exercise that should not be exceeded. It concludes that best practice in observational research is to replicate this optimal sized observational exercise multiple times in order to establish reliability and credibility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The boundary conditions for any such generalisations can only be established by replication within other highly specific environments.
“The boundary conditions for any such generalisations can only be established by replication within other highly specific environments [64].”
From Marketing Tactics for Sustainable Fashion and the Circular Economy: The Impact of Ethical Labels on Fast Fashion Choice (2023) · cites this paper as [64]
Section 4 says replication can test a result’s reliability and help identify environmental variables when results differ; Section 6 recommends replicating studies using temporally and environmentally homogeneous samples. The paper does not say that the boundary conditions of generalisations can only be established through replication in other highly specific environments. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rosenbaum, P. R. (2002) · Springer series in statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,414 · see citing works
Eight Myths About Causality and Structural Equation Models
Bollen, K. A., & Pearl, J. (2013) · Handbooks of sociology and social research · doi:10.1007/978-94-007-6094-3_15 · Cited by 680 · see citing works
Anglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534 · see citing works
Rao, V. R. (2014) · Springer eBooks · doi:10.1007/978-3-540-87753-0 · Cited by 464 · see citing works
In the 21st Century, what is an acceptable response rate?
Morton, S., Bandara, D. K., Robinson, E. M., & Carr, P. A. (2012) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1753-6405.2012.00854.x · Cited by 379 · see citing works
Introductory Statistics for Biology
Parker, R. E. (1991) · Cambridge University Press eBooks · doi:10.1017/cbo9781139168328 · Cited by 298 · see citing works
on behalf of the HES Response Rate Group, Mindell, J. S., Giampaoli, S., Gößwald, A., Kamtsiuris, P., Mann, C., Männistö, S., Morgan, K., Shelton, N. J., Verschuren, W. M. M., et al. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0072-4 · Cited by 206 · see citing works
Barriers to the conduct of randomised clinical trials within all disease areas
Djurisic, S., Rath, A., Gaber, S., Garattini, S. K., Bertelè, V., Bikeye, S.-N. N., Hivert, V., Neugebauer, E. A. M., Laville, M., Hiesmayr, M., et al. (2017) · Trials · doi:10.1186/s13063-017-2099-9 · Cited by 169 · see citing works
Participation bias, self-selection bias, and response bias
Elston, D. M. (2021) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2021.06.025 · Cited by 161 · see citing works
Andrew, C. O., & Hildebrand, P. E. (2019) · Routledge eBooks · doi:10.4324/9780429041778 · Cited by 117 · see citing works
Structural equation modeling in social science research
Babin, B. J., & Svensson, G. (2012) · European Business Review · doi:10.1108/09555341211242132 · Cited by 97 · see citing works
Does the Australasian “Health Star Rating” Front of Pack Nutritional Label System Work?
Hamlin, R., & McNeill, L. (2016) · Nutrients · doi:10.3390/nu8060327 · Cited by 76 · see citing works
Self-citationThe past, present and future of observational research in marketing
Lee, N. J., & Broderick, A. J. (2007) · Qualitative Market Research An International Journal · doi:10.1108/13522750710740790 · Cited by 64 · see citing works
Clinical Knowledge from Observational Studies: Everything You Wanted to Know but Were Afraid to Ask
Gershon, A. S., Jafarzadeh, S. R., Wilson, K. C., & Walkey, A. J. (2018) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201801-0118pp · Cited by 56 · see citing works
THE ORIGINS OF AGRISCIENCE: OR WHERE DID ALL THAT SCIENTIFIC AGRICULTURE COME FROM?
Hillison, J. (1996) · Journal of Agricultural Education · doi:10.5032/jae.1996.04008 · Cited by 55 · see citing works
Occam's Razor: From Ockham's via Moderna to Modern Data Science
van den Berg, H. A. (2018) · Science Progress · doi:10.3184/003685018x15295002645082 · Cited by 52 · see citing works
Observational research rigour alone does not justify causal inference
Ejima, K., Li, P., Smith, D. L., Nagy, T. R., Kadish, I., van Groen, T., Dawson, J., Yang, Y., Patki, A., & Allison, D. B. (2016) · European Journal of Clinical Investigation · doi:10.1111/eci.12681 · Cited by 45 · see citing works
Langton, S. J. (1990) · Agroforestry Systems · doi:10.1007/bf00123472 · Cited by 43 · see citing works
Research Methods in Community Medicine
Abramson, J. H., & Abramson, Z. H. (2008) · Wiley eBooks · doi:10.1002/9780470987230 · Cited by 37 · see citing works
Partially Replicated Latin Squares
Youden, W. J., & Hunter, J. S. (1955) · Biometrics · doi:10.2307/3001719 · Cited by 35 · see citing works
Boon, M. H., Burns, J., Craig, P., Griebler, U., Heise, T. L., Katikireddi, S. V., Rehfuess, E. A., Shepperd, S., Thomson, H., & Bero, L. A. (2022) · American Journal of Public Health · doi:10.2105/ajph.2021.306658 · Cited by 30 · see citing works
The Potential of Conjoint Analysis for Communication Research
Pelzer, E. (2018) · Communication Research Reports · doi:10.1080/08824096.2018.1559138 · Cited by 28 · see citing works
Bailey, R. L., Sahni, S., Chocano-Bedoya, P. O., Daly, R. M., Welch, A. A., Bischoff-Ferrari, H. A., & Weaver, C. M. (2018) · Advances in Nutrition · doi:10.1093/advances/nmy111 · Cited by 28 · see citing works
Hu, Z., & Qin, J. (2018) · Statistics in Medicine · doi:10.1002/sim.7808 · Cited by 24 · see citing works
History of the Statistical Design of Agricultural Experiments
Verdooren, L. R. (2020) · Journal of Agricultural Biological and Environmental Statistics · doi:10.1007/s13253-020-00394-3 · Cited by 22 · see citing works
The rise and fall of the Latin Square in marketing: A cautionary tale
Hamlin, R. P. (2005) · European Journal of Marketing · doi:10.1108/03090560510581809 · Cited by 22 · see citing works
Self-citationHamlin, R., & McNeill, L. (2018) · Nutrients · doi:10.3390/nu10070906 · Cited by 21 · see citing works
Self-citationBeyerbach, J., Stadelmaier, J., HOFFMANN, G. E., Balduzzi, S., Bröckelmann, N., & Schwingshackl, L. (2021) · Advances in Nutrition · doi:10.1093/advances/nmab095 · Cited by 19 · see citing works
Sorkin, J. D., MANARY, M. J., Smeets, P. A. M., MacFarlane, A. J., Astrup, A. V., Prigeon, R. L., Hogans, B. B., Odle, J., Davis, T. A., Tucker, K. L., et al. (2021) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqab223 · Cited by 18 · see citing works
Mulvany, J. L., Hetherington, V. J., & VanGeest, J. B. (2019) · The Foot · doi:10.1016/j.foot.2019.05.005 · Cited by 13 · see citing works
Hamlin, R. P. (2017) · Australasian Marketing Journal (AMJ) · doi:10.1016/j.ausmj.2017.05.002 · Cited by 10 · see citing works
Self-citation(2019) · CRC Press eBooks · doi:10.1201/9780429187445 · Cited by 8 · see citing works
Maitland, S. B., Brauer, P. M., Mutch, D. M., Royall, D., Klein, D., Tremblay, A., Rhéaume, C., Dhaliwal, R., & Jeejeebhoy, K. N. (2021) · Nutrients · doi:10.3390/nu13124258 · Cited by 5 · see citing works
Design of Observational Nutrition Studies
Pounis, G. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-814556-2.00001-4 · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 35 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- [1]Drummond K.E. Reyes A. Cooke N.K. Stage V.C. Nutrition Research: Concepts and Applications John Wiley Hoboken, NJ, USA 2022
- [4]Longchamps L. Tremblay N. Panneton B. Observational Studies in Agriculture: Paradigm Shift Required Proceedings of the 15th International Conference on Precision Agriculture Montreal, QC, Canada 24–27 June 2018 Available online: https://www.ispag.org/proceedings/?action=download&item=5436 (accessed on 1 July 2022)
- [32]Banks S. Experimentation in Marketing McGraw-Hill New York, NY, USA 1965
- [33]Tabachnik B.G. Fidell L.S. Using Multivariate Statistics 7th ed. Pearson London, UK 2018
- [34]Dey A. Mukerjee R. Fractional Factorial Plans John Wiley & Sons Hoboken, NJ, USA 2009
- [37]Louviere J.J. Conjoint Analysis Advanced Methods of Marketing Research Blackwell Publications Cambridge, MA, USA 1994 223 259
Cited by
Sorted by citations
Wang, C., He, S., Xie, G., Zhang, S., Xiong, Z., Lu, H., Wang, Q., Xie, L., Wang, W., Zou, Y., et al. (2025) · Cardiovascular Diabetology · doi:10.1186/s12933-025-02972-6 · Cited by 47 · see citing works
Hamlin, R. P., & McNeill, L. S. (2023) · Sustainability · doi:10.3390/su151310331 · Cited by 10 · see citing works
Self-citationDeveloping Food Consumer Attitudes towards Ionizing Radiation and Genetic Modification
Junaedi, I., McNeill, L. S., & Hamlin, R. (2024) · Nutrients · doi:10.3390/nu16203427 · Cited by 4 · see citing works
Self-citationHong, D., Li, X., Sheng, G., Yang, H., Wang, W., & Zou, Y. (2025) · Lipids in Health and Disease · doi:10.1186/s12944-025-02579-6 · Cited by 4 · see citing works
Peluso, I., Raguzzini, A., Toti, E., Boccia, G., Ferrara, R., Munzi, D., Brustio, P. R., Rainoldi, A., Cavedon, V., Milanese, C., et al. (2025) · Journal of Functional Morphology and Kinesiology · doi:10.3390/jfmk10030305 · Cited by 1 · see citing works
The top 50 cited publications in the field of vitamin B12 deficiency: A bibliometric analysis review
Aldakhil, M. D., Aldakhil, H. A., Istanboli, A., Alamri, R. S., Aljuhani, L. I., Alahmed, M. A., Basalamah, K. A., Alzibali, Y. M., Almufleh, A. S., Harthi, O. M. A., et al. (2025) · Journal of Family Medicine and Primary Care · doi:10.4103/jfmpc.jfmpc_2096_24
- Authors
- Hamlin, R.
- Affiliations
- University of Otago (NZ)
- Corresponding author
- Hamlin, Robert
- ORCID
- Hamlin, Robert
- Venue
- Nutrients
- Publisher
- MDPI AG
- Published
- 2022-11-03
- Volume
- 14
- Issue
- 21
- Pages
- 4649
- ISSN
- 2072-6643 (ISSN-L)
- PMC
- PMC9658032
- OpenAlex
- W4308198425
- Cited by
- 6
- References
- 35
- Citations by year
- 2025: 4 · 2024: 1 · 2023: 1
- FWCI
- 1.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Nutrients (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Health Sciences › Health Professions › Health Information Management › Methodologies in Health Research and Practice
- Keywords
- randomized controlled trials, research reliability, certainty, risk, uncertainty, Fisher's principle, sample size, replication, nutrition research, research credibility
- MeSH terms
- Research Design, Reproducibility of Results, Sample Size, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
