The Individual Participant Data Integrity Tool for assessing the integrity of randomised trials
Open accessAbstract
Increasing concerns about the trustworthiness of research have prompted calls to scrutinise studies' Individual Participant Data (IPD), but guidance on how to do this was lacking. To address this, we developed the IPD Integrity Tool to screen randomised controlled trials (RCTs) for integrity issues. Development of the tool involved a literature review, consultation with an expert advisory group, piloting on two IPD meta‐analyses (including 73 trials with IPD), preliminary validation on 13 datasets with and without known integrity issues, and evaluation to inform iterative refinements. The IPD Integrity Tool comprises 31 items (13 study‐level, 18 IPD‐specific). IPD‐specific items are automated where possible, and are grouped into eight domains, including unusual data patterns, baseline characteristics, correlations, date violations, patterns of allocation, internal and external inconsistencies, and plausibility of data. Users rate each item as having either no issues, some/minor issue(s), or many/major issue(s) according to decision rules, and justification for each rating is recorded. Overall, the tool guides decision‐making by determining whether a trial has no concerns, some concerns requiring further information, or major concerns warranting exclusion from evidence synthesis or publication. In our preliminary validation checks, the tool accurately identified all five studies with known integrity issues. The IPD Integrity Tool enables users to assess the integrity of RCTs via examination of IPD. The tool may be applied by evidence synthesists, editors and others to determine whether an RCT should be considered sufficiently trustworthy to contribute to the evidence base that informs policy and practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
None of the previously proposed tools have involved a comprehensive evaluation and subsequent selection of potential trustworthiness checks.
“However, none of these have involved a comprehensive evaluation and subsequent selection of potential trustworthiness checks.”
From Assessing the feasibility and impact of clinical trial trustworthiness checks via an application to Cochrane Reviews: Stage 2 of the INSPECT-SR project (2025) · cites this paper as [12]
Section 3.1 describes how the authors identified, selected, piloted and refined checks for their own tool. The Discussion criticises existing checks as time-consuming, but neither passage says that no previously proposed tool comprehensively evaluated and selected potential checks. Searches for that comparison found no passage addressing it. (LLM, conf=0.70)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Health Measurement Scales: A practical guide to their development and use
Streiner, D. L., Norman, G. R., & Cairney, J. (2014) · doi:10.1093/med/9780199685219.001.0001 · Cited by 2,564 · see citing works
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · see citing works
Carlisle, J. B. (2021) · Anaesthesia · doi:10.1111/anae.15263 · Cited by 173 · cited 2× · see citing works
“Yet, some trustworthiness issues may only be detected when IPD are available, as evidenced by a 20-fold increase in detection of false trial data in studies submitted to Anaesthesia , 8 and for several COVID-19 ivermectin studies.”
A checklist to assess Trustworthiness in RAndomised Controlled Trials (TRACT checklist)
Mol, B. W., Lai, S., Rahim, A., Bordewijk, E. M., Wang, R., Eekelen, R. V., Gurrin, L. C., Thornton, J. G., Wely, M. V., & Li, W. (2023) · doi:10.21203/rs.3.rs-2119486/v1 · Cited by 149 · see citing works
Self-citationMol, B. W., Lai, S., Rahim, A., Bordewijk, E. M., Wang, R., van Eekelen, R., Gurrin, L. C., Thornton, J. G., van Wely, M., & Li, W. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00130-8 · Cited by 141 · cited 8× · see citing works
“19”
InfluentialSelf-citationCheck for publication integrity before misconduct
Grey, A., Bolland, M. J., Avenell, A., Klein, A. A., & Gunsalus, C. K. (2020) · Nature · doi:10.1038/d41586-019-03959-6 · Cited by 114 · cited 1× · see citing works
“18,20,42,48 ,42,48 This is particularly important for prognostic factors which may influence outcomes.”
Systematic review and statistical analysis of the integrity of 33 randomized controlled trials
Bolland, M. J., Avenell, A., Gamble, G. D., & Grey, A. (2016) · Neurology · doi:10.1212/wnl.0000000000003387 · Cited by 111 · cited 2× · see citing works
“18,20,42,48 ,42,48 This is particularly important for prognostic factors which may influence outcomes.”
Weibel, S., Popp, M., Reis, S., Skoetz, N., Garner, P., & Sydenham, E. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1599 · Cited by 93 · cited 2× · see citing works
“15 However, we recommend a more nuanced approach, considering the timing of retrospective registration (e.g., during vs. after study completion) and barriers to prospective registration (e.g., processing delays by the registry such as during COVID-19).”
The lesson of ivermectin: meta-analyses based on summary data alone are inherently unreliable
Lawrence, J. M., Meyerowitz-Katz, G., Heathers, J. A. J., Brown, N. J. L., & Sheldrick, K. A. (2021) · Nature Medicine · doi:10.1038/s41591-021-01535-y · Cited by 88 · see citing works
Ivermectin for COVID-19: Addressing Potential Bias and Medical Fraud
Hill, A., Mirchandani, M., & Pilkington, V. (2022) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofab645 · Cited by 57 · see citing works
Heathers, J. A., Anaya, J., van der Zee, T., & Brown, N. J. (2018) · PeerJ Preprints · doi:10.7287/peerj.preprints.26968v1 · Cited by 44 · see citing works
Assessing Research Misconduct in Randomized Controlled Trials
Li, W., Bordewijk, E. M., & Mol, B. W. (2021) · Obstetrics & Gynecology · doi:10.1097/aog.0000000000004513 · Cited by 37 · see citing works
Self-citationWhen beauty is but skin deep: dealing with problematic studies in systematic reviews
Boughton, S. L., Wilkinson, J., & Bero, L. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000152 · Cited by 32 · see citing works
Bordewijk, E. M., Wang, R., van Wely, M., Costello, M. F., Norman, R. J., Teede, H., Gurrin, L. C., Mol, B. W., & Li, W. (2020) · Human Reproduction Update · doi:10.1093/humupd/dmaa031 · Cited by 18 · see citing works
Self-citationDetection of data fabrication using statistical tools
Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · doi:10.31234/osf.io/jkws4 · Cited by 16 · see citing works
Stamp out fake clinical data by working together.
Bero, L. (2022) · Nature · doi:10.1038/d41586-022-00025-6 · Cited by 15 · see citing works
Temporal Trends, Characteristics, and Citations of Retracted Articles in Cardiovascular Medicine
Wadhwa, R. R., Rasendran, C., Popovic, Z. B., Nissen, S. E., & Desai, M. Y. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.18263 · Cited by 14 · see citing works
Hunter, K., Aberoumand, M., Libesman, S., Sotiropoulos, J., Williams, J., Li, W., Aagerup, J., Mol, B., Wang, R., Barba, A., et al. (2023) · doi:10.1101/2023.12.11.23299797 · Cited by 9 · cited 1× · see citing works
“32 Existing checks have been criticised for being too time-consuming 16 and for requiring too extensive statistical expertise to complete the checks.”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 33
- Pantziarka P, Meheus L. Journal retractions in oncology: a bib- liometric study. Future Oncol. 2019;15(31):3597-3608.
- Audisio K, Robinson NB, Soletti GJ, et al. A survey of retrac- tions in the cardiovascular literature. Int J Cardiol. 2022;349: 109-114.
- Bennett C, Chambers LM, Al-Hafez L, et al. Retracted articles in the obstetrics literature: lessons from the past to change the future. Am J Obstet Gynecol MFM. 2020;2(4):100201.
- Li W, Gurrin LC, Mol BW. Violation of research integrity prin- ciples occurs more often than we think. Reprod Biomed Online. 2022;44(2):207-209.
- Van Scoy LJ, Duda SH, Scott AM, et al. A mixed methods study exploring requests for unproven COVID therapies such as iver- mectin and healthcare distrust in the rural south. Prev Med Rep. 2023;31:102104.
- Bausell RB. Questionable research practices (QRPs) and their devastating scientific effects. In: Bausell RB, ed. The Problem with Science: The Reproducibility Crisis and What to Do About It. Oxford University Press; 2021.
- Parker L, Boughton S, Lawrence R, Bero L. Experts identified warning signs of fraudulent research: a qualitative study to inform a screening tool. J Clin Epidemiol. 2022;151:1-17.
- Wilkinson J, Heal C, Antoniou GA, et al. Protocol for the devel- opment of a tool (INSPECT-SR) to identify problematic ran- domised controlled trials in systematic reviews of health interventions. medRxiv; 2023.
- Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.
- Seidler AL, Duley L, Katheria AC, et al. Systematic review and network meta-analysis with individual participant data on cord management at preterm birth (iCOMP): study protocol. BMJ Open. 2020;10(3):e034595.
- Seidler AL, Aberoumand M, Hunter KE, et al. Deferred cord clamping, cord milking, and immediate cord clamping at pre- term birth: a systematic review and individual participant data meta-analysis. Lancet. 2023;402(10418):2209-2222.
- Seidler AL, Libesman S, Hunter KE, et al. Short, medium, and long deferral of umbilical cord clamping compared with umbil- ical cord milking and immediate clamping at preterm birth: a systematic review and network meta-analysis with individual participant data. Lancet. 2023;402(10418):2223-2234.
- Sotiropoulos JX, Oei JL, Schmölzer GM, et al. NETwork meta- analysis of trials of initial oxygen in preterm newborns (NETMOTION): a protocol for systematic review and individual participant data network meta-analysis of preterm infants <32 weeks' gestation randomized to initial oxygen concentration for resuscitation. Neonatology. 2022;119(4):517-524.
- Boudry C, Howard K, Mouriaux F. Poor visibility of retracted articles: a problem that should no longer be ignored. BMJ. 2023;381:e072929.
- World Medical Association. World Medical Association Decla- ration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191-2194.
- De Angelis C, Drazen JM, Frizelle FA, et al. Clinical trial regis- tration: a statement from the International Committee of Medi- cal Journal Editors. N Engl J Med. 2004;351(12):1250-1251.
- Dickersin K, Rennie D. The evolution of trial registries and their use to assess the clinical trial Enterprise. JAMA. 2012; 307(17):1861-1864.
- Hardwicke TE, Wagenmakers E-J. Reducing bias, increasing transparency and calibrating confidence with preregistration. Nat Hum Behav. 2023;7(1):15-26.
- Serghiou S, Axfors C, Ioannidis JPA. Lessons learnt from regis- tration of biomedical research. Nat Hum Behav. 2023;7(1):9-12.
- Mosimann JE, Wiseman CV, Edelman RE. Data fabrication: can people generate random digits? Account Res. 1995;4(1): 31-55.
- Evans SJW. Statistical aspects of the detection of fraud. In: Lock S, Wells F, eds. Fraud and Misconduct in Biomedical Research. 2nd ed. BMJ Publishing Group; 1996:226-239.
- Dahlberg JE, Davidian NM. Scientific forensics: how the office of research integrity can assist institutional investigations of research misconduct during oversight review. Sci Eng Ethics. 2010;16(4):713-735.
- Alfirevic Z. Retracted papers are only the tip of the iceberg of untrustworthy evidence. Am J Obstet Gynecol MFM. 2020;2(4): 100223.
- Pandit JJ. On statistical methods to test if sampling in trials is genuinely random. Anaesthesia. 2012;67(5):456-462.
- Wald A, Wolfowitz J. On a test whether two samples are from the same population. Ann Math Stat. 1940;11(2):147-162.
- van den Bor RM, Vaessen PWJ, Oosterman BJ, Zuithoff NPA, Grobbee DE, Roes KCB. A computationally simple central monitoring procedure, effectively applied to empirical trial data with known fraud. J Clin Epidemiol. 2017;87:59-69.
- Kirkwood AA, Cox T, Hackshaw A. Application of methods for central statistical monitoring in clinical trials. Clin Trials. 2013; 10(5):783-806.
- Stewart LA, Cochrane Working Group on Meta-Analysis Using Individual Patient Data. Practical methodology of meta- analyses (overviews) using updated individual patient data. Stat Med. 1995;14(19):2057-2079.
- Buyse M, George SL, Evans S, et al. The role of biostatistics in the prevention, detection and treatment of fraud in clinical tri- als. Stat Med. 1999;18(24):3435-3451.
- Bordewijk EM, Li W, van Eekelen R, et al. Methods to assess research misconduct in health-related research: a scoping review. J Clin Epidemiol. 2021;136:189-202.
- Cochrane. Cochrane policy for managing potentially problem- atic studies; 2021. Accessed February 3, 2023. https://www. cochranelibrary.com/cdsr/editorial-policies#problematic-studies
- Committee on Publication Ethics. Code of Conduct and Best Practice Guidelines for Journal Editors. COPE; 2011.
- Williams ACC, Hearn L, Moore RA, et al. Effective quality con- trol in the medical literature: investigation and retraction vs inaction. J Clin Epidemiol. 2023;157:156-157.
Cited by
Sorted by citations
Wilkinson, J., Heal, C., Antoniou, G. A., Flemyng, E., Ahnström, L., Alteri, A., Avenell, A., Barker, T. H., Borg, D. N., Brown, N. J. L., et al. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111824 · Cited by 28 · see citing works
Self-citationHow should we assess trustworthiness of randomized controlled trials?
Wilkinson, J., & Tovey, D. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111670 · Cited by 8 · see citing works
- Authors
- Hunter, K. E., Aberoumand, M., Libesman, S., Sotiropoulos, J. X., Williams, J. G., Aagerup, J., Wang, R., Mol, B. W., Li, W., Barba, A., Shrestha, N., Webster, A. C., & Seidler, A. L.
- Affiliations
- The University of Sydney (AU)
- Monash University (AU)
- Corresponding author
- Hunter, Kylie E.
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2024-08-13
- Volume
- 15
- Issue
- 6
- Pages
- 917-939
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W4401526352
- Cited by
- 28
- References
- 18
- Citations by year
- 2026: 16 · 2025: 11 · 2024: 1
- FWCI
- 8.10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Research Synthesis Methods (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- University of Sydney
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- individual patient data, research integrity, randomized controlled trials, data inconsistency, automated screening, evidence synthesis, research trustworthiness
- MeSH terms
- Data Accuracy, Data Interpretation, Statistical, Decision Making, Humans, Research Design, Reproducibility of Results, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
