CoChat
Sign Up
PreprintCited by 9Academic integrity and plagiarismMeta-analysis and systematic reviewsBenford’s Law and Fraud Detection

Development of the Individual Participant Data (IPD) Integrity Tool for assessing the integrity of randomised trials using individual participant data

Open access
Hunter, KE, Aberoumand, M, Libesman, S, Sotiropoulos, JX, Williams, J, Li, W, Aagerup, J, Mol, BW +5 more·2023·Research Synthesis Methods
DOI 10.1101/2023.12.11.23299797

Preprint — published as 10.1002/jrsm.1739

Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction: Increasing concerns about integrity in medical research have prompted the development of tools to detect untrustworthy studies. Existing tools focus on evaluating aggregate or published data, though some trustworthiness issues may only be detected upon scrutiny of individual participant data (IPD). To address this, we developed the IPD Integrity Tool for detecting integrity issues in randomised controlled trials with IPD available. This manuscript describes the development of this tool. Methods: We conducted a literature review to collate and map existing integrity items. These were discussed with an expert advisory group, and agreed items were included in a standardised tool and automated where possible. We piloted this tool in two IPD meta-analyses, and conducted preliminary validation checks on 13 datasets with and without known integrity issues in a blinded manner. Results: The literature review identified 120 integrity items: 54 could be conducted at the publication or aggregate data (AD) level, 48 required IPD, and 18 were possible with aggregate data, but more comprehensive with IPD. Based on these items, an initial reduced tool was developed in a consensus process involving 13 advisors with different backgrounds (countries, profession, education). This initial tool included 11 items across four domains for AD, and 12 items across 8 domains requiring IPD. The tool was iteratively refined throughout piloting on two IPD meta-analyses including a total of 116 trials (73 with IPD, and 43 with only AD available), and preliminary validation using an additional 13 datasets. All five studies with known integrity issues were accurately identified during validation. The final version of the tool included seven domains with 13 items for AD and eight domains with 18 items requiring IPD. Conclusions: The quality of evidence informing health care relies on trustworthy data. This manuscript describes the development of a tool to enable researchers, editors, and other stakeholders to detect integrity issues in randomised trials using IPD. Detailed instructions on the application of this tool will be published subsequently.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.12.11.23299797

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A variety of trustworthiness tools have recently been proposed, differing in development, content and structure, and the appropriateness of some trustworthiness checks included in some of these tools is unclear.

    “A variety of trustworthiness tools have recently been proposed (11–15), although they differ in terms of their development, content and structure, and the appropriateness of some of the trustworthiness checks that have been included in some of these tools is unclear (16).”

    From INSPECT-SR: a tool for assessing trustworthiness of randomised controlled trials (2025) · cites this paper as [15]

    The ‘What is research integrity?’ section describes several proposed tools and approaches, supporting the background about their variety. ‘Remaining gaps’ says guidance on applying integrity checks is lacking, and ‘Limitations’ says tool validation has no agreed gold standard. Neither passage specifically questions the appropriateness of checks included in other tools, which is the point attached to this citation. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Steneck NHP. Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics 2006;12:53-74.
  2. Research integrity is much more than misconduct. Nature 2019;570:5.
  3. Cochrane. Cochrane policy for managing potentially problematic studies London UK: Cochrane. Available at https://www.cochranelibrary.com/cdsr/editorial-policies#problematic- studies (2021), Accessed 3rd Feb 2023.
  4. Seidler AL, Libesman S, Hunter KE, et al. Short, medium, and long deferral of umbilical cord clamping compared with umbilical cord milking and immediate clamping at preterm birth: a systematic review and network meta-analysis with individual participant data. Lancet 2023.
  5. Committee on Publication Ethics. Code of Conduct and Best Practice Guidelines for Journal Editors. 2011;Harleston, UK: COPE.

Cited by

Sorted by citations

Related

Authors
Hunter, K., Aberoumand, M., Libesman, S., Sotiropoulos, J., Williams, J., Li, W., Aagerup, J., Mol, B., Wang, R., Barba, A., Shrestha, N., Webster, A., & Seidler, A.
Affiliations
  • The University of Sydney (AU)
  • Monash University (AU)
ORCID
Venue
Research Synthesis Methods
Publisher
openRxiv
Published
2023-12-13
Volume
15
Issue
6
Pages
940-949
ISSN
1759-2879 (ISSN-L)
DOI
10.1101/2023.12.11.23299797
OpenAlex
W4401711178
Cited by
9
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 3
FWCI
3.72
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Research Synthesis Methods (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion)
  • PubMed (publishedVersion)
Funders
University of Sydney
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
Keywords
research integrity, individual patient data, randomised trials, integrity assessment, meta-analysis, tool development, expert consensus
MeSH terms
Data Accuracy, Data Interpretation, Statistical, Humans, Research Design, Meta-Analysis as Topic, Reproducibility of Results, Randomized Controlled Trials as Topic, Consensus, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review