Cited by 749Meta-analysis and systematic reviewsStatistical Methods in Clinical TrialsAdvanced Causal Inference Techniques

Development of the instrument to assess the credibility of effect modification analyses (ICEMAN) in randomized controlled trials and meta-analyses

Open access
Schandelmaier, Stefan, Briel, Matthias, Varadhan, Ravi, Schmid, Christopher H., Devasenapathy, Niveditha, Hayward, Rodney A., Gagnier, Joel, Borenstein, Michael +8 more·2020·Canadian Medical Association Journal

Abstract

BACKGROUND: Most randomized controlled trials (RCTs) and meta-analyses of RCTs examine effect modification (also called a subgroup effect or interaction), in which the effect of an intervention varies by another variable (e.g., age or disease severity). Assessing the credibility of an apparent effect modification presents challenges; therefore, we developed the Instrument for assessing the Credibility of Effect Modification Analyses (ICEMAN). METHODS: To develop ICEMAN, we established a detailed concept; identified candidate credibility considerations in a systematic survey of the literature; together with experts, performed a consensus study to identify key considerations and develop them into instrument items; and refined the instrument based on feedback from trial investigators, systematic review authors and journal editors, who applied drafts of ICEMAN to published claims of effect modification. RESULTS: The final instrument consists of a set of preliminary considerations, core questions (5 for RCTs, 8 for meta-analyses) with 4 response options, 1 optional item for additional considerations and a rating of credibility on a visual analogue scale ranging from very low to high. An accompanying manual provides rationales, detailed instructions and examples from the literature. Seventeen potential users tested ICEMAN; their suggestions improved the user-friendliness of the instrument. INTERPRETATION: The Instrument for assessing the Credibility of Effect Modification Analyses offers explicit guidance for investigators, systematic reviewers, journal editors and others considering making a claim of effect modification or interpreting a claim made by others.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1503/cmaj.200077

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • We used the ICEMAN tool to determine whether subgroup differences were credible.

    “The Instrument to Assess the Credibility of Effect Modification Analyses (ICEMAN)21 recently introduced 8 criteria that we used to determine whether or not subgroup differences were credible.”

    From Aerobic Exercise and Weight Loss in Adults (2024) · cites this paper as [21]

    The Results say ICEMAN can be used to evaluate the credibility of effect-modification claims, supporting its use to assess subgroup differences. They also specify 8 core questions for meta-analyses but only 5 for individual RCTs; the citing sentence omits that qualification. (LLM, conf=0.85)

    Experimental — accuracy not yet measured
  • Up-to-date meta-analyses evaluating the effects of therapeutic interventions across clinically important subgroup populations were assessed for credibility using relevant guidance.

    “Up-to-date meta-analyses evaluating the effects of therapeutic interventions across clinically important subgroup populations were assessed in terms of their credibility using relevant guidance (7,8).”
  • The credibility of statistically significant subgroup effects was assessed using the Instrument to assess the Credibility of Effect Modification Analyse (ICEMAN) criteria.

    “We assessed the credibility of statistically significant subgroup effects (P value for test of interaction <0.05) using Instrument to assess the Credibility of Effect Modification Analyse (ICEMAN) criteria.43”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Schandelmaier, S., Briel, M., Varadhan, R., Schmid, C. H., Devasenapathy, N., Hayward, R. A., Gagnier, J., Borenstein, M., van der Heijden, G. J. M. G., Dahabreh, I. J., Sun, X., Sauerbrei, W., Walsh, M., Ioannidis, J. P. A., Thabane, L., & Guyatt, G. H.
Affiliations
Johns Hopkins University (US)
Venue
Canadian Medical Association Journal
Published
2020-08-09
Volume
192
Issue
32
Pages
E901-E906
ISSN
  • 0820-3946 (ISSN-L)
  • 1488-2329
OpenAlex
W3047950129
Cited by
749
References
50
Citations by year
2026: 151 · 2025: 144 · 2024: 177 · 2023: 130 · 2022: 98 · 2021: 36 · 2020: 12
FWCI
41.92
Type
journal article
Language
en
Open access
Locations
  • Canadian Medical Association Journal (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • VU Research Portal (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Funders
  • Brown University
  • Hamilton Health Sciences
  • Freiwillige Akademische Gesellschaft
  • McMaster University
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
  • Universiteit van Amsterdam
  • Vrije Universiteit Amsterdam
  • Albert-Ludwigs-Universität Freiburg
  • Sichuan University
  • Gottfried und Julia Bangerter-Rhyner-Stiftung
  • West China Hospital, Sichuan University
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
effect modification, subgroup analysis, interaction, credibility assessment, Iceman, instrument development, randomized controlled trials, meta-analysis, reporting quality, visual analogue scale, expert consensus
MeSH terms
Humans, Research Design, Meta-Analysis as Topic, Randomized Controlled Trials as Topic, Consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review