CoChat
Sign Up
Cited by 1,160Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsClinical Reasoning and Diagnostic Skills

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.

Assessing the Generalizability of Prognostic Information

Justice, Amy C., Covinsky, Kenneth E., Berlin, Jesse A.·1999·Annals of Internal Medicine
DOI 10.7326/0003-4819-130-6-199903160-00016
Open this paper in CoChat — free account

Abstract

Physicians are often asked to make prognostic assessments but often worry that their assessments will prove inaccurate. Prognostic systems were developed to enhance the accuracy of such assessments. This paper describes an approach for evaluating prognostic systems based on the accuracy (calibration and discrimination) and generalizability (reproducibility and transportability) of the system's predictions. Reproducibility is the ability to produce accurate predictions among patients not included in the development of the system but from the same population. Transportability is the ability to produce accurate predictions among patients drawn from a different but plausibly related population. On the basis of the observation that the generalizability of a prognostic system is commonly limited to a single historical period, geographic location, methodologic approach, disease spectrum, or follow-up interval, we describe a working hierarchy of the cumulative generalizability of prognostic systems. This approach is illustrated in a structured review of the Dukes and Jass staging systems for colon and rectal cancer and applied to a young man with colon cancer. Because it treats the development of the system as a "black box" and evaluates only the performance of the predictions, the approach can be applied to any system that generates predicted probabilities. Although the Dukes and Jass staging systems are discrete, the approach can also be applied to systems that generate continuous predictions and, with some modification, to systems that predict over multiple time periods. Like any scientific hypothesis, the generalizability of a prognostic system is established by being tested and being found accurate across increasingly diverse settings. The more numerous and diverse the settings in which the system is tested and found accurate, the more likely it will generalize to an untested setting.

Overview

Integrity signals

Registry

Crossref · 10.7326/0003-4819-130-6-199903160-00016

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 7

  1. Liabakk NB, 1996, Cancer Res, 56, 190
  2. Vecchio FM, 1995, Rays, 20, 15
  3. Deans GT, 1994, J Am Coll Surg, 179, 11
  4. Fucci L, 1994, Pathologica, 86, 371
  5. Secco GB, 1990, Minerva Chir, 45, 1347
  6. Hart A, 1990, Med Inf (Lond), 15, 229
  7. Brewster AC, 1985, Inquiry, 22, 377

Cited by

Sorted by citations

See all 1,160

Metadata

Authors
Justice, A. C., Covinsky, K. E., & Berlin, J. A.
Affiliations
San Francisco VA Medical Center (US)
Corresponding author
Justice, Amy C.
ORCID
Justice, Amy C.
Venue
Annals of Internal Medicine
Publisher
American College of Physicians
Published
1999-03-16
Volume
130
Issue
6
Pages
515-524
ISSN
  • 0003-4819 (ISSN-L)
  • 1539-3704
DOI
10.7326/0003-4819-130-6-199903160-00016
OpenAlex
W2089463633
Cited by
1,160
References
59
Citations by year
2026: 19 · 2025: 20 · 2024: 46 · 2023: 56 · 2022: 63 · 2021: 49 · 2020: 48 · 2019: 51 · 2018: 44 · 2017: 60 · 2016: 47 · 2015: 67 · 2014: 55 · 2013: 44 · 2012: 47
FWCI
24.34
Type
journal article
Language
en
Open access
closed
Licence
www.acpjournals.org/journal/aim/text-and-data-mining (tdm)
Locations
  • Annals of Internal Medicine (publishedVersion)
  • PubMed (publishedVersion)
Funders
Agency for Healthcare Research and Quality (5 T32 HS00009)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
prognostic systems, generalizability, reproducibility, transportability, calibration, discrimination (statistics), colon cancer, rectal cancer, Dukes staging, cancer staging systems, prediction accuracy, black-box evaluation
MeSH terms
Calibration, Colonic Neoplasms, Evaluation Studies as Topic, Follow-Up Studies, Humans, Male, Middle Aged, Neoplasm Staging, Prognosis, Rectal Neoplasms, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review