Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Objectives: Patient-reported outcomes (PROs) provide important information about the impact of treatment from the patients' perspective. However, missing PRO data may compromise the interpretability and value of the findings. We aimed to report: (1) a non-technical summary of problems caused by missing PRO data; and (2) a systematic review by collating strategies to: (A) minimise rates of missing PRO data, and (B) facilitate transparent interpretation and reporting of missing PRO data in clinical research. Our systematic review does not address statistical handling of missing PRO data. Data sources: MEDLINE and Cumulative Index to Nursing and Allied Health Literature (CINAHL) databases (inception to 31 March 2015), and citing articles and reference lists from relevant sources. Eligibility criteria: English articles providing recommendations for reducing missing PRO data rates, or strategies to facilitate transparent interpretation and reporting of missing PRO data were included. Methods: 2 reviewers independently screened articles against eligibility criteria. Discrepancies were resolved with the research team. Recommendations were extracted and coded according to framework synthesis. Results: 117 sources (55% discussion papers, 26% original research) met the eligibility criteria. Design and methodological strategies for reducing rates of missing PRO data included: incorporating PRO-specific information into the protocol; carefully designing PRO assessment schedules and defining termination rules; minimising patient burden; appointing a PRO coordinator; PRO-specific training for staff; ensuring PRO studies are adequately resourced; and continuous quality assurance. Strategies for transparent interpretation and reporting of missing PRO data include utilising auxiliary data to inform analysis; transparently reporting baseline PRO scores, rates and reasons for missing data; and methods for handling missing PRO data. Conclusions: The instance of missing PRO data and its potential to bias clinical research can be minimised by implementing thoughtful design, rigorous methodology and transparent reporting strategies. All members of the research team have a responsibility in implementing such strategies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The lack of reported calibration protocols could potentially affect the reliability and comparability of findings across studies, particularly for subjective measures such as esthetic outcomes and patient satisfaction measures (Mercieca‐Bebber et al. 2016).”
From Patient- and Clinician-Reported Outcomes and Outcome Measures Evaluating Timing of Implant Loading in the Edentulous Maxilla: A Systematic Review of Prospective Studies (2026) · cites this paper as Mercieca‐Bebber et al. 2016
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-117 · Cited by 10,865
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
U.S. Department of Health and Human Services FDA Center for Drug Evaluation and Research, U.S. Department of Health and Human Services FDA Center for Biologics Evaluation and Research, & U.S. Department of Health and Human Services FDA Center for Devices and Radiological Health (2006) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-4-79 · Cited by 2,856
Principled missing data methods for researchers
Dong, Y., & Peng, C.-Y. J. (2013) · SpringerPlus · doi:10.1186/2193-1801-2-222 · Cited by 2,208 · cited 2×
“Therefore, assuming the study is adequately powered, the results should not be altered too much if the MCAR are ignored in analysis; however, the standard error of the estimates will be inflated.(19) Many examples of MCAR are caused by poor study design and implementation, and are hence ‘preventable’ sources of missin…”
The prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884
Direction and impact of language bias in meta-analyses of controlled trials: empirical study
Jüni, P., Holenstein, F., Sterne, J., Bartlett, C., & Egger, M. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.115 · Cited by 892
Cherny, N. I., Sullivan, R., Dafni, U., Kerst, J. M., Sobrero, A., Zielinski, C., de Vries, E. G. E., & Piccart, M. J. (2015) · Annals of Oncology · doi:10.1093/annonc/mdv249 · Cited by 818
What contributions do languages other than English make on the results of meta-analyses?
Moher, D., Pham, Klassen, T. P., Schulz, K. F., Berlin, J. A., Jadad, A. R., & Liberati, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00188-8 · Cited by 453
Handling missing data in RCTs; a review of the top medical journals
Bell, M. L., Fiero, M., Horton, N. J., & Hsu, C.-H. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-118 · Cited by 395 · cited 1×
“Third, the presence of missing data undermines randomisation, and makes intention-to-treat analyses (analysing according to randomised groups) less valid as missing data create a need to make assumptions about the data that are not always verifiable.(14)”
Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards
Brundage, M., Blazeby, J., Revicki, D., Bass, B., de Vet, H., Duffy, H., Efficace, F., King, M., Lam, C. L. K., Moher, D., et al. (2013) · Quality of Life Research · doi:10.1007/s11136-012-0252-1 · Cited by 207
Self-citation
Research works printed but not linked: 55
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<517::AID-SIM799>3.0.CO;2-S · DOI · Paper page
- 10.1007/BF00540022 · DOI · Paper page
- Bernhard, 1998, Preface, Stat Med, 17, 511, 10.1002/(SICI)1097-0258(19980315/15)17:5/7<511::AID-SIM798>3.0.CO;2-C · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<511::aid-sim798>3.0.co;2-c
- 10.1093/annonc/mdu489 · DOI
- 10.1002/cncr.29489 · DOI
- 10.1186/1745-6215-9-51 · DOI · Paper page
- 10.1093/annonc/mdw080 · DOI
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<781::AID-SIM821>3.0.CO;2-O · DOI · Paper page
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<667::AID-SIM813>3.0.CO;2-6 · DOI · Paper page
- 10.1177/0962280213476378 · DOI · Paper page
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<739::AID-SIM818>3.0.CO;2-M · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<739::aid-sim818>3.0.co;2-m
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<697::AID-SIM815>3.0.CO;2-Y · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<697::aid-sim815>3.0.co;2-y
- 10.1186/1477-7525-7-57 · DOI
- 10.1371/journal.pmed.1000097 · DOI · Paper page
- 10.1186/1471-2288-9-59 · DOI · Paper page
- 10.1200/JCO.2007.12.1749 · DOI
- Cella, 1994, Data collection strategies for patient-reported information, Qual Manag Health Care, 2, 28, 10.1097/00019514-199402040-00006 · DOI
- Fallowfield, 2005, Health-related quality of life in patients undergoing drug therapy for advanced non-small-cell lung cancer, Lung Cancer, 48, 365, 10.1016/j.lungcan.2004.11.018 · DOI
- 10.1002/sim.5519 · DOI · Paper page
- 10.1023/A:1023900802254 · DOI
- 10.1634/theoncologist.2009-0291 · DOI · Paper page
- Aaronson, 1990, Quality of life research in cancer clinical trials: a need for common rules and language, Oncology (Williston Park, NY), 4, 59
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<541::AID-SIM801>3.0.CO;2-# · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<541::aid-sim801>3.0.co;2-#
- 10.1177/1740774512449532 · DOI
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<633::AID-SIM810>3.0.CO;2-U · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<633::aid-sim810>3.0.co;2-u
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<613::AID-SIM808>3.0.CO;2-9 · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<613::aid-sim808>3.0.co;2-9
- 10.1016/j.biopsych.2006.01.017 · DOI
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<587::AID-SIM806>3.0.CO;2-# · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<587::aid-sim806>3.0.co;2-#
- 10.1002/(SICI)1097-0258(19980315/15)17:5/7<547::AID-SIM802>3.0.CO;2-F · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<547::aid-sim802>3.0.co;2-f
- 10.1007/BF00343916 · DOI
- 10.1016/S0197-2456(00)00130-6 · DOI
- 10.1002/(SICI)1099-1611(199905/06)8:3<260::AID-PON383>3.0.CO;2-Z · doi:10.1002/(sici)1099-1611(199905/06)8:3<260::aid-pon383>3.0.co;2-z
- 10.1136/jech.55.5.348 · DOI
- Revicki D , Fairclough D . Preventing missing data. In: Fayers P , Hays R, eds . Assessing quality of life in clinical trials: analysis and interpretation. 2nd edn. Oxford University Press, 2005:209–222. · doi:10.1093/oso/9780198527695.003.0011
- 10.1191/1740774504cn008xx · doi:10.1191/1740774504cn008xx
- Hayden, 1993, Pitfalls in quality-of-life assessment: lessons from a Southwest Oncology Group breast cancer clinical trial, Oncol Nurs Forum, 20, 1415
- 10.1007/s11136-014-0871-9 · DOI
- 10.1001/jama.2013.879 · DOI · Paper page
- 10.1007/s11136-010-9793-3 · DOI
- Mercieca-Bebber, 2016, Patient-reported outcomes in head and neck and thyroid cancer randomised controlled trials: a systematic review of completeness of reporting and impact on interpretation, Eur J Cancer, 56, 144, 10.1016/j.ejca.2015.12.025 · DOI
- 10.1093/annonc/mdv283 · DOI
- 10.1016/S0140-6736(13)62228-X · DOI · Paper page
- Palmer, 2015, A framework for classifying root causes of missing data for patient reported outcomes in prospective studies, Qual Life Res, 24, a2016
- Fairclough D . Design and analysis of quality of life studies in clinical trials. New York: Chapman & Hall, 2010. · DOI · Paper page
- 10.1586/erp.11.9 · DOI · Paper page
- 10.1097/00005650-199905000-00006 · DOI · Paper page
- 10.1371/journal.pone.0110216 · DOI · Paper page
- Calvert, 2015, Putting patients at the heart of health-care research, Lancet, 385, 1073, 10.1016/S0140-6736(15)60599-2 · DOI
- 10.1081/CNV-120005928 · DOI · Paper page
- Gheorghe, 2015, Evaluating the generalisability of trial results: introducing a centre- and trial-level generalisability index, Pharmacoeconomics, 33, 1195, 10.1007/s40273-015-0298-3 · DOI
- Rutherford, 2015, Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis, Qual Life Res, 3, 3
- 10.1111/j.1365-2648.2007.04448.x · DOI · Paper page
- Edwards P , Roberts I , Clarke M , et al . Methods to increase response rates to postal questionnaires. Cochrane Database Syst Rev 2007;(2):MR000008. doi:10.1002/14651858.MR000008.pub3 · DOI · Paper page
- 10.1111/j.1524-4733.2007.00278.x · DOI
- 10.1016/j.ejca.2004.10.017 · DOI · Paper page
Cited by
Calvert, M., Kyte, D., Mercieca-Bebber, R., Slade, A., Chan, A.-W., King, M. T., and the SPIRIT-PRO Group, Hunn, A., Bottomley, A., Regnault, A., et al. (2018) · JAMA · doi:10.1001/jama.2017.21903 · Cited by 814
“Inappropriate method selection may lead to potentially biased and misleading results.(22,32) The protocol should ac-”
BackgroundMethodInfluentialSelf-citationMercieca-Bebber, R., King, M. T., Calvert, M. J., Stockler, M. R., & Friedlander, M. (2018) · Patient Related Outcome Measures · doi:10.2147/prom.s156279 · Cited by 659
“Often the missing data mechanism has a stronger impact on the results than does the proportion of missing data.(52,56) For example, if analysis methods assume that missing data are not related to the patients’ illness (eg, as with “complete case” analysis methods), when in fact the illness may have led to the data not…”
Self-citationCoens, C., Pe, M., Dueck, A. C., Sloan, J., Basch, E., Calvert, M., Campbell, A., Cleeland, C., Cocks, K., Collette, L., et al. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30790-9 · Cited by 323
Self-citationCalvert, M., King, M., Mercieca-Bebber, R., Aiyegbusi, O., Kyte, D., Slade, A., Chan, A.-W., Basch, E., Bell, J., Bennett, A., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045105 · Cited by 144
Self-citationPannuti, C. M., Hassan, M. A., Reis, I. N. R., Villar, C. C., Francisco, H., & Romito, G. A. (2026) · Clinical Oral Implants Research · doi:10.1111/clr.14451 · Cited by 1
Metadata
- Authors
- Mercieca-Bebber, R., Palmer, M. J., Brundage, M., Calvert, M., Stockler, M. R., & King, M. T.
- ORCID
- Calvert, Melanie
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2016-06
- Volume
- 6
- Issue
- 6
- Pages
- e010938
- ISSN
- 2044-6055
- PMC
- PMC4916640
- Cited by
- 121
- References
- 11
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review. https://app.cochat.ai/papers/10.1136/bmjopen-2015-010938
- The paper
- Mercieca-Bebber, R., Palmer, M. J., Brundage, M., Calvert, M., Stockler, M. R., & King, M. T. (2016). Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review. BMJ Open. https://doi.org/10.1136/bmjopen-2015-010938
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
