Bias Modelling in Evidence Synthesis
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Summary: Policy decisions often require synthesis of evidence from multiple sources, and the source studies typically vary in rigour and in relevance to the target question. We present simple methods of allowing for differences in rigour (or lack of internal bias) and relevance (or lack of external bias) in evidence synthesis. The methods are developed in the context of reanalysing a UK National Institute for Clinical Excellence technology appraisal in antenatal care, which includes eight comparative studies. Many were historically controlled, only one was a randomized trial and doses, populations and outcomes varied between studies and differed from the target UK setting. Using elicited opinion, we construct prior distributions to represent the biases in each study and perform a bias-adjusted meta-analysis. Adjustment had the effect of shifting the combined estimate away from the null by approximately 10%, and the variance of the combined estimate was almost tripled. Our generic bias modelling approach allows decisions to be based on all available evidence, with less rigorous or less relevant studies downweighted by using computationally simple methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Self-citationCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · cited 3×
“, 2003), or looking at the effect of inadequate allocation concealment and inadequate allocation sequence generation (Schulz et al., 1995; Moher et al., 1998; Kjaergard et al., 2001; Juni et al., 2001; Balk et al., 2002).”
BackgroundMethodMoher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 1×
“…effect, through comparing randomized and non-randomized studies (Deeks et al., 2003), or looking at the effect of inadequate allocation concealment and inadequate allocation sequence generation (Schulz et al., 1995; Moher et al., 1998; Kjaergard et al., 2001; Juni et al., 2001; Balk et al., 2002).”
BackgroundEvaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · cited 3×
“It is clear that differences between populations will cause both internal and external bias with respect to the target question, but unclear how to determine the extent of each bias.”
BackgroundMethodGuidelines for evaluating and expressing the uncertainty of NIST measurement results
Taylor, B. N. (1994) · National Institute of Standards and Technology, NIST Technical Note 1297 · doi:10.6028/nist.tn.1297 · Cited by 2,833
Guidelines for evaluating and expressing the uncertainty of NIST measurement results
Taylor, B. N., & Kuyatt, C. E. (2018) · doi:10.6028/nist.tn.1297-1993 · Cited by 2,582
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Kjaergard, L. L., Villumsen, J., & Gluud, C. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,283
Statistical Methods in Medical Research
Armitage, P., Berry, G., & Matthews, J. N. S. (2002) · Wiley eBooks · doi:10.1002/9780470773666 · Cited by 1,630
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Bias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636
Publication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568 · cited 2×
“Alternatively, the models could be fitted within a full Bayesian framework, making proper allowance for the uncertainty in τ2 and the study sampling variances s2 i (Smith et al., 1995).”
MethodMultiple-bias modelling for analysis of observational data
Greenland, S. (2005) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.00349.x · Cited by 497
Balk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478 · cited 2×
“, 2003), or looking at the effect of inadequate allocation concealment and inadequate allocation sequence generation (Schulz et al., 1995; Moher et al., 1998; Kjaergard et al., 2001; Juni et al., 2001; Balk et al., 2002).”
BackgroundGreenland, S. (2001) · Biostatistics · doi:10.1093/biostatistics/2.4.463 · Cited by 359 · cited 1×
“Weighting the analysis by quality scores is known to be inadequate (Greenland and O’Rourke, 2001); among other reasons, many quality items contributing to the score may have no association with bias, and there is no allowance for the direction of biases.”
BackgroundMoja, L. P., Telaro, E., D'Amico, R., Moschetti, I., Coe, L., & Liberati, A. (2005) · BMJ · doi:10.1136/bmj.38414.515938.8f · Cited by 239
Spiegelhalter, D. J., & Best, N. G. (2003) · Statistics in Medicine · doi:10.1002/sim.1586 · Cited by 235
Self-citationChilcott, J., Lloyd Jones, M., Wight, J., Forman, K., Wray, J., Beverley, C., & Tappenden, P. (2003) · Health Technology Assessment · doi:10.3310/hta7040 · Cited by 210
Multiparameter Evidence Synthesis in Epidemiology and Medical Decision-Making: Current Approaches
Ades, A. E., & Sutton, A. J. (2006) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2005.00377.x · Cited by 159
Models for Potentially Biased Evidence in Meta-Analysis using Empirically Based Priors
Welton, N. J., Ades, A. E., Carlin, J. B., Altman, D. G., & Sterne, J. A. C. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00548.x · Cited by 150
Greenland, S. (2003) · Journal of the American Statistical Association · doi:10.1198/01621450338861905 · Cited by 141 · cited 1×
“In observational studies, it may be desirable to use a detailed model of adjustment for uncontrolled confounding (Greenland, 2003; Schneeweiss et al., 2005), particularly where studies are reported in more detail than in the anti-D immunoglobulin example.”
BackgroundIdentification and impact of outcome selection bias in meta‐analysis
Williamson, P. R., & Gamble, C. (2005) · Statistics in Medicine · doi:10.1002/sim.2025 · Cited by 132 · cited 1×
“We do not here address publication bias or selective reporting of outcomes, for which methods have been discussed elsewhere (Rothstein et al., 2005; Williamson and Gamble, 2005).”
MethodMeWAnalysis by the Confidence Profile Method: The Statistical Synthesis of Evidence
Lenth, R. V. (1993) · Technometrics · doi:10.1080/00401706.1993.10485067 · Cited by 121
Allowing for uncertainty due to missing data in meta‐analysis—Part 1: Two‐stage methods
White, I. R., Higgins, J. P. T., & Wood, A. M. (2008) · Statistics in Medicine · doi:10.1002/sim.3008 · Cited by 114
Schneeweiss, S., Glynn, R. J., Tsai, E. H., Avorn, J., & Solomon, D. H. (2005) · Epidemiology · doi:10.1097/01.ede.0000147164.11879.b5 · Cited by 112
Meta-Analysis by the Confidence Profile Method: The Statistical Synthesis of Evidence
Lenth, R. V., Eddy, D. M., Hasselblad, V., & Shachter, R. D. (1993) · Technometrics · doi:10.2307/1269690 · Cited by 80
Wolpert, R. L., & Mengersen, K. L. (2004) · Statistical Science · doi:10.1214/088342304000000350 · Cited by 63
Welton, N. J., Ades, A. E., Caldwell, D. M., & Peters, T. J. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00558.x · Cited by 57
de Craen, A. J. M., van Vliet, H. A. A. M., & Helmerhorst, F. M. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.07.002 · Cited by 22
Research works printed but not linked: 16
- Bowman, 1978, Rh isoimmunization during pregnancy: antenatal prophylaxis, Can. Med. Ass. J., 118, 623
- Bowman, 1978, Antenatal prophylaxis of Rh isoimmunization: 28-weeks’-gestation service program, Can. Med. Ass. J., 118, 627
- Bowman, 1987, Failures of intravenous Rh immune globulin prophylaxis: an analysis of the reasons for such failures, Transfusn Med. Rev., 1, 101, 10.1016/S0887-7963(87)70010-8 · doi:10.1016/s0887-7963(87)70010-8
- Contreras, 1998, The prevention of Rh haemolytic disease of the fetus and newborn—general background, Br. J. Obstetr. Gyn., suppl., 18, 7, 10.1111/j.1471-0528.1998.tb10285.x · doi:10.1111/j.1471-0528.1998.tb10285.x
- Evans, 1993, Statistical Distributions
- Hermann, 1984, Antenatal prophylaxis of Rh immunization with 250 μg anti-D immunoglobulin, Acta Obstetr. Gyn. Scand., suppl., 124, 1
- Higgins, 2005, The Cochrane handbook for systematic reviews of interventions 4.2.5
- Huchet, 1987, Application anté-partum du traitement préventif d’immunisation Rhésus D chez les femmes Rhésus négatif, J. Gyn. Obstetr. Biol. Reprodn, 16, 101
- Lee, 1995, Multicentre trial of antepartum low-dose anti-D immunoglobulin, Transfusn Med., 5, 15, 10.1111/j.1365-3148.1995.tb00180.x · doi:10.1111/j.1365-3148.1995.tb00180.x
- MacKenzie, 1999, Routine antenatal Rhesus D immunoglobulin prophylaxis: the results of a prospective 10 year study, Br. J. Obstetr. Gyn., 106, 492, 10.1111/j.1471-0528.1999.tb08304.x · doi:10.1111/j.1471-0528.1999.tb08304.x
- Mayne, 1997, Rate of RhD sensitisation before and after implementation of a community based antenatal prophylaxis programme, Br. Med. J., 315, 1588, 10.1136/bmj.315.7122.1588 · doi:10.1136/bmj.315.7122.1588
- Oakley, 2007, Uncertainty in prior elicitations: a nonparametric approach, Biometrika, 94, 427, 10.1093/biomet/asm031 · DOI
- Parsons, 1998, A comparison of the rate of Rh(D) alloimmunisation between Nova Scotia and Scotland, Br. J. Obstetr. Gyn., suppl., 18, 39
- Sterne, 2008, Incorporation of potentially biased evidence in systematic reviews and meta-analyses
- Tovey, 1983, The Yorkshire antenatal anti-D immunoglobulin trial in primigravidae, Lancet, 244, 10.1016/S0140-6736(83)90232-5 · doi:10.1016/s0140-6736(83)90232-5
- Trolle, 1989, Prenatal Rh-immune prophylaxis with 300 μg immune globulin anti-D in the 28th week of pregnancy, Acta Obstetr. Gyn. Scand., 68, 45, 10.3109/00016348909087688 · doi:10.3109/00016348909087688
Cited by
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
(2008) · Cited by 3,790
Checking consistency in mixed treatment comparison meta-analysis
Dias, S., Welton, N. J., Caldwell, D. M., & Ades, A. E. (2010) · Statistics in Medicine · doi:10.1002/sim.3767 · Cited by 2,521
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569
Self-citationBias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043
Metadata
- Authors
- Turner, R. M., Spiegelhalter, D. J., Smith, G. C. S., & Thompson, S. G.
- Venue
- Journal of the Royal Statistical Society Series A: Statistics in Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-07-08
- Volume
- 172
- Issue
- 1
- Pages
- 21-47
- ISSN
- 0964-1998 (ISSN-L)
- 1467-985X
- OpenAlex
- W2054224089
- Cited by
- 321
- References
- 35
- FWCI
- 18.73
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Funders
- UK Medical Research Council (U.1052.00.001, U.1052.00.005)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.