Comparing methods of measurement: why plotting difference against standard method is misleading
Abstract
When comparing a new method of measurement with a standard method, one of the things we want to know is whether the difference between the measurements by the two methods is related to the magnitude of the measurement. A plot of the difference against the standard measurement is sometimes suggested, but this will always appear to show a relation between difference and magnitude when there is none. A plot of the difference against the average of the standard and new measurements is unlikely to mislead in this way. We show this theoretically and by a practical example.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A plot of the difference against a standard measurement will always appear to show a relation between difference and magnitude when there is none.
“although this is controversial, because a plot of the difference against a “standard measurement” will always appear to show a relation between difference and magnitude when there is none (9).”
From Understanding bland altman analysis (2015) · cites this paper as [9]
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Measuring agreement in method comparison studies
Bland, J. M., & Altman, D. G. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800204 · Cited by 9,619 · see citing works
Self-citationUnderstanding bland altman analysis
Giavarina, D. (2015) · Biochemia Medica · doi:10.11613/bm.2015.015 · Cited by 4,022 · see citing works
Development, validation and utilisation of food-frequency questionnaires – a review
Cade, J., Thompson, R., Burley, V., & Warm, D. (2002) · Public Health Nutrition · doi:10.1079/phn2001318 · Cited by 1,423 · see citing works
How accurate is pulse rate variability as an estimate of heart rate variability?
Schäfer, A., & Vagedes, J. (2013) · International Journal of Cardiology · doi:10.1016/j.ijcard.2012.03.119 · Cited by 661 · see citing works
Food-frequency questionnaires: a review of their design, validation and utilisation
Cade, J. E., Burley, V. J., Warm, D. L., Thompson, R. L., & Margetts, B. M. (2004) · Nutrition Research Reviews · doi:10.1079/nrr200370 · Cited by 518 · see citing works
Cecconi, M., Rhodes, A., Poloniecki, J., Della Rocca, G., & Grounds, R. M. (2009) · Critical Care · doi:10.1186/cc7129 · Cited by 351 · see citing works
Oscillometric Blood Pressure Estimation: Past, Present, and Future
Forouzanfar, M., Dajani, H. R., Groza, V. Z., Bolic, M., Rajan, S., & Batkin, I. (2015) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2015.2434215 · Cited by 196 · see citing works
Clifton, L., & Clifton, D. A. (2019) · Trials · doi:10.1186/s13063-018-3108-3 · Cited by 178 · see citing works
Drawz, P. E., Pajewski, N. M., Bates, J. T., Bello, N. A., Cushman, W. C., Dwyer, J. P., Fine, L. J., Goff, D. C., Haley, W. E., Krousel-Wood, M., et al. (2017) · Hypertension · doi:10.1161/hypertensionaha.116.08076 · Cited by 170 · see citing works
Kang, S.-G., Kang, J. M., Ko, K.-P., Park, S.-C., Mariani, S., & Weng, J. (2017) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2017.03.009 · Cited by 146 · see citing works
Skeie, S., Thue, G., Nerhus, K., & Sandberg, S. (2002) · Clinical Chemistry · doi:10.1093/clinchem/48.7.994 · Cited by 131 · see citing works
Calculation of Bioavailable and Free Testosterone in Men: A Comparison of 5 Published Algorithms
de Ronde, W., van der Schouw, Y. T., Pols, H. A., Gooren, L. J., Muller, M., Grobbee, D. E., & de Jong, F. H. (2006) · Clinical Chemistry · doi:10.1373/clinchem.2005.063354 · Cited by 131 · see citing works
van Eijk, A. M., Hill, J., Noor, A. M., Snow, R. W., & ter Kuile, F. O. (2015) · The Lancet Global Health · doi:10.1016/s2214-109x(15)00049-2 · Cited by 130 · see citing works
Valbuena, V. S. M., Seelye, S., Sjoding, M. W., Valley, T. S., Dickson, R. P., Gay, S. E., Claar, D., Prescott, H. C., & Iwashyna, T. J. (2022) · BMJ · doi:10.1136/bmj-2021-069775 · Cited by 127 · see citing works
Zaidi, H., Ruest, T., Schoenahl, F., & Montandon, M.-L. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.05.031 · Cited by 108 · see citing works
Stöckl, D., Rodríguez Cabaleiro, D., Van Uytfanghe, K., & Thienpont, L. M. (2004) · Clinical Chemistry · doi:10.1373/clinchem.2004.036095 · Cited by 79 · see citing works
Miida, T., Nishimura, K., Okamura, T., Hirayama, S., Ohmura, H., Yoshida, H., Miyashita, Y., Ai, M., Tanaka, A., Sumino, H., et al. (2012) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2012.08.022 · Cited by 74 · see citing works
Bordenave, S., Flavier, S., Fédou, C., Brun, J. F., & Mercier, J. (2007) · Diabetes & Metabolism · doi:10.1016/j.diabet.2007.04.003 · Cited by 69 · see citing works
Tractenberg, R. E., Jonklaas, J., & Soldin, S. J. (2010) · International Journal of Analytical Chemistry · doi:10.1155/2010/234808 · Cited by 64 · see citing works
Step count accuracy and reliability of two activity tracking devices in people after stroke
Schaffer, S. D., Holzapfel, S. D., Fulk, G., & Bosch, P. R. (2017) · Physiotherapy Theory and Practice · doi:10.1080/09593985.2017.1354412 · Cited by 54 · see citing works
Miraki-Moud, F., Grossman, A. B., Besser, M., Monson, J. P., & Camacho-Hübner, C. (2005) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2004-2090 · Cited by 44 · see citing works
Wang, Q., Zhou, Y., Ghassemi, P., McBride, D., Casamento, J. P., & Pfefer, T. J. (2021) · Sensors · doi:10.3390/s22010215 · Cited by 48 · see citing works
Chang, Y., Xu, L., Han, F., Keenan, B. T., Kneeland-Szanto, E., Zhang, R., Zhang, W., Yu, Y., Zuo, Y., Pack, A. I., et al. (2019) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.7720 · Cited by 44 · see citing works
Agreement between simple questions about sleep duration and sleep diaries in a large online survey
Miller, C. B., Gordon, C. J., Toubia, L., Bartlett, D. J., Grunstein, R. R., D'Rozario, A. L., & Marshall, N. S. (2015) · Sleep Health · doi:10.1016/j.sleh.2015.02.007 · Cited by 44 · see citing works
Lanata, A., Guidi, A., Baragli, P., Valenza, G., & Scilingo, E. P. (2015) · PLOS ONE · doi:10.1371/journal.pone.0140783 · Cited by 42 · see citing works
Lee, Q. Y., Redmond, S. J., Chan, G. S., Middleton, P. M., Steel, E., Malouf, P., Critoph, C., Flynn, G., O’Lone, E., & Lovell, N. H. (2013) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-12-19 · Cited by 34 · see citing works
Zweiker, R., Schumacher, M., Fruhwald, F. M., Watzinger, N., & Klein, W. (2000) · Journal of Hypertension · doi:10.1097/00004872-200018080-00004 · Cited by 29 · see citing works
Dioum, A., Gartner, A., Cissé, A. S., Delpeuch, F., Maire, B., Wade, S., & Schutz, Y. (2005) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/81.3.597 · Cited by 27 · see citing works
- Authors
- Bland, J. M., & Altman, D. G.
- Affiliations
- St George's Hospital (GB)
- ORCID
- Altman, D.G.
- Venue
- The Lancet
- Publisher
- Elsevier BV
- Published
- 1995-10
- Volume
- 346
- Issue
- 8982
- Pages
- 1085-1087
- ISSN
- 0099-5355 (ISSN-L)
- 0140-6736
- OpenAlex
- W2037232743
- Cited by
- 2,343
- References
- 1
- Citations by year
- 2026: 29 · 2025: 56 · 2024: 56 · 2023: 69 · 2022: 62 · 2021: 93 · 2020: 100 · 2019: 77 · 2018: 79 · 2017: 102 · 2016: 104 · 2015: 98 · 2014: 109 · 2013: 111 · 2012: 114
- FWCI
- 35.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Lancet (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Scientific Measurement and Uncertainty Evaluation
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- method comparison, measurement agreement, new measurement method, scatter plots, measurement error, diagnostic accuracy
- MeSH terms
- Humans, Reference Standards, Reference Values, Statistics as Topic, Weights and Measures, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review