The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Pathologists’ diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and reproducibility study
Open accessAbstract
Objective To quantify the accuracy and reproducibility of pathologists’ diagnoses of melanocytic skin lesions. Design Observer accuracy and reproducibility study. Setting 10 US states. Participants Skin biopsy cases (n=240), grouped into sets of 36 or 48. Pathologists from 10 US states were randomized to independently interpret the same set on two occasions (phases 1 and 2), at least eight months apart. Main outcome measures Pathologists’ interpretations were condensed into five classes: I (eg, nevus or mild atypia); II (eg, moderate atypia); III (eg, severe atypia or melanoma in situ); IV (eg, pathologic stage T1a (pT1a) early invasive melanoma); and V (eg, ≥pT1b invasive melanoma). Reproducibility was assessed by intraobserver and interobserver concordance rates, and accuracy by concordance with three reference diagnoses. Results In phase 1, 187 pathologists completed 8976 independent case interpretations resulting in an average of 10 (SD 4) different diagnostic terms applied to each case. Among pathologists interpreting the same cases in both phases, when pathologists diagnosed a case as class I or class V during phase 1, they gave the same diagnosis in phase 2 for the majority of cases (class I 76.7%; class V 82.6%). However, the intraobserver reproducibility was lower for cases interpreted as class II (35.2%), class III (59.5%), and class IV (63.2%). Average interobserver concordance rates were lower, but with similar trends. Accuracy using a consensus diagnosis of experienced pathologists as reference varied by class: I, 92% (95% confidence interval 90% to 94%); II, 25% (22% to 28%); III, 40% (37% to 44%); IV, 43% (39% to 46%); and V, 72% (69% to 75%). It is estimated that at a population level, 82.8% (81.0% to 84.5%) of melanocytic skin biopsy diagnoses would have their diagnosis verified if reviewed by a consensus reference panel of experienced pathologists, with 8.0% (6.2% to 9.9%) of cases overinterpreted by the initial pathologist and 9.2% (8.8% to 9.6%) underinterpreted. Conclusion Diagnoses spanning moderately dysplastic nevi to early stage invasive melanoma were neither reproducible nor accurate in this large study of pathologists in the USA. Efforts to improve clinical practice should include using a standardized classification system, acknowledging uncertainty in pathology reports, and developing tools such as molecular markers to support pathologists’ visual assessments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Yaşlı Kadınlarda Üreme Sağlığı
Kokanalı, D., & Engin Üstün, Y. (n.d.) · DergiPark (Istanbul University) · Cited by 170,090
Egner, J. R. (2010) · JAMA · doi:10.1001/jama.2010.1525 · Cited by 5,997
Problems of Spectrum and Bias in Evaluating the Efficacy of Diagnostic Tests
Ransohoff, D. F., & Feinstein, A. R. (1978) · New England Journal of Medicine · doi:10.1056/nejm197810262991705 · Cited by 1,734
Improving Diagnosis in Health Care
Committee on Diagnostic Error in Health Care, Board on Health Care Services, Institute of Medicine, & The National Academies of Sciences, E. (2015) · doi:10.17226/21794 · Cited by 1,376
The Genetic Evolution of Melanoma from Precursor Lesions
Shain, A. H., Yeh, I., Kovalyshyn, I., Sriharan, A., Talevich, E., Gagnon, A., Dummer, R., North, J., Pincus, L., Ruben, B., et al. (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1502583 · Cited by 1,065
Shain, A. H., & Bastian, B. C. (2016) · Nature Reviews Cancer · doi:10.1038/nrc.2016.37 · Cited by 896
Diagnostic Concordance Among Pathologists Interpreting Breast Biopsy Specimens
Elmore, J. G., Longton, G. M., Carney, P. A., Geller, B. M., Onega, T., Tosteson, A. N. A., Nelson, H. D., Pepe, M. S., Allison, K. H., Schnitt, S. J., et al. (2015) · JAMA · doi:10.1001/jama.2015.1405 · Cited by 751
Self-citationInviting Patients to Read Their Doctors' Notes: A Quasi-experimental Study and a Look Ahead
Delbanco, T., Walker, J., Bell, S. K., Darer, J. D., Elmore, J. G., Farag, N., Feldman, H. J., Mejilla, R., Ngo, L., Ralston, J. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-7-201210020-00002 · Cited by 685
Self-citationVariability in Radiologists' Interpretations of Mammograms
Elmore, J. G., Wells, C. K., Lee, C. H., Howard, D. H., & Feinstein, A. R. (1994) · New England Journal of Medicine · doi:10.1056/nejm199412013312206 · Cited by 601
Self-citationFarmer, E. R., Gonin, R., & Hanna, M. P. (1996) · Human Pathology · doi:10.1016/s0046-8177(96)90157-4 · Cited by 440
Barnhill, R. L., Argenyi, Z. B., From, L., Glass, L. F., Maize, J. C., Mihm, M. C., Rabkin, M. S., Ronan, S. G., White, W. L., & Piepkorn, M. (1999) · Human Pathology · doi:10.1016/s0046-8177(99)90193-4 · Cited by 426
Self-citationSkin biopsy rates and incidence of melanoma: population based ecological study
Welch, H. G., Woloshin, S., & Schwartz, L. M. (2005) · BMJ · doi:10.1136/bmj.38516.649537.e0 · Cited by 330
Corona, R., Mele, A., Amini, M., De Rosa, G., Coppola, G., Piccardi, P., Fucci, M., Pasquini, P., & Faraggiana, T. (1996) · Journal of Clinical Oncology · doi:10.1200/jco.1996.14.4.1218 · Cited by 266
Discordance in the histopathologic diagnosis of melanoma at a melanoma referral center
Shoo, B. A., Sagebiel, R. W., & Kashani-Sabet, M. (2010) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2009.09.043 · Cited by 221
Physician expressions of uncertainty during patient encounters
Gordon, G. H., Joos, S. K., & Byrne, J. (2000) · Patient Education and Counseling · doi:10.1016/s0738-3991(99)00069-5 · Cited by 174
Interobserver Reproducibility of Histopathologic Prognostic Variables in Primary Cutaneous Melanomas
Scolyer, R. A., Shaw, H. M., Thompson, J. F., et al. (2003) · The American Journal of Surgical Pathology · doi:10.1097/00000478-200312000-00011 · Cited by 168
The Delphi Method, III: Use of Self-Ratings To Improve Group Estimates
(1969) · RAND Corporation eBooks · doi:10.7249/rm6115 · Cited by 162
Brochez, L., Verhaeghe, E., Grosshans, E., Haneke, E., Pierard, G., Ruiter, D. J., & Naeyaert, J. M. (2002) · The Journal of Pathology · doi:10.1002/path.1061 · Cited by 152
Nitti, D., BELLUCO, C., FASSINA, A., Ninfo, V., LISE, M., MC, M., R, B., & P, D. P. (1998) · Padua Research Archive (University of Padova) · doi:10.1016/s0959-8049 · Cited by 146
The MPATH-Dx reporting schema for melanocytic proliferations and melanoma
Piepkorn, M. W., Barnhill, R. L., Elder, D. E., et al. (2013) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2013.07.027 · Cited by 136
Self-citationDuncan, L. M., Berwick, M., BRUIJN, J. A., Byers, H. R., Mihm, M. C., & Barnhill, R. L. (1993) · Journal of Investigative Dermatology · doi:10.1038/jid.1993.55 · Cited by 135
Self-citationThe Melanoma Epidemic: Is Increased Surveillance the Solution or the Problem?
Swerlick, R. A., & Chen, S. H. (1996) · Archives of Dermatology · doi:10.1001/archderm.132.8.881 · Cited by 123
A multiobserver, population-based analysis of histologic dysplasia in melanocytic nevi
Piepkorn, M. W., Barnhill, R. L., Cannon‐Albright, L., et al. (1994) · Journal of the American Academy of Dermatology · doi:10.1016/s0190-9622(08)81499-5 · Cited by 104
Self-citationReliability of the Histopathologic Diagnosis of Melanocytic Dysplasia
Weinstock, M. A., Barnhill, R. L., Rhodes, A. R., & Brodsky, G. L. (1997) · Archives of Dermatology · doi:10.1001/archderm.1997.03890440019002 · Cited by 95
Self-citationHeenan, P. J., Matz, L. R., Blackwell, J. B., Kelsall, G. R. H., SINGH, A., Seldam, R. E. J. T., & Holman, C. (1984) · Histopathology · doi:10.1111/j.1365-2559.1984.tb02388.x · Cited by 90
Reliability of the Histopathologic Diagnosis of Malignant Melanoma in Childhood
Wechsler, J. (2002) · Archives of Dermatology · doi:10.1001/archderm.138.5.625 · Cited by 88
Interobserver reproducibility of ulceration assessment in primary cutaneous melanomas
Spatz, A., Cook, M. G., Elder, D. E., et al. (2003) · European Journal of Cancer · doi:10.1016/s0959-8049(03)00325-3 · Cited by 88
Self-citationFerrara, G., Argenziano, G., Soyer, H. P., Corona, R., Sera, F., Brunetti, B., Cerroni, L., Chimenti, S., Shabrawi‐Caelen, L. E., Ferrari, A., et al. (2002) · Cancer · doi:10.1002/cncr.10768 · Cited by 81
A bibliography of publications on observer variability
Feinstein, A. R. (1985) · Journal of Chronic Diseases · doi:10.1016/0021-9681(85)90016-5 · Cited by 79
McCullough, B. J., Johnson, G. R., Martin, B. I., & Jarvik, J. G. (2012) · Radiology · doi:10.1148/radiol.11110618 · Cited by 79
Discordance of histopathologic parameters in cutaneous melanoma: Clinical implications
Patrawala, S. A., Maley, A. M., Greskovich, C., Stuart, L. N., Parker, D. C., Swerlick, R. A., & Stoff, B. K. (2015) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2015.09.008 · Cited by 66
Reliability and inter-observer agreement of dermoscopic diagnosis of melanoma and melanocytic naevi
Carli, P., GIORGI, V. D., Naldi, L., & Dosi, G. (1998) · European Journal of Cancer Prevention · doi:10.1097/00008469-199810000-00005 · Cited by 66
Elmore, J. G., Nelson, H. D., PEPE, M. S., Longton, G., Tosteson, A. N. A., Geller, B. M., Onega, T., Carney, P. A., Jackson, S. L., Allison, K. H., et al. (2016) · Annals of Internal Medicine · doi:10.7326/m15-0964 · Cited by 66
Self-citationDuray, P. H., DerSimonian, R., Barnhill, R. L., et al. (1992) · Journal of the American Academy of Dermatology · doi:10.1016/0190-9622(92)70248-e · Cited by 59
Self-citationMurali, R., Hughes, M. T., Fitzgerald, P., Thompson, J. F., & Scolyer, R. A. (2009) · Annals of Surgery · doi:10.1097/sla.0b013e31819ed973 · Cited by 55
Meyer, L. J., Piepkorn, M. W., Goldgar, D. E., Lewis, C. M., Cannon‐Albright, L., Zone, J. J., & Skolnick, M. H. (1996) · Journal of the American Academy of Dermatology · doi:10.1016/s0190-9622(96)80061-2 · Cited by 53
Self-citationA bibliography of publications on observer variability (final installment)*1
Elmore, J. G. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90128-a · Cited by 51
Self-citationBraun, R. P., Gutkowicz–Krusin, D., Rabinovitz, H. S., Cognetta, A. B., Hofmann‐Wellenhof, R., Ahlgrimm‐Siess, V., Polsky, D., Oliviero, M., Kolm, I., Googe, P. B., et al. (2012) · Dermatology · doi:10.1159/000336886 · Cited by 50
Boolean Algebra and Clinical Taxonomy
Feinstein, A. R. (1963) · New England Journal of Medicine · doi:10.1056/nejm196310312691801 · Cited by 50
(1997) · Journal of Clinical Pathology · doi:10.1136/jcp.50.3.202 · Cited by 43
Carney, P. A., Reisch, L. M., Piepkorn, M. W., Barnhill, R. L., Elder, D. E., Knezevich, S. R., Geller, B. M., Longton, G., & Elmore, J. G. (2016) · Journal of Cutaneous Pathology · doi:10.1111/cup.12751 · Cited by 42
Self-citationNiebling, M. G., Haydu, L. E., Karim, R. Z., et al. (2014) · Annals of Surgical Oncology · doi:10.1245/s10434-014-3682-x · Cited by 37
Gaudi, S., Zarandona, J. M., Raab, S. S., English, J. C., & Jukic, D. M. (2012) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2012.06.034 · Cited by 36
Krieger, N., Hiatt, R. A., Sagebiel, R. W., Clark, W. H., & Mihm, M. C. (1994) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)90193-7 · Cited by 28
Reliability of skin biopsy pathology.
Boiko, P. E., & Piepkorn, M. W. (1994) · PubMed · Cited by 26
Self-citationUse of Digital Whole Slide Imaging in Dermatopathology
Onega, T., Reisch, L. M., Frederick, P. D., Geller, B. M., Nelson, H. D., Lott, J. P., Radick, A. C., Elder, D. E., Barnhill, R. L., Piepkorn, M. W., et al. (2015) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-015-9836-y · Cited by 25
Self-citation
Cited by
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Current state of melanoma diagnosis and treatment
Davis, L. E., Shalin, S. C., & Tackett, A. J. (2019) · Cancer Biology & Therapy · doi:10.1080/15384047.2019.1640032 · Cited by 922
Automated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834
Guidelines of care for the management of primary cutaneous melanoma
Swetter, S. M., Tsao, H., Bichakjian, C. K., Curiel-Lewandrowski, C., Elder, D. E., Gershenwald, J. E., Guild, V., Grant-Kels, J. M., Halpern, A. C., Johnson, T. M., et al. (2019) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2018.08.055 · Cited by 727
Self-citationThe Rapid Rise in Cutaneous Melanoma Diagnoses
Welch, H. G., Mazer, B. L., & Adamson, A. S. (2021) · New England Journal of Medicine · doi:10.1056/nejmsb2019760 · Cited by 418
Metadata
- Authors
- Elmore, J. G., Barnhill, R. L., Elder, D. E., Longton, G. M., Pepe, M. S., Reisch, L. M., Carney, P. A., Titus, L. J., Nelson, H. D., Onega, T., Tosteson, A. N. A., Weinstock, M. A., Knezevich, S. R., & Piepkorn, M. W.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2017-06-28
- Volume
- 357
- Pages
- j2813
- ISSN
- 1756-1833 (ISSN-L)
- 0959-8138
- OpenAlex
- W2732701910
- Cited by
- 436
- References
- 46
- FWCI
- 15.40
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.