The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet
Open accessAbstract
BACKGROUND: Magnetic resonance imaging (MRI) of the knee is the preferred method for diagnosing knee injuries. However, interpretation of knee MRI is time-intensive and subject to diagnostic error and variability. An automated system for interpreting knee MRI could prioritize high-risk patients and assist clinicians in making diagnoses. Deep learning methods, in being able to automatically learn layers of features, are well suited for modeling the complex relationships between medical images and their interpretations. In this study we developed a deep learning model for detecting general abnormalities and specific diagnoses (anterior cruciate ligament [ACL] tears and meniscal tears) on knee MRI exams. We then measured the effect of providing the model's predictions to clinical experts during interpretation. METHODS AND FINDINGS: Our dataset consisted of 1,370 knee MRI exams performed at Stanford University Medical Center between January 1, 2001, and December 31, 2012 (mean age 38.0 years; 569 [41.5%] female patients). The majority vote of 3 musculoskeletal radiologists established reference standard labels on an internal validation set of 120 exams. We developed MRNet, a convolutional neural network for classifying MRI series and combined predictions from 3 series per exam using logistic regression. In detecting abnormalities, ACL tears, and meniscal tears, this model achieved area under the receiver operating characteristic curve (AUC) values of 0.937 (95% CI 0.895, 0.980), 0.965 (95% CI 0.938, 0.993), and 0.847 (95% CI 0.780, 0.914), respectively, on the internal validation set. We also obtained a public dataset of 917 exams with sagittal T1-weighted series and labels for ACL injury from Clinical Hospital Centre Rijeka, Croatia. On the external validation set of 183 exams, the MRNet trained on Stanford sagittal T2-weighted series achieved an AUC of 0.824 (95% CI 0.757, 0.892) in the detection of ACL injuries with no additional training, while an MRNet trained on the rest of the external data achieved an AUC of 0.911 (95% CI 0.864, 0.958). We additionally measured the specificity, sensitivity, and accuracy of 9 clinical experts (7 board-certified general radiologists and 2 orthopedic surgeons) on the internal validation set both with and without model assistance. Using a 2-sided Pearson's chi-squared test with adjustment for multiple comparisons, we found no significant differences between the performance of the model and that of unassisted general radiologists in detecting abnormalities. General radiologists achieved significantly higher sensitivity in detecting ACL tears (p-value = 0.002; q-value = 0.019) and significantly higher specificity in detecting meniscal tears (p-value = 0.003; q-value = 0.019). Using a 1-tailed t test on the change in performance metrics, we found that providing model predictions significantly increased clinical experts' specificity in identifying ACL tears (p-value < 0.001; q-value = 0.006). The primary limitations of our study include lack of surgical ground truth and the small size of the panel of clinical experts. CONCLUSIONS: Our deep learning model can rapidly generate accurate clinical pathology classifications of knee MRI exams from both internal and external datasets. Moreover, our results support the assertion that deep learning models can improve the performance of clinical experts during medical imaging interpretation. Further research is needed to validate the model prospectively and to determine its utility in the clinical setting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig 2 MRNet architecture. The MRNet is a convolutional neural network (CNN) that takes as input a series of MRI images and outputs a classification prediction. AlexNet features from each slice of the MRI series are combined using a max pooling (element-wise maximum) operation. The resulting vector is fed through a fully connected layer to produce a single output probability. We trained a different MRNet for each task (abnormality, anterior cruciate ligament [ACL] tear, meniscal tear) and series type (sagittal, coronal, axial), resulting in 9 different MRNets (for external validation, we use only the sagittal plane ACL tear MRNet). For each model, the output probability represents the probability that the model assigns to the series for the presence of the diagnosis.](https://figures.cochat.ai/paper-figures/fc6b31b98e7049efbf562bfb0d687acb/deep-learning-assisted-diagnosis-for-knee-magnetic-resonance-imaging-development-figure-2.jpg)




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using Oxford Levels of Evidence [19] to examine study quality to measure the extent that methodological safeguards (ie, internal study validity) against bias were implemented, 2 studies provided Level 2b evidence as modeling summarizations [20,23], and 2 studies provided Level 4 evidence [21,22].”
From Considering Clinician Competencies for the Implementation of Artificial Intelligence–Based Tools in Health Care: Findings From a Scoping Review (2022) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All 4 studies were undertaken to characterize the performance of internally developed niche AI software systems when used by health care professionals in specific practice settings (Table 1) [20-23].”
From Considering Clinician Competencies for the Implementation of Artificial Intelligence–Based Tools in Health Care: Findings From a Scoping Review (2022) · cites this paper as [23]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“deep learning models have demonstrated diagnostic accuracy comparable to that of human experts in narrow clinical tasks for several medical domains and imaging modalities, including chest and extremity X-rays8–10, computed tomography (CT)11, magnetic resonance imaging (MRI)12, whole slide images (WSI)13,14, and dermatology images15.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“Only 4 research studies were identified in our search [20-23] that addressed professional competencies observed by this new AI landscape; however, none of the identified studies described new AI-related clinical competencies that had to be developed.”
From Considering Clinician Competencies for the Implementation of Artificial Intelligence–Based Tools in Health Care: Findings From a Scoping Review (2022) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Upon full-text screening, 4 articles met our inclusion criteria [20-23].”
From Considering Clinician Competencies for the Implementation of Artificial Intelligence–Based Tools in Health Care: Findings From a Scoping Review (2022) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697
(2017) · Cited by 11,625
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Research works printed but not linked: 2
- MD Zeiler, 2014, Computer vision—ECCV 2014, 818
- LG Nyúl, 1999, On standardizing the MR image intensity scale, Magn Reson Med, 42, 1072, 10.1002/(SICI)1522-2594(199912)42:6<1072::AID-MRM11>3.0.CO;2-M · DOI · Paper page
Cited by
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
“(2020) Histology Trainable attention Lymph nodes Ji (2019) Histology Grad-CAM Musculoskeletal Bien et al. (2018) MRI CAM Chang et al. (2020) MRI CAM Cheng et al. (2019) X-ray Grad-CAM V. Gupta et al. (2020) X-ray Grad-CAM Jamaludin et al. (2017) MRI Guided backpropagation Y.”
BackgroundMethodInfluentialHuman-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020
“For example, a head CT might have findings supporting a diagnosis of stroke, but an MRI could definitively confirm.”
BackgroundInfluentialSelf-citationAI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
Metadata
- Authors
- Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., Halabi, S., Zucker, E., Fanton, G., Amanatullah, D. F., Beaulieu, C. F., Riley, G. M., Stewart, R. J., Blankenberg, F. G., Larson, D. B., Jones, R. H., Langlotz, C. P., Ng, A. Y., & Lungren, M. P.
- Affiliations
- Stanford University (US)
- Stanford Medicine (US)
- Corresponding author
- Bien, Nicholas
- ORCID
- Venue
- PLOS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-11-27
- Volume
- 15
- Issue
- 11
- Pages
- e1002699
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC6258509
- OpenAlex
- W2902874468
- Cited by
- 734
- References
- 42
- Citations by year
- 2026: 61 · 2025: 115 · 2024: 119 · 2023: 105 · 2022: 117 · 2021: 92 · 2020: 79 · 2019: 43 · 2018: 1
- FWCI
- 14.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Stanford Bio-X
- National Institute of Biomedical Imaging and Bioengineering (R01EB000898)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Health Sciences › Medicine › Surgery › Knee injuries and reconstruction techniques
- Keywords
- knee MRI, MRNet, convolutional neural networks, anterior cruciate ligament rupture, meniscal tears, knee injury diagnosis, computer-aided diagnosis, external validation
- MeSH terms
- Anterior Cruciate Ligament Injuries, Tibial Meniscus Injuries, Deep Learning, Adult, Automation, Diagnosis, Computer-Assisted, Female, Humans, Image Interpretation, Computer-Assisted, Knee, Magnetic Resonance Imaging, Male, Middle Aged, Predictive Value of Tests, Retrospective Studies, Reproducibility of Results, Databases, Factual, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. https://app.cochat.ai/papers/10.1371/journal.pmed.1002699
- The paper
- Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., Halabi, S., Zucker, E., Fanton, G., Amanatullah, D. F., Beaulieu, C. F., Riley, G. M., Stewart, R. J., Blankenberg, F. G., Larson, D. B., . . . Lungren, M. P. (2018). Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLOS Medicine. https://doi.org/10.1371/journal.pmed.1002699
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.