Deep learning-enabled medical computer vision
Open accessThe DOI matches its Crossref record; the references it cites are not checked yet; 35 of the 135 sources its claims cite were checked for support.
Abstract
A decade of unprecedented progress in artificial intelligence (AI) has demonstrated the potential for many fields—including medicine—to benefit from the insights that AI techniques can extract from data. Here we survey recent progress in the development of modern computer vision techniques—powered by deep learning—for medical applications, focusing on medical imaging, medical video, and clinical deployment. We start by briefly summarizing a decade of progress in convolutional neural networks, including the vision tasks they enable, in the context of healthcare. Next, we discuss several example medical imaging applications that stand to benefit—including cardiology, pathology, dermatology, ophthalmology–and propose new avenues for continued work. We then expand into general medical video, highlighting ways in which clinical workflows can integrate computer vision to enhance care. Finally, we discuss the challenges and hurdles required for real-world clinical deployment of these technologies.
Overview
CoChat Summary
TL;DR
This review surveys deep learning-based computer vision in medicine, covering medical imaging, medical video, and clinical deployment. It argues that CNNs can match or exceed expert performance in several specialties, but real-world deployment requires attention to data quality, bias, generalizability, community participation, and trust.
Key findings
- Modern medical CV progress is attributed to deep learning, GPU compute, and open labeled datasets; the ImageNet ILSVRC 2012 GPU-powered DL approach was an inflection point, and classification accuracy reached human-level performance by the competition’s final year in 2017.
- In medical imaging, CNNs have matched or surpassed expert physicians in tasks such as dermatology and radiology; examples include gigapixel pathology whole-slide images of around 100,000 × 100,000 pixels versus typical CNN inputs around 200 × 200 pixels, nearly 1 million annotated open-source chest X-ray images, and 10,000 annotated echocardiogram videos open-sourced alongside EchoNet-Dynamic.
- In ophthalmology, CNNs can diagnose diabetic retinopathy at physician-level grading from fundus photographs and can classify non-ocular risk factors from fundus photographs, including age, gender, smoking, hemoglobin-A1c, body-mass index, systolic blood pressure, and diastolic blood pressure, as well as signs of anemia and chronic kidney disease.
- For clinical deployment, the paper highlights four considerations: assessment of data, planning for model limitations, community participation, and trust building; it also notes that dozens of companies have obtained US FDA and European CE approval for medical imaging AI, and that CMS approved reimbursement for a radiology stroke triage use-case.
Methods
- Narrative survey/review of recent progress in deep learning-enabled computer vision for medical applications, focusing on medical imaging, medical video, and clinical deployment.
- No datasets were generated or analyzed; the paper states that data sharing is not applicable. It synthesizes prior literature and examples rather than conducting primary experiments or a systematic review/meta-analysis.
Limitations
- Paper-stated: no datasets were generated or analyzed during the current study, and data sharing is not applicable.
- Evident from the text: as a narrative review, it does not describe a systematic search strategy, inclusion criteria, or quantitative synthesis; the cited applications are illustrative rather than exhaustively evaluated.
- The paper discusses field-level limitations including data access and ownership, bias, out-of-distribution performance, privacy, regulation, and the need for prospective clinical trials; these are limitations of the field rather than of the review itself.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
21 Valid · 6 Not valid · 62 Not checked · 22 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Dozens of companies have obtained US FDA and European CE approval for medical imaging AI 37 , and commercial markets have begun to form as sustainable business models are created.”
§ Deep learning-enabled medical computer vision > MEDICAL IMAGING
What the sources say
- Partly supportedcites [37] Benjamens, S., Dhunnoo, P. & Meskó, B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ Digit. Med. 3, 118 (2020). sourceExperimental — accuracy not yet measured
The Results section reports 21 FDA-approved radiology AI devices and algorithms in its validated set, and the Abstract reports 30 radiology technologies in the broader set. These are counts of technologies, not companies; the cited passages do not establish that dozens of companies obtained approval, and the paper does not report European CE approval. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Scientists have taken this field one step further by building entire medical AI devices designed for monitoring, such as at-home smart toilets outfitted with diagnostic CNNs on cameras 51 .”
§ Deep learning-enabled medical computer vision > MEDICAL IMAGING
What the sources say
- Partly supportedcites [51] Park, S.-M. et al. A mountable toilet system for personalized health monitoring via the analysis of excreta. Nat. Biomed. Eng. 4, 624–635 (2020). sourceExperimental — accuracy not yet measured
The Abstract describes a smart toilet for longitudinal health monitoring, and the Discussion says its CNN classifies stool from images. But the Discussion’s ‘Clinical implications’ section explicitly says the current system is not yet a screening or diagnostic tool, so calling its CNN diagnostic overstates what the paper reports. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using large datasets, CV models can acquire many pattern-recognition abilities-from physician-level diagnostics 7 to medical scene perception 8 .”
§ Deep learning-enabled medical computer vision > INTRODUCTION
What the sources say
- Supportedcites [7] Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017). source
Citation 7 is attached to physician-level diagnostics, not medical scene perception, which has a separate citation. The Abstract reports that a CNN trained on 129,450 clinical images performed on par with 21 board-certified dermatologists in image-based skin-cancer classification. No page number is provided in our copy. (LLM, conf=0.94)
- Supportedcites [8] Yeung, S. et al. A computer vision system for deep learning-based detection of patient mobilization activities in the ICU. NPJ Digit Med. 2, 11 (2019). source
Citation 8 is attached to medical scene perception, not physician-level diagnostics. The Introduction describes CV algorithms that detect patient and staff activities in clinical environments, and the Methods describe a model trained to detect mobility activities in ICU video. Those passages support the cited example of medical scene perception. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Some studies analyze tool movement or other cues to assess surgeon skill 119,121,123,124 , through established ratings such as the Global Operative Assessment of Laparoscopic Skills (GOALS) criteria for laparoscopic surgery 125 .”
§ Deep learning-enabled medical computer vision > MEDICAL VIDEO Surgical applications
What the sources say
- Partly supportedcites [119] Zia, A., Sharma, Y., Bettadapura, V., Sarin, E. L. & Essa, I. Video and accelerometer-based motion analysis for automated surgical skills assessment. Int. J. Comput. Assist. Radiol. Surg. 13, 443–455 (2018). sourceExperimental — accuracy not yet measured
Zia et al. (2018) supports the general idea of motion-based skill assessment scored against an established rating scheme: the Introduction (p. 2) states that 'recent works have proposed techniques that analyze motion from videos [2,3,4,5] and wearable sensors to assess surgical skills [6,7]' and that these perform 'OSATS assessments', with the OSATS criteria (RT, TM, IH, SH, FO, KP, OP) listed on the same page and reused in the Background (p. 2) and abstract. However, the element this citation is attached to — the Global Operative Assessment of Laparoscopic Skills (GOALS) for laparoscopic surgery — appears nowhere in our copy: searches for 'GOALS', 'Global Operative Assessment of Laparoscopic Skills' and laparoscopic assessment returned no passage, and the paper's dataset is open/basic suturing and knot-tying (Sec. 4.1, p. 6), not laparoscopy. So the named rating instrument and the laparoscopic setting are not supported by this source. (LLM, conf=0.72)
- Supportedcites [121] Jin, A. et al. Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) 691–699 (IEEE, 2018). source
Section 5.3 (p. 6) states the paper uses frame-by-frame tool presence, identity and location to extract metrics that reflect surgical skill (instrument usage times, path length) and describes GOALS as "a validated rubric for grading surgeons on laparoscopic surgery performance" across five domains. Section 5.4 (p. 8) and Table 4 (p. 8) show the approach validated against modified GOALS ratings, with subcomponent scores correlating with the extracted tool-movement metrics (heat maps, trajectories). So the source both analyzes tool movement/cues to assess skill and is grounded in the GOALS criteria for laparoscopic surgery, as the citing sentence attributes. (LLM, conf=0.90)
- Not checked yetcites [123] Lin, H. C., Shafran, I., Yuh, D. & Hager, G. D. Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. Comput. Aided Surg. 11, 220–230 (2006). source
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A series of works have demonstrated that CNNs can match the performance of board-certified dermatologists at classifying malignant skin lesions from benign ones 7,93,94 .”
§ Deep learning-enabled medical computer vision > DERMATOLOGY
What the sources say
- Supportedcites [7] Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017). source
The Abstract of Esteva et al. (2017) states the CNN was tested against 21 board-certified dermatologists on biopsy-proven images in two binary use cases — keratinocyte carcinomas versus benign seborrheic keratoses and malignant melanomas versus benign nevi — and 'achieves performance on par with all tested experts across both tasks'. The Discussion/conclusion passages repeat that the CNN 'matches the performance of at least 21 dermatologists tested across three critical diagnostic tasks' and of classifying 'with the accuracy of a board-certified dermatologist'. This is exactly the paraphrased content the citing sentence attributes to this reference (one of the 'series of works'). (LLM, conf=0.90)
- Partly supportedcites [93] Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29, 1836–1842 (2018). sourceExperimental — accuracy not yet measured
Our copy supports the core point at equal or greater strength: the Discussion (p. 4) states that in level-I the dermatologists' dichotomous malignant-vs-benign classification 'was significantly outperformed by the CNN', and the Results/Diagnostic accuracy section (pp. 3-4) gives CNN specificity 82.5% versus mean dermatologist specificity 71.3% (level-I) and 75.7% (level-II), with ROC AUC 0.86 versus 0.79/0.82; the Conclusion (p. 6) says the CNN was 'superior to most but not all dermatologists'. Two qualifiers attributed here are not in our copy: the readers are never described as board-certified — Methods (p. 2) says they were recruited via International Dermoscopy Society mailing lists and self-stratified only by years of dermoscopy experience ('Beginner' <2, 'Skilled' 2-5, 'Expert' >5 years) — and the classification was restricted to dermoscopic images of melanocytic lesions (melanoma vs benign nevi), with the Discussion (p. 7) noting the test sets 'did not display the full range of lesions (e.g. pigmented basal cell carcinoma or seborrheic keratosis)'. So the source backs a CNN matching or exceeding dermatologists on melanocytic dermoscopy, not on skin lesions generally or on a board-certified population. (LLM, conf=0.78)
- Not checked yetcites [94] Brinker, T. J. et al. Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur. J. Cancer 113, 47–54 (2019). source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
38 more claims · 6 with their cited sources checked
Create a free account to see all 111 claims and to verify the 100 references still open.
- Probably not a substantial claim. 48
- It may state two separate claims as one. 37
- Not yet classified. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Subsequent generations of data-driven approaches, including statistical machine learning and deep learning, significantly improved predictive performance but remained fundamentally reactive, offering recommendations or risk scores without the ability to autonomously plan, act, or adapt over time [2-4].”
From Medical AI Agents for Clinical Decision Support: Viewpoint Using the Planning, Action, Reflection, and Memory (PARM) Analytical Lens (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Deep learning-enabled medical computer vision > DERMATOLOGY
“These studies have sequentially tested increasing numbers of dermatologists (25-7 57-93 , 157-94 )”
§ Deep learning-enabled medical computer vision > References
“93. Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.”
Citations
The citation lists are queued
They are fetched later and appear here then.
References 161 · Cited by 1,406
Research works printed but not linked: 8
- Krizhevsky, A., Sutskever, I. & Hinton, G. E. in Advances in Neural Information Processing Systems 25 (eds Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q.) 1097–1105 (Curran Associates, Inc., 2012).
- LeCun, Y. & Bengio, Y. in The Handbook of Brain Theory and Neural Networks 255–258 (MIT Press, 1998).
- Goodfellow, I. et al. Generative adversarial nets. In Advances inneural information processing systems 2672–2680 (2014).
- Jing, L. & Tian, Y. in IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE, 2020).
- Singh, S. P. et al. 3D Deep Learning On Medical Images: A Review. Sensors 20, https://doi.org/10.3390/s20185097 (2020). · DOI
- Wang, D., Khosla, A., Gargeya, R., Irshad, H. & Beck, A. H. Deep learning for identifying metastatic breast cancer. Preprint at https://arxiv.org/abs/1606.05718 (2016). · arXiv
- Luo, Z. et al. Computer vision-based descriptive analytics of seniors’ daily activities for long-term health monitoring. In Machine Learning for Healthcare (MLHC) 2 (JMLR, 2018).
- Ren, J. et al. in Advances in Neural Information Processing Systems 32 (eds Wallach, H. et al.) 14707–14718 (Curran Associates, Inc., 2019).
Metadata
- Authors
- Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R.
- Affiliations
- Salesforce (United States) (US)
- Google (United States) (US)
- Stanford University (US)
- Scripps Research Institute (US)
- Corresponding author
- Esteva, Andre
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2021-01-08
- Volume
- 4
- Issue
- 1
- Pages
- 5-5
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC7794558
- OpenAlex
- W3120894151
- Cited by
- 1,406
- References
- 161
- Citations by year
- 2026: 291 · 2025: 300 · 2024: 298 · 2023: 269 · 2022: 169 · 2021: 79
- FWCI
- 17.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- deep learning, convolutional neural networks, diagnostic imaging, medical videos, clinical deployment, pathology, dermatology, ophthalmology, clinical workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep learning-enabled medical computer vision. https://app.cochat.ai/papers/10.1038/s41746-020-00376-2
- The paper
- Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021). Deep learning-enabled medical computer vision. npj Digital Medicine. https://doi.org/10.1038/s41746-020-00376-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.