Journal article
Evaluating commercial multimodal AI for diabetic eye screening and implications for an alternative regulatory pathway
NPJ digital medicine, Vol.9(1), 42
2026
DOI: 10.1038/s41746-025-02216-7
PMCID: PMC12800013
PMID: 41398461
Abstract
Autonomous AI for diabetic eye examination is among the most validated and trusted medical AI systems, supported by extensive real-world evidence demonstrating safety, efficacy, improved outcomes, increased productivity, and cost savings. Yet its adoption remains limited. In contrast, commercially available off-the-shelf generative AI models (OTSAIs) are being rapidly tested in medical settings despite a lack of such real-world validation. These models have shown strong performance on medical reasoning tasks, prompting interest in their potential for clinical deployment. We evaluated four OTSAIs-GPT-4o and GPT-4o-mini (OpenAI, San Francisco, CA), Grok (xAI, San Francisco, CA), and Gemini (Google, Mountain View, CA)-on a specific diagnostic task: diabetic eye examination. The OTSAIs were bundled to ensure consistency, and performance was assessed using a level 3 reference standard, the publicly available Messidor-2 dataset. GPT-4o achieved the highest area under the receiver operator characteristic curve (AUC), 0.83. Grok achieved 0.63, and AUC was not calculable for Gemini. The AUC of retina specialists on the same task was estimated at 0.94, so the emergent performance of OTSAIs does not match that of clinical experts, nor does it approach FDA endpoints for consideration as a medical device. Nevertheless, as the performance of these OTSAIs approaches theoretical limits in the future, there might be a regulatory path through task-specific licensing by State Medical Boards for specific clinical tasks. This path may be modeled after licensing for physician assistants, where trust in the bundled OTSAI, to be used in an assistive fashion, is achieved through rigorous validation for safety and efficacy according to widely accepted regulatory considerations for both patient-facing AI, as well as for SaMD processes.
Details
- Title: Subtitle
- Evaluating commercial multimodal AI for diabetic eye screening and implications for an alternative regulatory pathway
- Creators
- Matthew S Hunt - University of IowaTinglong Dai - Johns Hopkins UniversityMichael D Abràmoff - Veterans Affairs Medical Center, Iowa City, IA, USA. michael-abramoff@uiowa.edu
- Resource Type
- Journal article
- Publication Details
- NPJ digital medicine, Vol.9(1), 42
- DOI
- 10.1038/s41746-025-02216-7
- PMID
- 41398461
- PMCID
- PMC12800013
- NLM abbreviation
- NPJ Digit Med
- ISSN
- 2398-6352
- eISSN
- 2398-6352
- Publisher
- NATURE PORTFOLIO
- Grant note
- Robert C. Watzke Endowment
The publicly available Messidor-2 dataset was kindly provided by the Messidor program partners (see https://www.adcis.net/en/third-party/messidor/).
- Language
- English
- Electronic publication date
- 12/15/2025
- Date published
- 2026
- Academic Unit
- Electrical and Computer Engineering; Fraternal Order of Eagles Diabetes Research Center; Ophthalmology and Visual Sciences
- Record Identifier
- 9985090654102771
Metrics
11 Record Views