Logo image
Back
Evaluating GPT-4’s Semantic Understanding of Obstetric-based Healthcare Text through Nurse Ruth
Journal article   Open access   Peer reviewed

Evaluating GPT-4’s Semantic Understanding of Obstetric-based Healthcare Text through Nurse Ruth

Tia Pope, Stephanie Gilbertson-White and Ahmad Patooghy
ACM transactions on intelligent systems and technology
05/13/2025
DOI: 10.1145/3735647
url
https://doi.org/10.1145/3735647View
Published (Version of record) Open Access

Abstract

Nurse Ruth, an AI-driven assistant, is designed to support obstetric nursing in resource-limited environments and for non-specialist healthcare providers. To develop and validate Nurse Ruth, we introduced novel evaluation metrics—Semantic Transparency Metric (STM) and Semantic Understanding Metric (SUM)—to assess response accuracy, contextual relevance, and robustness against conventional and adversarial clinical queries. Through iterative refinement and targeted knowledge integration, Nurse Ruth surpassed the 80% threshold for STM and SUM, reinforcing its ability to provide clear, evidence-based, and contextually precise clinical guidance. While excelling in response clarity and contextual accuracy, further improvements are needed to enhance recall in complex, multi-domain obstetric scenarios. A comparative evaluation against leading AI models (GPT-4o, GPT-4, and GPT-o1) for semantic validation demonstrated Nurse Ruth’s superiority. It achieved 100% accuracy on obstetric challenge queries, outperforming general-purpose AI models in both precision and efficiency. Unlike these models, Nurse Ruth delivered concise, rapid responses, making it the most effective system for real-world clinical applications. These findings validate Nurse Ruth’s semantic understanding and establish a replicable framework for AI-driven decision support in specialized medical fields. Future work will focus on refining recall in multi-faceted obstetric cases and validating real-world clinical impact.
 Applied computing Applied computing / Life and medical sciences Applied computing / Life and medical sciences / Bioinformatics Applied computing / Life and medical sciences / Health care information systems Applied computing / Life and medical sciences / Health informatics Computing methodologies Computing methodologies / Artificial intelligence Computing methodologies / Artificial intelligence / Natural language processing Computing methodologies / Artificial intelligence / Natural language processing / Information extraction Computing methodologies / Artificial intelligence / Natural language processing / Lexical semantics Computing methodologies / Machine learning Computing methodologies / Machine learning / Learning settings

Details

Metrics

13 Record Views
Logo image