The EAU Guidelines Bot represents an accurate and reliable tool for Sexual and Reproductive Health Guidelines navigation, but further validation is required to evaluate its applicability in clinical practice.
The EAU Guidelines Bot demonstrated high accuracy in responding to Sexual and Reproductive Health guidelines-based questions.
The EAU Guidelines Bot demonstrated high completeness in its responses to guidelines-based questions.
The EAU Guidelines Bot demonstrated high clarity in its responses to guidelines-based questions.
The grade of recommendation (strong vs. weak) had no impact on the quality of the EAU Guidelines Bot responses.
This study represents the first external evaluation of the EAU Guidelines Bot, suggesting improved reliability compared to general AI tools.
The study methodology involved systematic expert review of bot responses using a structured rating approach.
This research evaluated an official chatbot created by the European Association of Urology (EAU) to help urologists quickly find guidance from the organization's Sexual and Reproductive Health Guidelines. The researchers created 228 questions directly based on the guidelines and asked the bot each question, then had expert urologists rate the responses on accuracy, completeness, and clarity using a 5-point scale. This was the first independent, external test of this specific guidelines-focused bot. The results were highly positive across all three measures. The bot answered correctly about 98% of the time, provided complete information in about 98% of cases, and gave clear responses in nearly 99% of cases. Notably, the bot performed equally well regardless of whether a question was based on a strong or weak clinical recommendation. The authors highlight that this level of reliability appears to be an improvement over general-purpose AI tools like ChatGPT, which have previously shown inconsistent accuracy when tested on medical guidelines. This research suggests that purpose-built, guidelines-specific AI bots may be more trustworthy for clinical decision support than general AI chatbots. However, the authors caution that the questions tested were straightforward and derived directly from written guidelines, meaning the bot's performance on complex, ambiguous, or real-world clinical questions remains unknown. Further validation in more realistic clinical settings would be needed before drawing broader conclusions about its usefulness in everyday medical practice.
Santarelli V, Lombardo R, Romagnoli M, Sequi M, Coppola L, Rosato E, et al.. (2026). Accuracy, readability, and understandability of European Association of Urology guidelines bot for Sexual and Reproductive Health Guidelines.. The journal of sexual medicine. https://doi.org/10.1093/jsxmed/qdag041