Frontier LLMs are highly reproducible internally but diverge from one another asymmetrically, with ChatGPT and Gemini behaving near-identically while Claude diverges conservatively in unruptured intracranial aneurysm treatment recommendations.
Within-model reproducibility for all three LLMs was almost perfect across five repeated runs of the same clinical vignettes.
ChatGPT and Gemini showed near-identical pairwise inter-model agreement, while Claude diverged substantially from both.
The three LLMs produced non-unanimous recommendations in 19.4% of cases, with Claude as the sole outlier in the majority of those disagreements.
Gemini was significantly more pro-treatment than Claude in head-to-head comparison.
ChatGPT and Gemini demonstrated significant pro-treatment propensity compared to the neurovascular multidisciplinary team (MDT) consensus.
Claude was significantly more conservative than the Unruptured Intracranial Aneurysm Treatment Score (UIATS) reference standard.
The study retrospectively analysed 67 UIA cases referred to a neurovascular service over a one-year period using de-identified clinical vignettes submitted to three frontier LLMs.
This research suggests that when patients or clinicians ask different AI chatbots — specifically ChatGPT, Gemini, and Claude — whether to treat an unruptured brain aneurysm (a blood vessel bulge in the brain that hasn't yet burst), they may get meaningfully different answers depending on which AI they consult. The study found that each individual AI is very consistent with itself, giving nearly the same answer if asked the same question multiple times. However, ChatGPT and Gemini tend to agree with each other and lean toward recommending treatment, while Claude more often recommends a 'watch and wait' approach. In about 1 in 5 cases, the three AIs did not agree on a recommendation. When compared to actual decisions made by specialist medical teams and a validated clinical scoring tool, ChatGPT and Gemini recommended treatment more often than the human specialists did, while Claude recommended treatment less often than the scoring tool suggested was appropriate. Neither pattern is necessarily 'right' or 'wrong' in every case, but the disagreements highlight that the AI a patient happens to use could shape their expectations and concerns before they even see a doctor. This research suggests that as more patients turn to AI tools for health information before seeing specialists, clinicians need to be aware that different AI systems may have given a patient conflicting or biased information. The authors call for further research into why these differences exist across AI models and how patients emotionally respond to AI-generated medical recommendations.
Narayan A, Mariajoseph F, Farooq M, Praeger A, Chandra R, Sher I, et al.. (2026). Inter-Model Variability of ChatGPT, Gemini and Claude in Treatment Recommendations for Unruptured Intracranial Aneurysms.. Neurosurgical review. https://doi.org/10.1007/s10143-026-04498-1