Large language models as sources of diabetic retinopathy patient education: a comparative evaluation of safety, health information quality, and readability.

Six widely accessible LLMs were evaluated on 50 standardized diabetic retinopathy patient questions across multiple domains by five independent raters.

• Models tested: ChatGPT, Gemini, DeepSeek, Doubao, Copilot, and Perplexity.
• Raters used guideline-based reference answer elements.
• Rater-based outcomes were Safety, Accuracy, Empathy, DISCERN, EQIP, JAMA benchmark criteria, and Global Quality Score (GQS).
• Readability was assessed with ARI, Coleman-Liau, Flesch-Kincaid Grade Level, Gunning Fog, SMOG, and Flesch Reading Ease Score.

Inter-rater agreement was moderate for Safety and good for the non-binary outcomes.

• Fleiss' kappa for Safety was 0.581 (moderate).
• ICC (2,1) values for non-binary outcomes ranged from 0.806 to 0.870 (good).

Most LLM responses were classified as safe, with no significant between-model difference in safety.

• Safe response rates ranged from 86.0% to 94.0%.
• No significant between-model difference was observed.
• Limitations remained in safety escalation.

Accuracy scores were generally high across models, with statistically significant between-model differences but a small effect size.

• Differences were statistically significant but the effect size was small.
• Paired nonparametric tests were used for between-model comparisons.

Empathy varied more noticeably across models, with Copilot scoring highest and Doubao and Perplexity lower.

• Copilot had the highest empathy scores.
• Doubao and Perplexity scored lower on empathy.
• Empathy was identified as a remaining limitation.

Perplexity achieved the highest DISCERN and JAMA scores, while Copilot performed best on EQIP; GQS differences were small.

• Perplexity led on DISCERN and JAMA benchmark criteria (the latter reflecting transparency).
• Copilot led on EQIP.
• GQS differences between models were small.
• Transparency was noted as a limitation overall.

Readability differed significantly across models, with Gemini generally producing the most readable responses, but all models exceeded the recommended sixth-grade reading level.

• All Flesch Reading Ease Scores were below the recommended threshold.
• Gemini was generally the most readable.
• Readability was assessed using six indices.

No single LLM performed best across all domains.

• Perplexity led on DISCERN/JAMA, Copilot on empathy and EQIP, Gemini on readability.
• Safety did not differ significantly between models.

LLM-generated DR information may support general patient education but should not replace professional ophthalmic consultation.

• Models were generally safe and accurate.
• Limitations remained in safety escalation, empathy, transparency, and readability.