COMPARATIVE ANALYSIS OF THREE ARTIFICIAL INTELLIGENCE- POWERED CHATBOTS IN ANSWERING TURKISH UNDERGRADUATE PEDIATRIC DENTISTRY CURRICULUM QUESTIONS

Çakıcı Ş., Akkoç S.

8. Uluslararası Sağlık Bilimleri ve Yaşam Kongresi, Burdur, Turkey, 16 - 19 April 2025, pp.386, (Summary Text)

  • Publication Type: Conference Paper / Summary Text
  • City: Burdur
  • Country: Turkey
  • Page Numbers: pp.386
  • Kütahya Health Sciences University Affiliated: Yes

Abstract

Background: Artificial intelligence-powered chatbots are increasingly being integrated into dental education and treatment planning. However, the reliability and depth of their expertise in this specialized field remain unclear. Aim:This study aims to evaluate the performance of three artificial intelligence-powered chatbots in answering from the undergraduate pediatric dentistry curriculum questions. Methods: A set of 40 questions was determined from the five-option multiple choice questions used in the undergraduate education of pediatric dentistry at the faculty of the dentistry. The researchers categorized the questions according to the distribution of topics within the curriculum. The questions were submitted in Turkish to ChatGPT-4.0, Gemini-2.0 Flash, and DeepSeek-V3 by a single researcher, in the same order and simultaneously, on March 21, 2025. The accuracy of the responses was compared to an answer key, and the answers were classified as correct or incorrect. Statistical analysis was conducted using Pearson’s chi-squared test and Fleiss kappa (p<0.05). Results: The findings revealed no significant difference in the accuracy rates of the responses given by ChatGPT-4.0, Gemini-2.0 Flash, and DeepSeek-V3 (p=0.502). ChatGPT-4.0 achieved the highest accuracy rate (70%), followed by DeepSeek-V3 (65%) and Gemini-2.0 Flash (57.5%). The kappa values, indicating the agreement level among the three artificial intelligence-powered chatbots' responses, demonstrated poor consistency (κ=0.025). Discussion and Conclusion: Although artificial intelligence-powered chatbots hold promise for enhancing dental education, their current reliability remains limited. The observed discrepancies in accuracy and lack of consistency highlight the need for further improvements.

 

Keywords: Artificial intelligence, Chatbot, ChatGPT, Curriculum, Pediatric dentistry