Diagnostic Accuracy of Four Large Language Models for the Cipto Triage Method: A Comparative Evaluation for Emergency Department Triage in Resource-Limited Settings
Abstract
Triage accuracy remains suboptimal globally, particularly in developing countries employing simplified systems such as the Cipto Triage Method (CTM), and LLM performance on three-level triage systems in non-English contexts has not been evaluated. This study evaluated the diagnostic accuracy of four LLMs (DeepSeek V4 Pro, DeepSeek V4 Flash, Tencent Hy3, MiMo V2.5) for CTM classification as decision support tools for emergency department (ED) triage in resource-limited settings. The models were tested on a stratified random sample of 1,200 clinical records from the MIETIC corpus (v1.0.0; 5,528 unique cases after deduplication), relabeled from the Emergency Severity Index to CTM by four board-certified emergency physicians (400 cases per category), with physician consensus as the reference standard. Primary outcomes were overall accuracy and Cohen kappa; secondary outcomes included per-class sensitivity, specificity, positive and negative predictive value with 95% Wilson-score confidence intervals, undertriage detection rate, and API cost. Accuracy ranged from 66.5% to 68.1% (kappa 0.50–0.52). DeepSeek V4 Flash achieved the highest accuracy (68.1%, 95% CI 65.4–70.7%) at $0.207 total cost. No pairwise McNemar comparison was statistically significant (p > 0.05). All models exceeded benchmarks previously reported for ChatGPT on the Canadian Triage and Acuity Scale, while kappa values were comparable to GPT-4 on the Korean Triage and Acuity Scale. Undertriage detection varied markedly: MiMo V2.5 identified 56.6% of undertriaged cases versus 11.9% for DeepSeek V4 Pro. Lower-cost models performed equivalently to more expensive alternatives, though undertriage detection remained suboptimal across all models. These findings support the potential role of LLMs as adjunctive decision aids in resource-limited EDs, pending prospective validation in authentic clinical settings.
Downloads
References
G. Fitzgerald, G. Jelinek, D. Scott, and M. Gerdtz, “Emergency department triage revisited,” Emerg. Med. J., vol. 27, no. 2, pp. 86–92, 2010, doi: https://doi.org/10.1136/emj.2009.077081.
J. M. Zachariasse, V. van der Hagen, N. Seiger, K. Mackway-Jones, M. van Veen, and H. A. Moll, “Performance of triage systems in emergency care: A systematic review and meta-analysis,” BMJ Open, vol. 9, p. e026471, 2019, doi: https://doi.org/10.1136/bmjopen-2018-026471.
H. Habib, S. Sulistio, I. A. Albar, et al., “Validation of the Cipto Triage Method: A single-centre study from Indonesia,” Open Access Emerg. Med., vol. 12, pp. 137–143, 2020, doi: https://doi.org/10.2147/OAEM.S246598.
B. Haas et al., “Survival of the fittest: The hidden cost of undertriage of major trauma,” J. Am. Coll. Surg., vol. 211, no. 6, pp. 804–811, 2010, doi: https://doi.org/10.1016/j.jamcollsurg.2010.08.014.
M. A. Robertson and E. M. Molyneux, “Triage in the developing world—can it be done?,” Arch. Dis. Child., vol. 85, pp. 208–213, 2001, doi: https://doi.org/10.1136/adc.85.3.208.
N. Schwalbe and B. Wahl, “Artificial intelligence and the future of global health,” Lancet, vol. 395, no. 10236, pp. 1579–1586, May 2020, doi: https://doi.org/10.1016/S0140-6736(20)30226-9.
M. Mulyadi, B. Dedi, W.-L. Hou, I.-C. Huang, and B.-O. Lee, “Nurses’ experiences of emergency department triage during the COVID-19 pandemic in Indonesia,” J. Nurs. Scholarsh., vol. 54, no. 1, pp. 15–23, 2022, doi: https://doi.org/10.1111/jnu.12709.
H. Habib, R. M. Mulyana, S. Sulistio, and I. A. Albar, “Hospital modern triage and its application in Indonesia,” Medika, Jakarta, Indonesia, 2016. [Online]. Available: https://www.researchgate.net/publication/311715654
K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, et al., “Large language models encode clinical knowledge,” Nature, vol. 620, no. 7972, pp. 172–180, 2023, doi: https://doi.org/10.1038/s41586-023-06291-2.
T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, et al., “Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models,” PLOS Digit. Health, vol. 2, p. e0000198, 2023, doi: https://doi.org/10.1371/journal.pdig.0000198.
J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, et al., “The future landscape of large language models in medicine,” Commun. Med., vol. 3, p. 141, 2023, doi: https://doi.org/10.1038/s43856-023-00370-1.
A. Rao, M. Pang, J. Kim, M. Kamineni, et al., “Assessing the utility of ChatGPT throughout the entire clinical workflow: Development and usability study,” J. Med. Internet Res., vol. 25, p. e48659, 2023, doi: https://doi.org/10.2196/48659.
A. Gilson, C. Safranek, T. Huang, V. Socrates, et al., “How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment,” JMIR Med. Educ., vol. 9, p. e45312, 2023, doi: https://doi.org/10.2196/45312.
S. A. Alowais, S. S. Alghamdi, N. Alsuhebany, T. Alqahtani, et al., “Revolutionizing healthcare: The role of artificial intelligence in clinical practice,” BMC Med. Educ., vol. 23, p. 689, 2023, doi: https://doi.org/10.1186/s12909-023-04698-z.
M. Fernandes, S. M. Vieira, F. Leite, C. Palos, S. Finkelstein, and J. M. C. Sousa, “Clinical decision support systems for triage in the emergency department using intelligent systems: A review,” Artif. Intell. Med., vol. 102, p. 101762, Jan. 2020, doi: https://doi.org/10.1016/j.artmed.2019.101762.
C. Y. K. Williams, T. Zack, B. Y. Miao, M. Sushil, et al., “Use of a large language model to assess clinical acuity of adults in the emergency department,” JAMA Netw. Open, vol. 7, no. 5, p. e248895, 2024, doi: https://doi.org/10.1001/jamanetworkopen.2024.8895.
J. H. Kim, S. K. Kim, J. Choi, and Y. Lee, “Reliability of ChatGPT for performing triage task in the emergency department using the Korean Triage and Acuity Scale,” Digit. Health, vol. 10, art. no. 20552076241227132, 2024, doi: https://doi.org/10.1177/20552076241227132.
C. Colakca, M. Ergin, H. S. Ozensoy, A. Sener, S. Gures, and A. Ozhasenekler, “Emergency department triaging using ChatGPT based on emergency severity index principles: A cross-sectional study,” Sci. Rep., vol. 14, art. no. 22106, 2024, doi: https://doi.org/10.1038/s41598-024-73229-7.
B. Arslan, C. Nuhoglu, M. Saticio, and E. Altinbilek, “Evaluating LLM-based generative AI tools in emergency triage: A comparative study of ChatGPT Plus, Copilot Pro, and triage nurses,” Am. J. Emerg. Med., vol. 89, pp. 174–181, 2025, doi: https://doi.org/10.1016/j.ajem.2024.12.024.
J. M. Franc, L. Cheng, A. Hart, R. Hata, and A. Hertelendy, “Repeatability, reproducibility, and diagnostic accuracy of a commercial large language model (ChatGPT) to perform emergency department triage using the Canadian triage and acuity scale,” Can. J. Emerg. Med., vol. 26, no. 1, pp. 40–46, 2024, doi: https://doi.org/10.1007/s43678-023-00616-w.
P. M. Bossuyt, J. B. Reitsma, D. E. Bruns, et al., “STARD 2015: An updated list of essential items for reporting diagnostic accuracy studies,” BMJ, vol. 351, p. h5527, 2015, doi: https://doi.org/10.1136/bmj.h5527.
G. S. Collins, J. A. Hudson, R. D. Riley, et al., “TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods,” BMJ, vol. 385, p. e078378, 2024, doi: https://doi.org/10.1136/bmj-2023-078378.
W. G. Cochran, Sampling Techniques, 3rd ed. New York, NY, USA: Wiley, 1977.
OpenRouter, “Create a chat completion,” OpenRouter API Reference, OpenRouter, Inc., 2026. [Online]. Available: https://openrouter.ai/docs/api/api-reference/chat/create-a-chat-completion. [Accessed: Jul. 14, 2026]
DeepSeek, “DeepSeek V4 Pro 0423,” OpenRouter model card, model identifier deepseek/deepseek-v4-pro, DeepSeek, released Apr. 24, 2026. [Online]. Available: https://openrouter.ai/deepseek/deepseek-v4-pro. [Accessed: Jul. 14, 2026]
DeepSeek, “DeepSeek V4 Flash 0423,” OpenRouter model card, model identifier deepseek/deepseek-v4-flash, DeepSeek, released Apr. 24, 2026. [Online]. Available: https://openrouter.ai/deepseek/deepseek-v4-flash. [Accessed: Jul. 14, 2026]
Tencent, “Hy3,” OpenRouter model card, model identifier tencent/hy3, Tencent, released Jul. 6, 2026. [Online]. Available: https://openrouter.ai/tencent/hy3. [Accessed: Jul. 14, 2026]
Xiaomi, “MiMo-V2.5,” OpenRouter model card, model identifier xiaomi/mimo-v2.5, Xiaomi, released Apr. 22, 2026. [Online]. Available: https://openrouter.ai/xiaomi/mimo-v2.5. [Accessed: Jul. 14, 2026]
D. G. Altman and J. M. Bland, “Diagnostic tests 2: Predictive values,” BMJ, vol. 309, no. 6947, p. 102, 1994, doi: https://doi.org/10.1136/bmj.309.6947.102.
J. Cohen, “A coefficient of agreement for nominal scales,” Educ. Psychol. Meas., vol. 20, no. 1, pp. 37–46, 1960, doi: https://doi.org/10.1177/001316446002000104.
J. L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychol. Bull., vol. 76, no. 5, pp. 378–382, 1971, doi: https://doi.org/10.1037/h0031619.
E. B. Wilson, “Probable inference, the law of succession, and statistical inference,” J. Am. Stat. Assoc., vol. 22, no. 158, pp. 209–212, 1927, doi: https://doi.org/10.1080/01621459.1927.10502953.
Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947, doi: https://doi.org/10.1007/BF02295996.
R. A. Armstrong, “When to use the Bonferroni correction,” Ophthalmic Physiol. Opt., vol. 34, no. 5, pp. 502–508, 2014, doi: https://doi.org/10.1111/opo.12131.
J. S. Hinson, D. A. Martinez, P. S. K. Schmitz, M. Toerper, et al., “Accuracy of emergency department triage using the Emergency Severity Index and independent predictors of under-triage and over-triage in Brazil: A retrospective cohort analysis,” Int. J. Emerg. Med., vol. 11, art. no. 3, 2018, doi: https://doi.org/10.1186/s12245-017-0161-8.
R. T. Sutton, D. Pincock, D. C. Baumgart, D. Sadowski, et al., “An overview of clinical decision support systems: Benefits, risks, and strategies for success,” NPJ Digit. Med., vol. 3, p. 17, 2020, doi: https://doi.org/10.1038/s41746-020-0221-y.
A. Rajaram, H. Li, J. K. Holodinsky, J. N. Hall, et al., “Opening the black box: Challenges and opportunities regarding interpretability of artificial intelligence in emergency medicine,” Can. J. Emerg. Med., vol. 27, no. 2, pp. 83–86, 2025, doi: https://doi.org/10.1007/s43678-024-00827-9.
C. Wang et al., “Patient triage and guidance in emergency departments using large language models: Multimetric study,” J. Med. Internet Res., vol. 27, p. e71613, 2025, doi: https://doi.org/10.2196/71613.
S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal, “Detecting hallucinations in large language models using semantic entropy,” Nature, vol. 630, pp. 625–630, 2024, doi: https://doi.org/10.1038/s41586-024-07421-0.
T. Nadarzynski, O. Miles, A. Cowie, and D. Ridge, “Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: A mixed-methods study,” Digit. Health, vol. 5, art. no. 2055207619871808, 2019, doi: https://doi.org/10.1177/2055207619871808.
I. Sarbay, G. B. Berikol, and I. U. Ozturan, “Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT),” Turk. J. Emerg. Med., vol. 23, no. 3, pp. 156–161, 2023, doi: https://doi.org/10.4103/tjem.tjem_79_23.
S.-H. Moon, J. L. Shim, K.-S. Park, and C.-S. Park, “Triage accuracy and causes of mistriage using the Korean Triage and Acuity Scale,” PLoS One, vol. 14, no. 9, p. e0216972, Sep. 2019, doi: https://doi.org/10.1371/journal.pone.0216972.
I. Nedos et al., “Is artificial intelligence ready for emergency department triage? A retrospective evaluation of multiple large language models in 39,375 patients at a university emergency department,” J. Clin. Med., vol. 15, no. 4, p. 1512, 2026, doi: https://doi.org/10.3390/jcm15041512.
G. Ben Haim, M. Saban, Y. Barash, D. Cirulnik, et al., “Evaluating large language model-assisted emergency triage: A comparison of acuity assessments by GPT-4 and medical experts,” J. Clin. Nurs., vol. 35, no. 10, pp. 4249–4255, 2026, doi: https://doi.org/10.1111/jocn.17490.
J. M. Hoppe, M. K. Auer, A. Struven, S. Massberg, et al., “ChatGPT with GPT-4 outperforms emergency department physicians in diagnostic accuracy: Retrospective analysis,” J. Med. Internet Res., vol. 26, p. e56110, 2024, doi: https://doi.org/10.2196/56110.
F. Gaber, M. Shaik, F. Allega, A. J. Bilecz, et al., “Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis,” NPJ Digit. Med., vol. 8, art. no. 263, 2025, doi: https://doi.org/10.1038/s41746-025-01684-1.
H. Fraser, D. Crossland, I. Bacher, M. L. Ranney, et al., “Comparison of diagnostic and triage accuracy of Ada Health and WebMD symptom checkers, ChatGPT, and physicians for patients in an emergency department: Clinical data analysis study,” JMIR Mhealth Uhealth, vol. 11, p. e49995, 2023, doi: https://doi.org/10.2196/49995.
Y. Raita, T. Goto, M. K. Faridi, D. F. Brown, et al., “Emergency department triage prediction of clinical outcomes using machine learning models,” Crit. Care, vol. 23, art. no. 64, 2019, doi: https://doi.org/10.1186/s13054-019-2351-7.
T. Goto, C. A. Camargo, M. K. Faridi, R. J. Freishtat, et al., “Machine learning-based prediction of clinical outcomes for children during emergency department triage,” JAMA Netw. Open, vol. 2, p. e186937, 2019, doi: https://doi.org/10.1001/jamanetworkopen.2018.6937.
B. M. Porto, “Improving triage performance in emergency departments using machine learning and natural language processing: A systematic review,” BMC Emerg. Med., vol. 24, art. no. 219, 2024, doi: https://doi.org/10.1186/s12873-024-01135-2.
A. Névéol, H. Dalianis, S. Velupillai, G. Savova, and P. Zweigenbaum, “Clinical natural language processing in languages other than English: Opportunities and challenges,” J. Biomed. Semantics, vol. 9, no. 1, p. 12, Dec. 2018, doi: https://doi.org/10.1186/s13326-018-0179-8.
S. Nourbakhsh, “Skyer: A novel benchmark for evaluating the effectiveness of large language models in emergency department triage,” Can. J. Emerg. Med., vol. 28, 2026, doi: https://doi.org/10.1007/s43678-026-01214-2, published online Jul. 11, 2026.
H. Kareemi, H. Li, A. Rajaram, J. K. Holodinsky, J. N. Hall, et al., “Establishing methodological standards for the development of artificial intelligence-based clinical decision support in emergency medicine,” Can. J. Emerg. Med., vol. 27, no. 2, pp. 87–95, 2025, doi: https://doi.org/10.1007/s43678-024-00826-w.
P. Hager, F. Jungmann, R. Holland, K. Bhagat, et al., “Evaluation and mitigation of the limitations of large language models in clinical decision-making,” Nat. Med., vol. 30, pp. 2613–2622, 2024, doi: https://doi.org/10.1038/s41591-024-03097-1.
G. Yao, W. Zhang, Y. Zhu, U.-K. Wong, Y. Zhang, and C. Yang, “Comparing the accuracy of large language models and prompt engineering in diagnosing real-world cases,” Int. J. Med. Inform., vol. 203, p. 106026, 2025, doi: https://doi.org/10.1016/j.ijmedinf.2025.106026.
H. Kareemi, A. Colak, K. Yadav, W. Michalowski, and C. Vaillancourt, “Barriers and facilitators to developing and implementing artificial intelligence-based clinical decision support in the emergency department: A qualitative study,” Can. J. Emerg. Med., vol. 28, 2026, doi: https://doi.org/10.1007/s43678-026-01189-0, published online Jun. 22, 2026.
N. Almehairi, G. Clark, and S. Davis, “How well does GPT-4 perform on an emergency medicine board exam? A comparative assessment,” Can. J. Emerg. Med., vol. 27, no. 12, pp. 969–973, 2025, doi: https://doi.org/10.1007/s43678-025-00951-0.
Copyright (c) 2026 Ahmad Ridwan Fauzi, Adelia Chitra Sazkia, Muhammad Ardiansyah
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlikel 4.0 International (CC BY-SA 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
.png)
.png)
.png)
.png)
.png)
