FDA Clears LLM for Triage: Interface or Decision-Maker?

Acute pancreatitis presents a significant diagnostic and management challenge in emergency departments, often requiring rapid and accurate triage to prevent severe complications and optimize resource allocation. The condition's varied presentation and potential for rapid deterioration demand a clinician's keen judgment, but also highlight an area where computational assistance could theoretically improve consistency and speed. The question has always been how deeply such assistance would integrate into the decision-making process itself.²

The recent FDA clearance of an LLM for bedside triage in acute pancreatitis marks a critical juncture. This is not merely a tool for information retrieval or documentation; it is an active participant in the initial assessment of patients presenting with a complex, time-sensitive condition. The clearance compels the medical community to re-evaluate the established boundaries between human expertise and algorithmic recommendation, particularly in scenarios where shared decision-making is paramount.¹

The numbers behind the clearance

A study published in World Journal of Surgery explored the performance of large language models (LLMs) in bedside triage for acute pancreatitis, comparing GPT-4, GPT-5, and Gemini against human expert consensus. The investigators, led by Y.K. Çalışkan from the Department of Emergency Medicine at Hacettepe University, designed a scenario-based comparative evaluation to assess the models' accuracy in recommending appropriate triage levels. They developed 100 distinct clinical scenarios, each detailing a patient presentation consistent with acute pancreatitis, varying in severity, comorbidities, and initial laboratory findings.²

The study's primary endpoint was the accuracy of the LLM's triage recommendation compared to a consensus decision by a panel of three independent emergency medicine specialists. These specialists, blinded to the LLM outputs, reviewed each scenario and assigned a triage level based on established guidelines for acute pancreatitis management. The researchers then fed the same 100 scenarios into GPT-4, GPT-5, and Gemini, prompting each model to provide a triage recommendation. The models were evaluated on their ability to match the expert consensus.²

GPT-4 demonstrated an accuracy of 88% (95% CI, 81-93%) in matching expert consensus for triage recommendations. GPT-5, a newer iteration, performed even better, achieving an accuracy of 92% (95% CI, 86-96%). Gemini, another prominent LLM, showed an accuracy of 85% (95% CI, 77-90%). These figures represent a substantial level of agreement with human specialists, particularly for GPT-5, which approached the upper bounds of inter-rater reliability often observed among human clinicians. The models were particularly proficient in identifying severe cases requiring immediate intervention and mild cases suitable for less intensive monitoring.²

The study also examined the models' performance across different severity strata of acute pancreatitis. For mild cases, GPT-5 achieved 95% accuracy, while for moderate and severe cases, its accuracy remained high at 90% and 89%, respectively. GPT-4 showed similar trends, with slightly lower but still robust accuracy across all severity categories. The models' ability to differentiate between these critical distinctions suggests a capacity to process complex clinical information and apply established guidelines effectively. This is not merely pattern matching; it implies a deeper understanding of the clinical context.²

But the study also highlighted areas where the LLMs struggled. In scenarios involving rare comorbidities or atypical presentations, the models' accuracy dipped slightly, indicating a potential limitation in handling highly unusual cases not well-represented in their training data. For instance, in five specific scenarios involving patients with rare genetic predispositions to pancreatitis, GPT-4 and Gemini misclassified the triage level in two of them, while GPT-5 misclassified one. This suggests a need for continuous refinement and perhaps specialized training datasets for such edge cases.²

The implications of these results extend beyond acute pancreatitis. Another study, published in NPJ Digital Medicine, explored an integrative machine learning-based decision-support framework for injury prediction in elite women's football. This framework, while not an LLM, demonstrates the broader trend of AI moving into predictive and decision-support roles in medicine. The system integrated player load data, biomechanical assessments, and historical injury records to predict injury risk with an accuracy of 82% (95% CI, 78-85%). This predictive capability, though distinct from triage, underscores the growing reliance on AI to inform critical decisions in high-stakes environments.³

The open-label design of the Çalışkan study is an obvious caveat. While the expert panel was blinded to the LLM outputs, the LLMs themselves were not operating in a truly blinded fashion, as they were provided with the full scenario text. This is inherent to how LLMs function, but it means the comparison is between an LLM's direct interpretation of text and a human's interpretation, rather than a comparison of diagnostic accuracy against a gold standard. The study also relied on simulated scenarios, not real-world patient encounters, which may not fully capture the complexities of bedside triage, including non-verbal cues, patient history nuances, and the dynamic nature of emergency department flow.²

Still, the FDA clearance, predicated on data like Çalışkan's, forces a re-evaluation of the LLM's role. Is it an advanced interface, streamlining data presentation and guideline application for the clinician, or is it a nascent decision-maker, offering recommendations that clinicians are expected to follow? The distinction matters profoundly for accountability, liability, and the very nature of clinical autonomy. The study did not explore the impact of LLM recommendations on actual patient outcomes, nor did it assess the potential for alert fatigue or over-reliance by clinicians. These are critical gaps that future research must address.²

The question of shared decision-making, explored in a separate study in Journal of Adolescent Health concerning concussion recovery, becomes particularly relevant here. That research highlighted the complex interplay of parent and adolescent perspectives in medical decisions. While not directly about LLMs, it underscores the human element in medical choices and the need for transparency and understanding when external inputs, whether from family or an algorithm, influence care. The LLM's role must be clearly defined within this human-centric framework.¹

The current FDA clearance does not explicitly define the LLM as a decision-maker, but its function in providing triage recommendations places it squarely in a decision-support role that borders on prescriptive. The next trials will need to show not just accuracy in simulated environments, but also improved patient outcomes, reduced clinician burden, and clear guidelines for integrating these tools without eroding clinical judgment or patient trust. The regulatory landscape will undoubtedly evolve as these systems become more sophisticated and ubiquitous.