Events

Generalist Foundation Models Are Not Clinical Enough for Hospital Operations

Speaker: Lavender Jiang

Location: 1 MetroTech Center, Room 22nd Floor

Date: Monday, August 11, 2025

Abstract: General-purpose foundation models score highly on proxy medical benchmarks but are not clinical enough for hospital operations: hospitals need calibrated predictions on real patient notes that affect cost, capacity and quality. Using ReMedE, an operations-grounded evaluation suite built from the electronic health record (EHR) of a multi-hospital academic health system, we find that off-the-shelf generalist models underperform smaller domain-specialized models pretrained and finetuned on clinical notes by 1.563% - 26.977% AUC. To address this gap, we pretrain from scratch Lang-1, a family of language models on large-scale EHR data plus web texts. After task-specific finetuning, Lang1 improves performance on ReMedE, transfers across related tasks via a unified instruction format, and is more robust under temporal shift. Learning-dynamics analysis shows that pretraining alone is insufficient for ReMedE, even as other zero-shot abilities emerge. We conjecture this is due to the lack of coverage of specialized knowledge for clinical tasks on the internet. These results caution against relying on proxy healthcare benchmarks and off-the-shelf generalist models for deployment, and argue that in-domain pretrained and finetuned models, coupled with real-world evaluation, are required for safe clinical AI.

Bio: Lavender Jiang is a fifth-year PhD candidate (medical school track) at NYU CDS, advised by Eric Oermann and Kyunghyun Cho; she holds a B.S. in ECE & Mathematics at Carnegie Mellon University and is an Apple Scholar in AI/ML.

Please note that non-lab personnel who wish to attend in person need to obtain approval from one of the Lab PIs. Please list that PI as requested in the registration form. Please RSVP by filling out this Google Form. For online attendees, a Zoom link will be sent out prior to the event. Please reach out to global-ai-frontier-lab@nyu.edu with any questions.