Compare 9 Large Language Models in Healthcare
We benchmarked 9 LLMs using the MedQA dataset, a graduate-level clinical exam benchmark derived from USMLE questions. Each model answered the same multiple-choice clinical scenarios using a standardized prompt, enabling direct comparison of accuracy.
We also recorded latency per question by dividing total runtime by the number of MedQA items completed.
Healthcare LLMs benchmark
Benchmark methodology: This benchmark evaluates the supervised fine-tuning performance of healthcare LLMs vs large general-purpose models (GPT-4) on medical question answering tasks. See benchmark data sources.
MedQA:
Multiple-choice medical exam questions based on the United States Medical Licensing Examination.
Figure 1: USMLE-style multiple-choice clinical question example.
MedMCQA:
Large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions.
Figure 2: A large-scale medical entrance-exam multiple-choice question requiring the model to select the correct answer and interpret associated explanations about clinical findings.
PubMedQA: Biomedical question-answering benchmark using yes/no/maybe answers.
Figure 3: A biomedical yes/no/maybe question, where the model must judge the correctness of a clinical claim using the provided study context.
Healthcare LLM examples
BERT-like (Encoder-only)
Optimized for encoding and representing biomedical text, these models excel at extracting features for tasks such as classification.
ChatGPT / LLaMA-like (Decoder, instruction/chat-tuned)
Based on LLaMA-style architectures and optimized for interactive tasks and clinical dialogues.
GPT / PaLM-like (Decoder-only, generative)
Built similarly to GPT-3 or PaLM, these models are fine-tuned for general-purpose text generation and summarization.
General-purpose LLMs in healthcare
*Llama 3.1 Instruct Turbo with 405B parameters. See benchmark methodology.
Key takeaways:
- o1: Best performing model
- 03 mini: Best budget option
- GPT 4.1: Best speed and response time
Beyond accuracy and input cost, models also differ in their underlying approaches to medical question answering. For example, o3 uses a more step-by-step, analytical approach, whereas GPT-5 responds empathetically, organizes, and explains information clearly for non-experts:
Figure 4: Figure showing the differences between the GPT-5 and o3 answers.
Fine-tuning medical LLMs
The performance of the default ChatGPT (4o model) is compared with the existing ‘Clinical Medicine Handbook’ assistant. Both models are given the same prompt, and their responses are analyzed:
GPT 4o
Figure 5: The answer of GPT 4o default model is accurate but also highly summarized.1
Fine-tuned medical LLM
Figure 6: The answer from the specialized agent is better explained and detailed.2
Guides on fine-tuning LLMs
- Fine-tuning large language models for improved health communication3
- Fine-tuning large language models for specialized use cases4
For more: LLM fine-tuning and LLM training.
Applications of general-purpose LLMs
These models are general fine-tuned models that need domain adaptation to perform clinical tasks accurately. You can use these models in healthcare by leveraging:
- Continual pretraining on medical data to help the model better identify medical language by exposing it to clinical notes and biomedical literature (like PubMed).
- RAG to pull data from verified clinical documents to produce accurate responses at runtime.
- Instruction fine-tuning to enable the model to learn how to answer clinical questions or extract symptoms from text.
Figure 7: A general workflow of LLM fine-tuning for specialized use cases.11
Use cases of LLMs in clinical settings
1. Medical transcription
LLMs can help create medical transcriptions by:
- Listening to the organic dialogue between a patient and a clinician.
- Extracting critical medical details.
- Condensing medical data into compliant medical records that align with the relevant sections of an EHR.
Real-life use case: Google’s MedLM can capture &transform the patient-clinician conversation into a medical transcription.12
2. Electronic health records (EHR) enhancement
The proliferation of electronic health records (EHR) has accumulated a vast repository of patient data, which, if mined effectively, can become a goldmine for healthcare improvement.
Real-life use case: Google’s MedLM is also used by BenchSci, Accenture, and Deloitte for electronic health records enhancement (EHR).
- BenchSci has integrated MedLM into its ASCEND platform to improve the quality of preclinical research.
- Accenture uses MedLM to organize unstructured data from numerous sources, automating human operations that were previously time-consuming and error-prone.
- Deloitte works with MedLM to minimize friction in finding treatment. They use an interactive chatbot that helps health plan participants better understand the provider alternatives.13
3. Clinical decision support
Large language models can summarize complex medical concepts, allowing them to support valuable insights in the decision-making process.
Real-life use case: Memorial Sloan Kettering Cancer Center uses IBM Watson Oncology to assist oncologists by analyzing patient data and medical literature to recommend evidence-based treatment options.14
4. Medical research assistance
LLMs can parse and summarize vast amounts of data, can extract key findings from new research, and provide synthesized insights. For example, one of the most famous LLMs, ChatGPT, is used for text summarization.
Real-life use case: John Snow’s healthcare chatbot helps researchers find relevant scientific papers, extract key insights, and identify research trends. It is particularly valuable for navigating the vast amount of biomedical literature.15
Real-life use case: TidalHealth Peninsula Regional clinicians used the Micromedex with Watson solution for healthcare research, claiming that clinicians received their answers in less than one minute ~70% of the time.16
5. Automated patient communication
Large language models in healthcare can draft informative and compassionate responses to patients’ queries.
Some examples include:
- Medication management and reminders: A chatbot provides patients with regular reminders to take their diabetic medication and requests confirmation.
- Health monitoring and follow-up care: A post-operative patient sends their pain and wound status to a chatbot, which determines if the healing process is progressing.
- Informational and educational communication: A patient asks a chatbot how to manage high blood pressure, and the chatbot responds with nutrition and lifestyle tips.
Real-life use case: Boston Children’s Hospital uses Buoy Health, an AI-driven online symptom-checker chatbot, which provides patients with instant answers to health-related questions and initial consultations.
The chatbot can triage patients by analyzing their symptoms and advising whether they need to see a doctor.17
6. Predictive health outcomes
LLMs can assist in predictive analysis by discerning patterns within data.
Real-life use case: WVU pharmacists use a predictive algorithm to leverage LLMs to determine readmission risk. This approach will examine data from electronic health records (EHRs), which include patient demographics, clinical history, and socioeconomic determinants of health.
Based on this research, the WVU pharmacists identify patients at high risk of readmission and assign care coordinators to follow up with them after discharge. This can help reduce readmission rates.18
7- Personalized treatment plans
LLMs can suggest treatment plans tailored to an individual’s medical history and specific needs. Their ability to distill complex patient narratives into actionable insights can ensure that each patient receives a care plan that’s as unique as their health journey.
Real-life use case – Babylon Health: Babylon Health’s AI chatbot provides individualized health recommendations based on the user’s symptoms and medical history. It engages users in a conversation by asking relevant questions to analyze their issues better and giving tailored recommendations.19
8. Medical coding and billing
Large language models can automate audit processes by analyzing patient records and EHRs.
For example, Epic Systems, a major EHR provider, integrates LLMs into its software to assist with coding and billing. The LLMs can monitor for anomalies in access patterns to sensitive patient information or inconsistencies in coding and billing practices.20
However, LLMs are not ready for medical coding but promising: Researchers examined how frequently four LLMs (GPT-3.5, GPT-4, Gemini Pro, and Llama2-70b Chat) issued the correct CPT, ICD-9-CM, and ICD-10-CM codes.
Their findings show a significant opportunity for improvement. Researchers found that LLMs often generate code that transmits inaccurate information, with a maximum accuracy of 50%.21
9. Training and education
Large language models and generative AI in general can be leveraged as interactive educational tools, elucidating complex concepts or offering clarifications on perplexing topics.
Real-life use case: Oxford Medical Simulation uses LLMs integrated with VR technology to create immersive virtual patient simulations.
These simulations allow students to experience high-pressure scenarios, such as handling a cardiac arrest patient without any real-world consequences.
The LLMs power the virtual patients’ responses, making them more realistic and unpredictable, preparing students for the variability of real clinical environments.22
Challenges of LLMs in healthcare
Privacy concerns
Using an LLM-based health application that has not been properly developed, tested, or approved for medical use could present significant risks to users. One of the primary concerns is related to privacy. LLMs and associated tools process health-related data input by users as part of their services. However, how this data is handled and whether these applications comply with data protection laws and principles remains uncertain.23
Accuracy and reliability
LLMs are also prone to hallucinations, plausible-sounding but incorrect or misleading information.
For example, when given a medical query, GPT-3.5 incorrectly recommended tetracycline for a pregnant patient, despite correctly explaining its potential harm to the fetus.24
Figure 8: An example from GPT-3.5 showing the incorrect recommendation of a medicine.
Generalization vs. specialization
Healthcare encompasses a wide range of specialties, each with its own nuances. An LLM trained in general medical data might not have the detailed expertise needed for specific medical specialties.
Biases and ethical considerations
Beyond accuracy, there are ethical concerns, like the potential for LLMs to perpetuate biases in the training data. This could result in unequal care recommendations for different demographic groups.
For more details on the challenges of large language models in healthcare, you can check our articles on the risks of generative AI and generative AI ethics.
The future of LLMs in healthcare
Stanford’s analysis indicates that there is significant untapped potential for LLMs in healthcare.25
While many LLMs have been used for tasks such as augmenting diagnostics or patient communication, fewer have focused on administrative tasks that contribute to clinician burnout.
In the future, LLMs may evolve to interact with behavior, more context, and emotions, enabling them to provide more personalized and empathetic support.
Benchmark methodology
Benchmark methodology: This benchmark evaluates 9 popular general LLMs on graduate-level medical questions using the MedQA dataset, which draws its content from the United States Medical Licensing Examination (USMLE). Each question includes a clinical scenario and multiple-choice answer options.
LLM outputs: Each model was prompted to return a structured answer (e.g., “Answer: C”).26
Latency: The average time a model takes to generate a response to a single MedQA prompt. For example, if 100 questions take 1,115 seconds total to complete, the average latency is 11.15 seconds per question.
Benchmark data sources
Reference Links
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required.