Large language models (LLMs) are increasingly being applied in healthcare to support clinical tasks such as medical question answering, patient communication, and summarizing medical records. After analyzing on platforms like the Open Medical LLM Leaderboard and in peer-reviewed papers, I listed leading models in healthcare:1
Healthcare LLMs benchmark
Benchmark methodology: This benchmark evaluates the supervised fine-tuning performance of healthcare LLMs vs large general purpose models (GPT-4) on medical question answering tasks. See benchmark data sources.
MedQA:
Multiple-choice medical exam questions based on United States Medical Licensing Examination.
MedMCQA:
Large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions.
PubMedQA: Biomedical question-answering benchmark using yes/no/maybe answers.
Healthcare LLM examples
BERT-like (Encoder-only)
Optimized for encoding and representing biomedical text, these models excel at extracting features for tasks such as classification.
Model | Developer | Year | Parameters (B) | Open Source |
|---|---|---|---|---|
BioLinkBERT | — | 2022 | 0.34 | ✅ |
MedBERT | Stanford University | 2021 | 0.017 | ✅ |
Health Acoustic Representations (HeAR) | Google | 2024 | 0.31 | ❌ |
ChatGPT / LLaMA-like (Decoder, instruction/chat-tuned)
Based on LLaMA-style architectures and optimized for interactive tasks and clinical dialogues.
Model | Developer | Year | Parameters (B) | Open Source |
|---|---|---|---|---|
Polaris 3.0 | Hippocratic AI | 2025 | 4200 | ❌ |
MEDITRON-70B | EPFL (Swiss AI Lab) | 2023 | 70 | ✅ |
Me-LLaMA | PhysioNet (multi-institution) | 2024 | 70 | ✅ |
OpenBioLLM | – | 2024 | 70 | ✅ |
Radiology-Llama2 | Meta | 2023 | 70 | ✅ |
PMC-LLaMA | Shanghai AI Lab & SJTU | 2024 | 13 | ✅ |
ChatDoctor | UT Southwestern & collaborators | 2023 | 13 | ✅ |
Asclepius | KAIST & Yonsei Univ. | 2023 | 13 | ✅ |
MedAlpaca | Technical University of Munich | 2023 | 13 | ✅ |
Clinical Camel | University of Toronto (Vector Institute) | 2023 | 13 | ✅ |
GPT / PaLM-like (Decoder-only, generative)
Built similarly to GPT-3 or PaLM, these models are fine-tuned for general-purpose text generation and summarization.
Model | Developer | Year | Parameters (B) | Open Source |
|---|---|---|---|---|
Med-PaLM 2 | Google | 2023 | 340 | ❌ |
BioMedLM | Stanford CRFM (MosaicML) | 2022 | 2.7 | ✅ |
PubMedGPT | Stanford CRFM | 2023 | 2.7 | ✅ |
BioGPT | Microsoft Research | 2022 | 0.35 | ✅ |
General-purpose LLMs in healthcare
Key takeaways:
- o1 – Best performing model
- 03 mini – Best budget option
- GPT 4.1 – Best speed and response time
Benchmark methodology: This benchmark evaluates 9 popular general LLMs on graduate-level medical questions using the MedQA dataset, which draws its content from the United States Medical Licensing Examination (USMLE). Each question includes a clinical scenario and multiple-choice answer options.
LLM outputs: Each model was prompted to return a structured answer (e.g., “Answer: C”).2
Latency: The average time a model takes to generate a response to a single MedQA prompt. For example, if 100 questions take 1,115 seconds total to complete, the average latency is 11.15 seconds per question.
Fine-tuning medical LLMs
“Here, the performance of the default ChatGPT (4o model) is compared with the existing ‘Clinical Medicine Handbook’ assistant. Both models are given the same prompt, and their responses are analyzed:
GPT 4o
Fine-tuned medical LLM
Guides on fine-tuning LLMs
- Fine-tuning large language models for improved health communication5
- Fine-tuning large language models for specialized use cases6
For more: LLM fine-tuning and LLM training.
Use cases of general purpose LLMs
Model | Healthcare use case example | Method used | Open source |
|---|---|---|---|
GPT‑4 | RAG (Retrieval-Augmented Generation) | ❌ | |
Claude 3 | RAG + Prompt Engineering | ❌ | |
Qwen 3 | Continual pretraining + Fine-tuning | ✅ | |
Command R+ | RAG | ❌ | |
LLaMA 3 | Continual pretraining + Fine-tuning | ✅ |
These models are general fine-tuned models that need domain adaptation to perform clinical tasks accurately. You can use these models in healthcare by leveraging:
- Continual pretraining on medical data to help the model better identify medical language by exposing it to clinical notes and biomedical literature (like PubMed).
- RAG to pull data from verified clinical documents to produce accurate responses at runtime.
- Instruction fine-tuning to enable the model learn how to actually answer clinical questions or extract symptoms from text.
Source: MCP Digitalhealth12
Use cases of LLMs in clinical settings
1- Medical transcription
LLMs can help create medical transcriptions by:
- Listening to the organic dialogue between a patient and clinician
- Extracting important medical details
- Condensing medical data into compliant medical records that align with the relevant sections of an EHR
Real-life use case – Google’s MedLM can capture &transform the patient-clinician conversation into a medical transcription.13
2- Electronic health records (EHR) enhancement
The proliferation of electronic health records (EHR) has accumulated a vast repository of patient data, which, if mined effectively, can become a goldmine for healthcare improvement.
Real-life use case – Google’s MedLM is also used by BenchSci, Accenture, and Deloitte for electronic health records enhancement (EHR).
- BenchSci has integrated MedLM into its ASCEND platform to improve the quality of preclinical research.
- Accenture uses MedLM to organize unstructured data from numerous sources, automating human operations that were previously time-consuming and error-prone.
- Deloitte works with MedLM to minimize friction in finding treatment. They use an interactive chatbot that helps health plan participants better understand the provider alternatives.14
3- Clinical decision support
Large language models can summarize complex medical concepts allowing them to support valuable insights in the decision-making process.
Real-life use case – Memorial Sloan Kettering Cancer Center uses IBM Watson Oncology to assist oncologists by analyzing patient data and medical literature to recommend evidence-based treatment options.15
4- Medical research assistance
LLMs can parse and summarize vast amounts of data, can extract key findings from new research, providing synthesized insights. For example, one of the most famous LLMs, ChatGPT, is used for text summarization.
Real-life use case – John Snow’s healthcare chatbot helps researchers find relevant scientific papers, extract key insights, and identify research trends. It is particularly valuable for navigating the vast amount of biomedical literature.16
Real-life use case – TidalHealth Peninsula Regional clinicians used the Micromedex with Watson solution for healthcare research, claiming that, clinicians received their answers in less than one minute ~70% of the time.17
5- Automated patient communication
Large language models in healthcare can draft informative and compassionate responses to patients’ queries.
Some examples include:
- Medication management and reminders: A chatbot provides patients regular reminders to take their diabetic medication and requests confirmation.
- Health monitoring and follow-up care: A post-operative patient sends their pain and wound status to a chatbot, which determines if the healing process is progressing.
- Informational and educational communication: A patient asks a chatbot how to manage high blood pressure, and the chatbot responds with nutrition and lifestyle tips.
Real-life use case – Boston Children’s Hospital uses Buoy Health, an AI-driven online symptom checker chatbot, that provides patients with instant answers to health-related questions and initial consultations.
The chatbot can triage patients by analyzing their symptoms and advising whether they need to see a doctor.18
6- Predictive health outcomes
LLMs can assist in predictive analysis by discerning patterns within data.
Real-life use case – WVU Pharmacists using AI to reduce patient readmission rates: WVU pharmacists use a predictive algorithm to leverage LLMs to determine readmission risk. This approach will examine data from electronic health records (EHRs), which include patient demographics, clinical history, and socioeconomic determinants of health.
Based on this research, the WVU pharmacists identify patients at high risk of readmission and assign care coordinators to follow up with them after discharge. This can help reduce readmission rates.19
7- Personalized treatment plans
LLMs can suggest treatment plans tailored to an individual’s medical history and specific needs. Their ability to distill complex patient narratives into actionable insights can ensure that each patient receives a care plan that’s as unique as their health journey.
Real-life use case – Babylon Health: Babylon Health’s AI chatbot provides individualized health recommendations based on the user’s symptoms and medical history. It engages users in a conversation by asking relevant questions to analyze their issues better and giving tailored recommendations.20
8- Medical coding and billing
Large language models can automate audit processes by analyzing patient records and EHRs.
For example, Epic Systems, a major EHR provider, integrates LLMs into its software to assist with coding and billing. The LLMs can monitor for anomalies in access patterns to sensitive patient information or inconsistencies in coding and billing practices.21
However, LLMs are not ready for medical coding but promising: Researchers examined how frequently four LLMs (GPT-3.5, GPT-4, Gemini Pro, and Llama2-70b Chat) issued the correct CPT, ICD-9-CM, and ICD-10-CM codes.
Their findings show that there is a significant opportunity for improvement. Researchers discovered that LLMs frequently create codes that transmit inaccurate information, with a maximum accuracy of 50%.22
9- Training and education
Large language models and generative AI in general can be leveraged as interactive educational tools, elucidating complex concepts or offering clarifications on perplexing topics.
Real-life use case – Oxford Medical Simulation uses LLMs integrated with VR technology to create immersive virtual patient simulations.
These simulations allow students to experience high-pressure scenarios, such as handling a cardiac arrest patient without any real-world consequences.
The LLMs power the virtual patients’ responses, making them more realistic and unpredictable, preparing students for the variability of real clinical environments.23
Challenges of LLMs in healthcare
Privacy concerns
Using an LLM-based health application that has not been properly developed, tested, or approved for medical use could present significant risks to users. One of the primary concerns is related to privacy. LLMs and associated tools process health-related data input by users as part of their services. However, how this data is handled and whether these applications comply with data protection laws and principles remains uncertain.24
Accuracy and reliability
LLMs are also prone to hallucinations, plausible-sounding but incorrect or misleading information.
For example, when given a medical query, GPT-3.5 incorrectly recommended tetracycline for a pregnant patient, despite correctly explaining its potential harm to the fetus.25
Generalization vs. specialization
Healthcare encompasses a wide range of specialties, each with its nuances. An LLM trained in general medical data might not have the detailed expertise needed for specific medical specialties.
Biases and ethical considerations
Beyond accuracy, there are ethical concerns, like the potential for LLMs to perpetuate biases in the training data. This could result in unequal care recommendations for different demographic groups.
For more details on the challenges of large language models in healthcare, you can check our articles on the risks of generative AI and generative AI ethics.
The future of LLMs in healthcare
A Stanford’s analysis indicates that there is significant untapped potential for LLMs in healthcare.26
While many LLMs have been used for tasks like augmenting diagnostics or patient communication, fewer have focused on addressing the administrative tasks that contribute to clinician burnout.
In the future, LLMs may evolve to interact with behavior, more context, and emotions, enabling them to provide more personalized and empathetic support.
Benchmark data sources
Reference Links
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Comments 0
Share Your Thoughts
Your email address will not be published. All fields are required.