The effectiveness of large language models (LLMs) is determined not only by their accuracy and capabilities but also by the speed at which they engage with users.
We benchmarked the performance of leading language models across various use cases, measuring their responsiveness to user input. From the time it takes to generate the first word to the pace of the full response, the results reveal which models deliver faster interactions when it counts.
LLM latency benchmark results
We benchmarked the latency performance of the following large language models: GPT-4.1, Mistral-large, Claude-3-opus-20240229, Grok-2, and DeepSeek.We focus on two key metrics: First Token Latency, which is the time it takes for the model to start generating the first token of a response, and Per Token Latency, which is the time taken to generate each token throughout the response.
Explore our methodology to learn how we calculated these results.
Performance analysis by use case
The results show that model performance varies by task type. Two metrics are most relevant:
- First Token Latency, or time to first token, is defined as the time from request initiation to receiving the first response token, a crucial metric for assessing how quickly a model begins to respond.
- Latency per Token is the average time a model takes to generate each token after it starts responding. It reflects the model’s generation speed during output, which is critical for real-time use cases.
Across use cases, Grok consistently delivers the fastest initial response, while GPT-4 achieves the lowest per-token latency. Mistral generally occupies the middle ground, Claude performs slower, and DeepSeek shows the longest delays.
Q&A
In Q&A tasks such as customer support and virtual assistants, responsiveness is critical because users expect immediate answers.
- Grok: fastest start (0.345s) and efficient per-token rate (0.015s). Well-suited for live support.
- GPT-4: slower start (0.615s) but strong per-token latency (0.026s), making it efficient for longer answers.
- Mistral: moderate first token (0.495s) but higher per-token latency (0.041s). Best for short queries rather than long outputs.
- Claude: slow initial response (1.162s) with moderate per-token speed (0.049s). Initial delay may hinder usability.
- DeepSeek: longest delay (2.270s) and relatively high per-token latency (0.060s). Not suitable for time-sensitive Q&A.
Summary generation
Summarization benefits from both a quick start and efficient throughput, especially in cases such as summarizing call transcripts.
- Mistral: fastest first token (0.551s) but moderate per-token rate (0.029s). Best for short documents.
- GPT-4: competitive first token (0.589s) and the lowest per-token latency (0.021s). Performs well with longer texts.
- Grok: slightly slower start (0.594s) but efficient per-token rate (0.023s). Good balance for mixed-length summaries.
- Claude: slower first token (1.298s) and higher per-token latency (0.047s). Acceptable for non-urgent use.
- DeepSeek: significantly slower (3.942s first token, 0.068s per-token). Unsuitable where fast turnaround is needed.
Multi-source synthesis
This task requires combining information from multiple inputs, where both responsiveness and sustained speed matter.
- Grok: fastest start (0.374s) and quick per-token rate (0.017s). Suitable for real-time dashboards.
- GPT-4: moderate start (0.566s) but efficient per-token rate (0.024s). Suitable for more extended synthesis.
- Mistral: slightly faster start (0.520s) than GPT-4 but slower per-token rate (0.037s). Suitable for shorter synthesis.
- Claude: slower initial response (1.540s) with moderate per-token rate (0.045s). Throughput is acceptable, but responsiveness is limited.
- DeepSeek: slowest start (2.834s) and highest per-token latency (0.073s). Not appropriate for real-time tasks.
Language translation
Translation tasks benefit from low latency at both stages, especially in interactive or live settings.
- Grok: fastest start (0.354s) and low per-token rate (0.017s). Effective for real-time translation.
- GPT-4: slower start (0.766s) but best per-token rate (0.014s), making it efficient for longer passages.
- Mistral: moderate start (0.558s) with slower per-token rate (0.042s). Works for short sentences but is less efficient for long texts.
- Claude: slower start (1.191s) with similar per-token latency (0.046s) to Mistral. Adequate for non-live translation.
- DeepSeek: slowest overall (2.427s start, 0.067s per-token). Poor fit for translation under time pressure.
Business analysis
In business analysis, both responsiveness and steady throughput are important, depending on whether the use case involves live monitoring or batch reporting.
- Grok: fastest start (0.351s) and efficient per-token rate (0.017s). Well-suited for real-time analysis.
- GPT-4: moderate start (0.576s) with efficient per-token rate (0.026s). Suitable for reporting where near real-time is sufficient.
- Mistral: middle ground (0.529s start, 0.040s per-token). Best for routine tasks where balance is acceptable.
- Claude: slower response (1.368s start, 0.047s per-token). Works for batch reviews but is less suited for real-time dashboards.
- DeepSeek: slowest start (2.425s) and high per-token latency (0.072s). Limited use in timely analysis.
Coding
Coding tasks require sustained generation speed for longer outputs as well as prompt initial response.
- DeepSeek: slowest start (2.369s) and highest per-token rate (0.078s). Not suitable for coding environments that require rapid feedback.
- Grok: fastest start (0.344s) with competitive per-token rate (0.022s). Performs well across coding tasks.
- GPT-4: slightly slower start (0.561s) but fastest per-token rate (0.021s). A strong option for complex or lengthy outputs.
- Mistral: balanced start (0.502s) and moderate per-token rate (0.035s). Adequate for everyday coding.
- Claude: slower start (1.173s) and higher per-token latency (0.062s). Less suited for interactive coding tasks.
What is LLM latency, and why is it important?
LLM latency refers to the time it takes for a large language model (LLM) to generate a response after receiving a prompt, a key component in LLM performance benchmarking and real-world application responsiveness. Key performance metrics such as latency to first token and per-token latency are crucial in this evaluation.
Low latency is crucial for delivering smooth, real-time user experiences, especially in applications like chatbots, coding assistants, customer support, and translation tools, where inference performance directly affects user satisfaction. High latency can lead to frustration, reduced engagement, and lower user satisfaction.
As LLMs become integrated into more products and services, optimizing latency is key to maintaining performance, responsiveness, and overall product success.
Factors that affect LLM latency
The response time of a large language model (LLM) can vary significantly depending on a few key factors. Understanding these helps identify where improvements can be made.
- Model Size: Larger models typically require more processing power, which can lead to increased latency. More parameters often mean better results, but also slower response times. This tradeoff is a central concern in performance benchmarking efforts.
- Hardware Capabilities: The type of GPU or TPU used, available memory, memory bandwidth, and system architecture all play a role. High-performance hardware with high system throughput and efficient resource utilization can significantly reduce latency and improve LLM inference efficiency.
- Batch Size: Processing multiple requests in a batch can be efficient, but it may also delay the response for individual users, especially for the first token. Knowing how many requests are being processed simultaneously is essential for accurate latency measurements.
- Network Latency: In distributed systems or cloud environments, communication between components can introduce delays. The physical location of servers also matters.
Strategies to reduce LLM latency
Reducing latency is critical for improving the user experience, particularly in real-time applications. Here are a few effective strategies:
- Model Optimization: Techniques like quantization, pruning, and using distilled (smaller, faster) versions of models can speed up response times and improve latency without sacrificing accuracy. Additionally, these optimizations enhance cost efficiency by reducing computational resource demands, thereby improving overall inference performance.
- Improved Infrastructure: Running models on optimized inference engines and high-speed hardware (like modern GPUs or custom accelerators) can make a big difference. Additionally, deploying LLMs on infrastructure designed for high throughput ensures that more requests can be processed concurrently without sacrificing latency.
- Prompt Design: Crafting shorter, more efficient prompts reduces the number of input tokens the model has to process, saving both time and compute resources. Reducing unnecessary tokens, particularly by minimizing input sequence length and controlling how many tokens are used, also improves inter-token latency.
- Streaming & Caching: Streaming & Caching: Delivering the output as it’s being generated (streaming) and caching frequent responses can help minimize perceived delays, especially when generating long sequences with many output tokens.
LLM latency benchmark methodology
We benchmarked the response latency performance of various large language models (LLMs) across different use cases, focusing on two key metrics: First Token Latency (time to first token) and Latency per Token. We tested the following large language models: GPT-4.1, Mistral-large, Claude-3-opus-20240229, Grok-2, and DeepSeek. Model names and versions corresponded to those specified by their respective API providers at the time of testing.
We used the same input prompt, input size, and same model configuration for each use case to ensure a fair and consistent comparison. To minimize the impact of transient network delays and server load variations, we avoided sending concurrent requests during testing.
We accessed each model through its official API, provided by its respective developer or platform (e.g., OpenAI, Anthropic, xAI, etc.), ensuring that performance measurements, especially the time to first token, reflected real-world usage conditions.
For each use case, we prepared 10 distinct questions to represent realistic variations in input. We executed all requests by running each question 10 times per model to minimize the impact of transient network delays and variations in server load. We reported the median latency values to reduce the influence of outliers.
Before measurements, we sent a warm-up request with an empty or neutral input to mitigate cold-start effects such as cache loading or connection setup.
We measured latency using streaming response mode, enabling precise timing of the time to first token and per token latency throughout the response, which together provide an approximation of the overall end-to-end latency.
1. Q&A
We benchmarked the models using a set of 10 distinct questions designed to represent a variety of common factual and conceptual topics across technical, business, and general knowledge domains. These inputs averaged around 13 tokens per prompt, making them relatively short and concise.
This use case evaluates the models’ ability to generate clear, accurate, and informative answers suitable for educational, documentation, and customer support contexts. The required responses typically involve moderate-length explanations that balance detail with clarity.
2. Coding
We evaluated the models using a set of 10 distinct programming tasks of varying complexity, ranging from simple functions to more advanced API development. These tasks involved generating Python code snippets such as basic scripts, web applications using Flask or FastAPI, and data processing scripts.
This use case assesses the models’ ability to produce structured, functional, and coherent code that often requires longer and more complex outputs compared to typical text generation. The input prompts averaged around 20 tokens each, reflecting concise but descriptive programming requests.
3. Language translation
We benchmarked the models using a set of 10 diverse translation prompts covering multiple languages (Spanish, Chinese, Russian) and text types, including long academic passages, short everyday sentences, scientific abstracts, business emails, and literary excerpts. These inputs varied significantly in length and complexity, ranging from short sentences of around 10 tokens to detailed multi-paragraph texts exceeding several hundred tokens.
This use case evaluates the models’ ability to accurately comprehend and faithfully reproduce meaning across different languages and domains, preserving nuances, style, and technical content. By using varied text types and lengths, we tested both general translation quality and the models’ handling of specialized or formal language.
4. Business analysis
We evaluated the models using 10 distinct business analysis prompts, each simulating real-world decision-making scenarios across domains like sales performance, customer retention, supply chain bottlenecks, marketing ROI, employee productivity, and competitive strategy. The prompts included structured tabular data and open-ended analytical questions, requiring models to interpret multiple business metrics and generate concise, actionable insights. Inputs varied in complexity, with an average input length of approximately 105 tokens.
This use case tests a model’s ability to synthesize quantitative data, apply logical reasoning, and communicate recommendations clearly in a business context.
5. Multi-source synthesis
We assessed the models’ performance by providing 10 distinct prompts, each containing five customer reviews with varying and sometimes contradictory opinions about a product. The tasks required the models to analyze individual reviews in detail, highlight important positives, negatives, and contradictions, then synthesize these insights into a balanced overall evaluation. Finally, models were asked to produce actionable advice by listing critical factors prospective buyers should consider. The prompts featured a mix of qualitative feedback on packaging, delivery, product performance, customer service, and value, with an average input length of approximately 135 tokens.
This use case tests a model’s capability to comprehend nuanced, multi-source user feedback and generate clear, structured, and practical summaries.
6. Summary generation
We tasked models with producing academic-style summaries (~500 tokens) of technical articles on diverse topics such as AI in healthcare, climate change, renewable energy, blockchain, remote work, electric vehicles, cybersecurity, social media, urbanization, and quantum computing. Each summary was structured into main arguments, supporting ideas, and conclusions, with key terms highlighted and briefly explained.
This use case tests a model’s capability to comprehend detailed, technical articles and generate clear, structured, and academically styled summaries with key term explanations.
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required.