Bud Case Study
Bud Runtime · vLLM · TGI · GPU Inference Benchmark

Benchmarking Mistral 7B inference on GPUs

A head-to-head evaluation of three inference engines — vLLM, TGI, and Bud Runtime — serving Mistral 7B (7.3B parameters) on an NVIDIA A100 80GB GPU. Bud Runtime delivered the highest throughput and the lowest execution time, making it the strongest fit for production GenAI.

7.3B
Model parameters
5,584 t/s
Peak input throughput
2,621 t/s
Peak output throughput
4.88s
Fastest execution
Prepared byBud Ecosystem Inc.
HardwareNVIDIA A100 · 80 GB
ModelMistral-7B-Instruct-v0.3
DateNovember 2024
Benchmarking Mistral 7B Inference Performance on GPUs Bud
Overview

The inference engine decides production economics.

This case study examines the inference performance of Mistral 7B — a large language model with 7.3 billion parameters — to assess its viability for production-ready Generative AI. We benchmarked it across three inference engines: vLLM, TGI, and Bud Runtime.

Key performance metrics — input and output token throughput and execution time — reveal significant differences among the engines. Bud Runtime emerged as the most efficient, offering the highest throughput and the lowest execution time, making it ideal for production environments. These insights emphasise the critical role of inference-engine selection in optimising the deployment of LLMs like Mistral 7B.

From pilot to production

Despite the promise of large language models, the journey from proofs of concept to full-scale production remains a significant challenge. A primary hurdle is the high cost of LLM inference — driven by the computational power required, which typically necessitates GPUs.

The infrastructure for GPU-based inference can be prohibitively expensive, limiting broader adoption of GenAI. Many enterprises stall at the pilot stage, unable to justify the investment for production deployment. This underscores the need for cost-effective alternatives that deliver comparable performance without the financial burden of GPU resources.

The deeper question

If CPU-based inference can reach performance comparable to GPU-based solutions, it could significantly cut operating costs and let far more organisations run LLMs in production. This benchmark establishes the GPU baseline against which that viability is measured.

Bud Ecosystem · Case Study02
Benchmarking Mistral 7B Inference Performance on GPUs Bud
The Model

Mistral 7B.

Mistral 7B is a large language model featuring 7.3 billion parameters. It demonstrates exceptional performance across a variety of benchmarks — notably outperforming Llama 2 13B on all evaluated metrics and matching or exceeding Llama 1 34B in many areas.

7.3B
Parameters
GQA
Grouped-Query Attention
SWA
Sliding Window Attention
> 13B
Outperforms Llama 2 13B

The model is particularly strong in code-related tasks, closely rivalling CodeLlama 7B while maintaining robust performance in English language tasks. Two architectural features carry that efficiency:

  • Grouped-Query Attention (GQA) — an innovative mechanism that enhances inference speed, making the model more efficient for practical applications.
  • Sliding Window Attention (SWA) — a technique that enables the model to effectively process longer sequences while reducing computational cost.

Mistral 7B’s performance is validated through a comprehensive evaluation pipeline spanning commonsense reasoning, world knowledge, reading comprehension, mathematics, and coding — positioning it as a leading choice for enterprises seeking powerful, efficient LLM solutions.

Bud Ecosystem · Case Study03
Benchmarking Mistral 7B Inference Performance on GPUs Bud
The Benchmark

How we measured.

We evaluated the inference performance of Mistral 7B using three prominent LLM inference engines — vLLM, TGI, and Bud Runtime — on an NVIDIA A100 GPU with 80 GB of memory, chosen for its capability in handling large language models efficiently.

Model
mistralai/Mistral-7B-Instruct-v0.3
7.3B-parameter instruction-tuned model with GQA and SWA.
Inference engines compared
vLLM · TGI · Bud Runtime
Three engines run identically and measured side by side on the same hardware.
vLLMTGIBud Runtime
Hardware
NVIDIA A100 · 80 GB
A single high-memory GPU, sized to serve a 7.3B model without partitioning.
Figure 1. The benchmark setup. One model, one GPU, three inference engines — isolating engine choice as the only variable.

We focused on several key metrics: total input tokens, total output tokens, input token throughput (tokens/sec), output token throughput (tokens/sec), and overall execution time (seconds). Together these give a comprehensive view of each engine’s performance and efficiency during inference.

Bud Ecosystem · Case Study04
Benchmarking Mistral 7B Inference Performance on GPUs Bud
The Evidence

Benchmark Results

Input and output token throughput and execution time, measured across vLLM, TGI, and Bud Runtime on the NVIDIA A100.

Benchmarking Mistral 7B Inference Performance on GPUs Bud
Key Results

Bud Runtime leads on every metric.

Bud Runtime emerged as the standout performer — achieving the highest input token throughput at 5,584.06 tokens/sec and an output token throughput of 2,621.05 tokens/sec, with an execution time of just 4.88 seconds.

Output token throughput

Bud Runtime
2,621
TGI
1,843
vLLM
1,816

Full results

Inference engineInput tok/sOutput tok/sExecution
Bud Runtime5,584.062,621.054.88 s
TGI3,898.791,843.356.91 s
vLLM3,826.981,816.497.05 s
Bud Ecosystem · Case Study05
Benchmarking Mistral 7B Inference Performance on GPUs Bud

What the numbers mean

Bud Runtime · the standout

The highest input throughput (5,584.06 tok/s) and output throughput (2,621.05 tok/s), plus the lowest execution time (4.88 s), make Bud Runtime ideal where rapid response and high throughput matter — the scaling and responsiveness that production environments demand.

  • TGI. Competent but short of Bud Runtime — 3,898.79 tok/s input and 1,843.35 tok/s output, at 6.91 s execution.
  • vLLM. Slightly lower again — 3,826.98 tok/s input and 1,816.49 tok/s output, at 7.05 s execution.

Both engines can effectively run Mistral 7B, but neither matches the efficiency and speed of Bud Runtime under identical conditions.

Bottom line

The analysis highlights the critical role inference-engine selection plays in maximising LLM performance. Organisations deploying Mistral 7B in production should weigh these findings: choosing Bud Runtime can lead to improved operational efficiency and better user experiences through its superior throughput and execution time — a foundation for extending the same efficiency to lower-cost hardware.

Bud Ecosystem · Case Study06