A head-to-head evaluation of three inference engines — vLLM, TGI, and Bud Runtime — serving Mistral 7B (7.3B parameters) on an NVIDIA A100 80GB GPU. Bud Runtime delivered the highest throughput and the lowest execution time, making it the strongest fit for production GenAI.
This case study examines the inference performance of Mistral 7B — a large language model with 7.3 billion parameters — to assess its viability for production-ready Generative AI. We benchmarked it across three inference engines: vLLM, TGI, and Bud Runtime.
Key performance metrics — input and output token throughput and execution time — reveal significant differences among the engines. Bud Runtime emerged as the most efficient, offering the highest throughput and the lowest execution time, making it ideal for production environments. These insights emphasise the critical role of inference-engine selection in optimising the deployment of LLMs like Mistral 7B.
Despite the promise of large language models, the journey from proofs of concept to full-scale production remains a significant challenge. A primary hurdle is the high cost of LLM inference — driven by the computational power required, which typically necessitates GPUs.
The infrastructure for GPU-based inference can be prohibitively expensive, limiting broader adoption of GenAI. Many enterprises stall at the pilot stage, unable to justify the investment for production deployment. This underscores the need for cost-effective alternatives that deliver comparable performance without the financial burden of GPU resources.
If CPU-based inference can reach performance comparable to GPU-based solutions, it could significantly cut operating costs and let far more organisations run LLMs in production. This benchmark establishes the GPU baseline against which that viability is measured.
Mistral 7B is a large language model featuring 7.3 billion parameters. It demonstrates exceptional performance across a variety of benchmarks — notably outperforming Llama 2 13B on all evaluated metrics and matching or exceeding Llama 1 34B in many areas.
The model is particularly strong in code-related tasks, closely rivalling CodeLlama 7B while maintaining robust performance in English language tasks. Two architectural features carry that efficiency:
Mistral 7B’s performance is validated through a comprehensive evaluation pipeline spanning commonsense reasoning, world knowledge, reading comprehension, mathematics, and coding — positioning it as a leading choice for enterprises seeking powerful, efficient LLM solutions.
We evaluated the inference performance of Mistral 7B using three prominent LLM inference engines — vLLM, TGI, and Bud Runtime — on an NVIDIA A100 GPU with 80 GB of memory, chosen for its capability in handling large language models efficiently.
We focused on several key metrics: total input tokens, total output tokens, input token throughput (tokens/sec), output token throughput (tokens/sec), and overall execution time (seconds). Together these give a comprehensive view of each engine’s performance and efficiency during inference.
Input and output token throughput and execution time, measured across vLLM, TGI, and Bud Runtime on the NVIDIA A100.
Bud Runtime emerged as the standout performer — achieving the highest input token throughput at 5,584.06 tokens/sec and an output token throughput of 2,621.05 tokens/sec, with an execution time of just 4.88 seconds.
| Inference engine | Input tok/s | Output tok/s | Execution |
|---|---|---|---|
| Bud Runtime | 5,584.06 | 2,621.05 | 4.88 s |
| TGI | 3,898.79 | 1,843.35 | 6.91 s |
| vLLM | 3,826.98 | 1,816.49 | 7.05 s |
The highest input throughput (5,584.06 tok/s) and output throughput (2,621.05 tok/s), plus the lowest execution time (4.88 s), make Bud Runtime ideal where rapid response and high throughput matter — the scaling and responsiveness that production environments demand.
Both engines can effectively run Mistral 7B, but neither matches the efficiency and speed of Bud Runtime under identical conditions.
The analysis highlights the critical role inference-engine selection plays in maximising LLM performance. Organisations deploying Mistral 7B in production should weigh these findings: choosing Bud Runtime can lead to improved operational efficiency and better user experiences through its superior throughput and execution time — a foundation for extending the same efficiency to lower-cost hardware.