Bud Case Study
Bud Runtime × Intel · CPU Inference Optimization

Enhancing LLM inference on Intel CPUs

Bud Runtime turned CPU-based LLM inference from impractical into production-ready — lifting throughput from 9 to 520 tokens per second on 4th Gen Intel® Xeon® Scalable Processors with AMX, through custom kernels and deep memory and attention optimization.

520 t/s
CPU throughput, up from 9
~58×
Overall throughput gain
18×
Token gen on StarCoder-3B
AMX
4th Gen Intel® Xeon®
Prepared byBud Ecosystem Inc.
Hardware4th Gen Intel® Xeon® · AMX
EngineBud Runtime
DateNovember 2024
Enhancing LLM Inference Performance on Intel CPUs Bud
Overview

Smaller models changed the math.

Earlier research from OpenAI — the Kaplan scaling laws — indicated that increasing model parameters generally improves the accuracy and reasoning of LLMs. As parameters grow, so do the demands for computational parallelization and high-bandwidth memory, posing a significant challenge to traditional CPU-based computing.

By mid-2023, new scaling laws such as Google’s Chinchilla law began to reshape the industry. They suggest that smaller models trained with an optimal number of tokens can outperform larger ones — demonstrated by Chinchilla and Meta’s LLaMA-2 70B.

0.5–10B
Small language models
10–40B
Midsize LLMs
2023
Chinchilla scaling shift
MoE
Mixture-of-Experts gains

Combined with architectural advances like Mixture-of-Experts and innovative training-data preparation, smaller, compute- and memory-efficient models proved more accurate and effective than larger counterparts — a trend validated by Phi, Vicuna, Mistral, Gemma, Mamba, MiniCPM, and the Bud Jr series.

When paired with efficient model compression and quantization, these models — small language models from 500 million to 10 billion parameters, and midsize LLMs from 10 to 40 billion — require significantly less computational parallelization and memory bandwidth. That is exactly what makes CPU inference viable.

Bud Ecosystem · Case Study02
Enhancing LLM Inference Performance on Intel CPUs Bud
The Problem

CPUs were ready. The software was not.

Intel recognised this paradigm shift as an opportunity to reduce the total cost of ownership (TCO) and capital expenditure (CapEx) of deploying foundational models at scale on its infrastructure. But the existing software ecosystem was not built for CPU-based generative AI inference.

Inference runtimes and serving stacks such as OpenVINO, OneAPI, and IPEX excelled with classical and smaller neural networks — but were not designed for the demands of generative AI foundational models.

Why the legacy stack fell short

  • Generative models — LLMs, MLLMs, diffusion, and I-JEPA — can have billions of parameters and rely on complex attention mechanisms and key-value caches.
  • They demand significant computational parallelization, high-bandwidth memory, and substantial memory overhead.
  • The methodologies of the time required model-specific optimizations — quantization and pruning — that often compromised performance and accuracy.
The gap

A general-purpose, production-ready inference runtime for generative AI on CPUs simply did not exist — so the cost advantage of CPU infrastructure stayed out of reach.

Bud Ecosystem · Case Study03
Enhancing LLM Inference Performance on Intel CPUs Bud
The Solution

From 9 to 520 tokens per second.

Bud Runtime enabled Intel and its customers with a production-ready LLM inference runtime, serving engine, and comprehensive end-to-end software stack — delivering a remarkable enhancement in LLM performance on CPUs.

9 t/s
Starting throughput
520 t/s
With Bud Runtime
~58×
Improvement
6
Core optimizations
Bud Runtime
520 t/s
Legacy CPU stack
9 t/s
Figure 1. CPU token-generation throughput, before and after Bud Runtime — a roughly 58× gain on the same Intel hardware.

What made the jump

Custom CPU kernels
Kernels optimized specifically for CPU execution.
Memory tiling
Tiling and other memory optimizations.
Improved core utilization
Fuller use of available CPU cores.
KV decomposition
Key-value cache decomposition.
Attention optimization
Reworked attention computation.
Function fusing
Fused operations to cut overhead.
Bud Ecosystem · Case Study04
Enhancing LLM Inference Performance on Intel CPUs Bud
The Evidence

Key Results

Significant performance gains in CPU utilization for large language model inference — and proof that parallelized CPU inference scales.

Enhancing LLM Inference Performance on Intel CPUs Bud
Key Results

Parallelized CPU inference scales.

The experiments demonstrate significant performance gains in CPU utilization for large language model inference, confirming that CPUs are a practical inference target for real-world applications.

18×
Token gen increase (StarCoder-3B)
32
vCPUs
100
Parallel requests
AMX
On 4th Gen Intel® Xeon®
1

Performance improvement

Leveraging AMX on 4th Gen Intel® Xeon® Scalable Processors, the engine achieves an 18× increase in token generation per second for the bigcode/starcoderbase-3b model using 32 vCPUs and 100 parallel requests.

2

Vertical scaling

Token generation per second improves as the number of vCPUs increases — the engine scales vertically with the resources available to it.

3

Impact of request volume

Performance also rises with the number of requests. Higher vCPU counts require a larger volume of requests to fully optimize resource utilization.

4

Scalability and practicality

Parallelized inference on CPUs effectively enhances throughput, making large language models more feasible for real-world applications.

Bud Ecosystem · Case Study05
Enhancing LLM Inference Performance on Intel CPUs Bud

A practical path to lower-cost inference

What it unlocks

By turning CPUs into a viable inference target, Bud Runtime lets enterprises run generative AI on infrastructure they already own — reducing the TCO and CapEx that previously kept production deployments out of reach.

The combination of custom CPU kernels, memory tiling and other memory optimizations, improved core utilization, KV decomposition, attention optimization, and function fusing collectively facilitated more efficient and scalable inference for LLMs — maximizing their operational capabilities on Intel’s infrastructure.

Future research directions

The study points to further research opportunities and optimizations to boost the efficiency of large language model inference on CPUs — extending vertical scaling, refining request-batching strategies, and broadening coverage across model families.

Bottom line

Parallelized inference on CPUs effectively enhances throughput, making large language models more feasible for real-world applications — and a more sustainable, cost-efficient route to production GenAI at scale.

Bud Ecosystem · Case Study06