Bud Runtime turned CPU-based LLM inference from impractical into production-ready — lifting throughput from 9 to 520 tokens per second on 4th Gen Intel® Xeon® Scalable Processors with AMX, through custom kernels and deep memory and attention optimization.
Earlier research from OpenAI — the Kaplan scaling laws — indicated that increasing model parameters generally improves the accuracy and reasoning of LLMs. As parameters grow, so do the demands for computational parallelization and high-bandwidth memory, posing a significant challenge to traditional CPU-based computing.
By mid-2023, new scaling laws such as Google’s Chinchilla law began to reshape the industry. They suggest that smaller models trained with an optimal number of tokens can outperform larger ones — demonstrated by Chinchilla and Meta’s LLaMA-2 70B.
Combined with architectural advances like Mixture-of-Experts and innovative training-data preparation, smaller, compute- and memory-efficient models proved more accurate and effective than larger counterparts — a trend validated by Phi, Vicuna, Mistral, Gemma, Mamba, MiniCPM, and the Bud Jr series.
When paired with efficient model compression and quantization, these models — small language models from 500 million to 10 billion parameters, and midsize LLMs from 10 to 40 billion — require significantly less computational parallelization and memory bandwidth. That is exactly what makes CPU inference viable.
Intel recognised this paradigm shift as an opportunity to reduce the total cost of ownership (TCO) and capital expenditure (CapEx) of deploying foundational models at scale on its infrastructure. But the existing software ecosystem was not built for CPU-based generative AI inference.
Inference runtimes and serving stacks such as OpenVINO, OneAPI, and IPEX excelled with classical and smaller neural networks — but were not designed for the demands of generative AI foundational models.
A general-purpose, production-ready inference runtime for generative AI on CPUs simply did not exist — so the cost advantage of CPU infrastructure stayed out of reach.
Bud Runtime enabled Intel and its customers with a production-ready LLM inference runtime, serving engine, and comprehensive end-to-end software stack — delivering a remarkable enhancement in LLM performance on CPUs.
Significant performance gains in CPU utilization for large language model inference — and proof that parallelized CPU inference scales.
The experiments demonstrate significant performance gains in CPU utilization for large language model inference, confirming that CPUs are a practical inference target for real-world applications.
Leveraging AMX on 4th Gen Intel® Xeon® Scalable Processors, the engine achieves an 18× increase in token generation per second for the bigcode/starcoderbase-3b model using 32 vCPUs and 100 parallel requests.
Token generation per second improves as the number of vCPUs increases — the engine scales vertically with the resources available to it.
Performance also rises with the number of requests. Higher vCPU counts require a larger volume of requests to fully optimize resource utilization.
Parallelized inference on CPUs effectively enhances throughput, making large language models more feasible for real-world applications.
By turning CPUs into a viable inference target, Bud Runtime lets enterprises run generative AI on infrastructure they already own — reducing the TCO and CapEx that previously kept production deployments out of reach.
The combination of custom CPU kernels, memory tiling and other memory optimizations, improved core utilization, KV decomposition, attention optimization, and function fusing collectively facilitated more efficient and scalable inference for LLMs — maximizing their operational capabilities on Intel’s infrastructure.
The study points to further research opportunities and optimizations to boost the efficiency of large language model inference on CPUs — extending vertical scaling, refining request-batching strategies, and broadening coverage across model families.
Parallelized inference on CPUs effectively enhances throughput, making large language models more feasible for real-world applications — and a more sustainable, cost-efficient route to production GenAI at scale.