One decision in 7.4 milliseconds, on a laptop GPU: Bud Decision Runtime benchmarks

Bud Decision Runtime is an inference server that runs decision models on GPUs and CPUs behind one HTTP API. On the same machines it was faster than the publisher's own server, its fast path, PyTorch's compiler and vLLM in most tests, and on Laya its answers matched the publisher's original model on 99.8% of 1,800 labelled examples.

Bud Decision Runtime benchmarks: one Laya decision in 7.4 milliseconds over HTTP on an RTX 3050 Laptop GPU. At 48 clients it answers 46.2 requests per second against 17.2 for laya-serve, the publisher's server: 2.7 times as many.

Bud Decision Runtime is an inference server for decision models: models that read a piece of text, such as an email, a ticket or an agent's trace, and answer set questions about it (pick an option, give a score, yes or no). It runs four model families, Laya, Julia, GLiNER2.5-Decide and Kev, behind one HTTP API, on NVIDIA GPUs and on CPUs, and configures itself for the hardware it finds.

This post measures it against the engines you could serve these models with today: the publisher's own server and its hand-fused fast path, PyTorch's own compiler, and vLLM for the Kev models, which are built on Qwen. Same machines, same requests, same client. The runtime answers one Laya question in 7.4 ms over HTTP on an RTX 3050 Laptop GPU, is faster in most tests, and keeps its speed as clients are added. On Laya, its answers matched the publisher's original model on 99.8% of 1,800 labelled examples.

Measured 10 to 16 November 2026 on an RTX 3050 Laptop, a GTX 1650, a Xeon 6767P, a Core i7-11370H and an Apple M5, against vLLM 0.32. Every figure is the engine at its defaults; nothing was tuned by hand.

7.4 ms
one Laya question on a short state, over HTTP, on an RTX 3050 Laptop; median of 40 requests
1.39×
as fast as Laya's own TileLang fast path on the same GPU, one question; 6.8 against 9.4 ms, in-process
2.7×
the throughput of Laya's own server at 48 clients, on an RTX 3050 Laptop; 46.2 against 17.2 requests/s
3.5×
the throughput of Laya's own server at 16 clients, on a 16-thread Xeon; 39.7 against 11.5 requests/s
1.59×
PyTorch's best stock CPU path on a pass of real, mixed-length requests; 161 against 256 ms, Xeon, 16 threads
2.5×
vLLM's requests per second for one client on Kev-0.6B, on an RTX 3050 Laptop; 84.1 against 34.0/s, 11.8 against 29 ms
2.6×
vLLM's throughput at 16 clients on Kev-0.6B, on a 16-core Xeon; 78.5 against 30.7 requests/s
17×
vLLM's speed on five follow-up questions to a 2,300-token document, Kev-4B on a 16-core Xeon; 0.15 against 2.57 s
0.2%
of 1,800 labelled answers changed against the publisher's float32 model; accuracy 80.3% against 80.4%

Request sizes used throughout: short is a 30-word state with 1 question (about 50 tokens); typical is 60 words and 3 questions (about 240 tokens); long is 300 words and 8 questions (about 2,600 tokens). States from 1K tokens up to each model's full context have their own section.

What a decision runtime does

The text a decision model reads is called its state, and each question has a fixed answer type: pick one of these options, score this from 1 to 5, yes or no. Laya and Julia are ModernBERT encoders with an option-marker head, GLiNER2.5-Decide is a DeBERTa model, and Kev is a family of LoRA adapters with a pointer head on Qwen bases, from 0.5B to 9B parameters (Qwen2.5, Qwen3 and the hybrid Qwen3.5), that can keep a state's computation and reuse it for later questions.

The runtime serves all four behind one HTTP API. A Rust front end tokenises requests and packs their rows into passes; a device loop runs each pass on a GPU or CPU. Most of the speed comes from how passes are planned and how each pass is computed.

How it gets there

  • It decides its own configuration on the hardware it finds. At start the engine probes the device: which kernels pass a self-test there, which 16-bit type multiplies fastest (timed, not read from a table: a GTX 1650 reports a T4's compute capability but multiplies float16 at 0.16× float32), and which eight-bit engine is correct on this CPU. Nothing is tuned by hand: every number in this post is the engine at its defaults.
  • It sizes passes from measurement. At load it times passes of growing size and picks the shortest within 5% of the best throughput, and a larger one while a backlog waits. On a CPU a large model reads all its weights once per pass, so there the backlog pass may run well past the interactive target when the curve shows it pays: Kev-8B's went from 112 tokens to 1,024. Every pass size it will plan is captured as a CUDA graph before it reports ready.
  • Rows are packed, not padded. Requests of different lengths share a pass with no padding between them. Attention runs inside each row, so a row's answer is the same bits whatever shares its pass.
  • Multiplies are pinned and correctly rounded. One cuBLASLt algorithm defines each weight's bits; per band of pass sizes the engine uses the fastest algorithm that gives exactly those bits, timed on the card at load.
  • On a CPU, the small steps are compiled C++. A decoder layer's per-token steps (norms, rotations, SwiGLU), attention for short rows and Qwen3.5's delta rule run in C++ operators built on the host for its own instruction set at first start. As chains of PyTorch operators they cost more in dispatch than in arithmetic: a short Kev-0.6B request went from 59 to 20 ms a pass, and a Kev-0.8B pass over a 2,300-token document from 1,362 to 455 ms.
  • The CPU encoder is compiled. Laya's and Julia's encoder runs as one graph compiled by PyTorch's inductor with frozen, packed weights and attention inside the graph; one graph serves every pass size.
  • Queues are bounded by time and shared fairly. A request its deadline cannot survive is refused at once with a 503 and retry-after, each tenant queues apart, and a row asked again within a minute is answered from memory.

How we measured

Each engine served Laya on the same machine, driven by the same load generator: a fixed number of clients, each sending its next request as soon as the last is answered, every state new. Accuracy comes from one benchmark with six frozen labelled sets of 300 examples each (topic, emotion, sentiment, entailment, prompt injection, long reviews), scored against the publisher's own model in float32.

The publisher's engines ran in 16 bits as configured (autocast on the GPU, LAYA_CPU_AMP=bf16 on the Xeon). The runtime ran at its defaults, which chose bfloat16 on both.

Scaling with concurrent clients

A single request says little about a server. What matters is how throughput grows as clients are added, and what each of them waits. On typical requests:

  • The publisher's server does not scale. laya-serve answers one request at a time. Its throughput is flat from one client to 48 (17.2 requests a second on the GPU, 11.5 on the Xeon) and its latency grows in step with the queue, to 2.8 seconds at 48 clients.
  • The runtime's throughput rises with clients, and its latency is lower at every load. On the GPU it reaches 44 requests a second at four clients and 46 at 48, with a median wait of 75 ms at four clients against 232 ms for laya-serve. On the Xeon it goes from 20.8 at one client to 39.7 at 16.
Laya · RTX 3050 Laptop GPU, 16 bits · requests per second · higher is better
laya-serve · 1
17.2/s
Runtime · 1
36.2/s
laya-serve · 4
17.2/s
Runtime · 4
44.4/s
laya-serve · 16
17.2/s
Runtime · 16
46.3/s
laya-serve · 48
17.2/s
Runtime · 48
46.2/s
Figure 1 — typical requests (60 words, 3 questions) at 1, 4, 16 and 48 clients. laya-serve is flat at 17.2 requests a second; the runtime reaches 44.4 at four clients and holds 46.2 at 48, 2.7 times as many. Median latency for each point is in Table 1.
Laya · Xeon 6767P, 16 threads, bfloat16 on AMX · requests per second · higher is better
laya-serve · 1
9.6/s
Runtime · 1
20.8/s
laya-serve · 4
11.5/s
Runtime · 4
27.5/s
laya-serve · 16
11.5/s
Runtime · 16
39.7/s
Figure 2 — typical requests (60 words, 3 questions) at 1, 4 and 16 clients, each client sending back to back for 15 to 30 seconds. The runtime's Xeon runs shared the machine with a full Bud deployment, so its CPU figures are if anything conservative.
RequestClientslaya-serveRuntimeRuntime vs laya-serve
short136.6/s · 27 ms93.3/s · 10.7 ms2.5×
typical117.2/s · 58 ms36.2/s · 28 ms2.1×
typical417.2/s · 232 ms44.4/s · 75 ms2.6×
typical1617.2/s · 930 ms46.3/s · 289 ms2.7×
typical4817.2/s · 2.8 s46.2/s · 1.0 s2.7×
long13.2/s · 314 ms4.9/s · 206 ms1.5×
long43.2/s · 1.25 s5.0/s · 743 ms1.6×
long163.2/s · 5.0 s5.0/s · 2.5 s †1.6×
long483.2/s · 15.1 s5.0/s · 2.6 s †1.6×

Table 1 — RTX 3050 Laptop: requests answered per second and the median latency of answered requests. † The runtime refused the rest at once with a 503 (see "Past saturation" below). laya-serve was measured on 10 November 2026, the runtime on 12 November.

RequestClientslaya-serveRuntimeRuntime vs laya-serve
short114.2/s · 67 ms43.5/s · 21 ms3.1×
typical19.6/s · 101 ms20.8/s · 47 ms2.2×
typical411.5/s · 343 ms27.5/s · 141 ms2.4×
typical1611.5/s · 1.37 s39.7/s · 388 ms3.5×
long12.9/s · 338 ms4.3/s · 227 ms1.5×
long43.0/s · 1.31 s4.7/s · 813 ms1.6×

Table 2 — Xeon 6767P, 16 threads, same measures. laya-serve's long-request runs stopped at four clients, so the runtime's did too.

Past saturation

When more work arrives than a device can handle, a server has to either queue it or refuse some of it. laya-serve queues everything, so with 48 clients sending long requests a request waits 15 seconds. The runtime holds the work it accepts to what it can finish within its queue limit (two seconds of interactive work by default) and refuses the rest at once with a 503 and retry-after, which a client or a gateway can send to another replica.

It answers 1.6 times as many requests a second as laya-serve (5.0 against 3.2), and the ones it accepts wait 2.6 seconds instead of 15. In these runs the clients resend immediately after a refusal, so the refusal counts measure the load generator, not lost work.

In the decision benchmark's overload test, 256 clients sending typical requests, the runtime still answered 57.8 requests a second on the RTX 3050 (median 2.2 s) and 55.8 on the Xeon (2.7 s): throughput holds far past the point where every client could be served at once.

Against vLLM, on the Kev models

Kev's models are LoRA adapters on Qwen base models, which vLLM already serves. If a general engine serves them as fast, there is no reason to use a separate runtime for them. So we put each Kev model's base behind vLLM 0.32 on the same machine and sent the same requests.

vLLM takes prompts, not decision requests. Kev's model answers each question looking only at the state and at that question, so to get Kev's answers a client sends vLLM one prompt per question, the state followed by that question, all at once, and vLLM's prefix cache shares what it can of the state between them. The runtime takes the request as it is: it computes the state once, and each question attends to the state and to itself only.

vLLM has no pointer head, so our wrapper reads one next token per prompt instead, slightly less work than Kev's readout; only its speed counts. Both engines ran in bfloat16. On the Xeon each got the same 16 cores of one memory node, with the load generator on others. Each point is 30 seconds of closed-loop load at that many clients, every request new, measured 15 and 16 November 2026.

  • The runtime is ahead at every load, on both machines. It answers 1.75 to 7 times as many requests a second as vLLM with one client, and 1.26 to 6.9 times as many with 4 to 64. A typical Kev-0.6B request takes 11.8 ms on the RTX 3050 against 29 ms, and 27 ms on the Xeon against 70.
  • The gap is widest on Qwen3.5. vLLM's prefix cache shares a Qwen3.5 state only in whole blocks of 544 tokens on the GPU and 640 on the CPU, longer than these states, so it computes the state again for every question. On Kev-0.8B's long requests the runtime serves 7 times as many with one client on the Xeon, and 5.6 times as many at 16 clients on the RTX 3050.
  • It is closest under heavy load on the dense models. There both engines spend each pass on the same multiplies, and vLLM shares most of a dense model's state between questions: 1.26 times on Kev-0.6B's long requests at 64 clients on the RTX 3050, and 1.51 times on Kev-8B's at 64 clients on the Xeon.
Kev · RTX 3050 Laptop GPU, bfloat16 · 16 clients, typical requests · requests per second · higher is better
vLLM · 0.6B
74.6/s
Runtime · 0.6B
96.4/s
vLLM · 0.8B
34.2/s
Runtime · 0.8B
74.3/s
Figure 3 — Kev-0.6B (Qwen3) and Kev-0.8B (Qwen3.5) at 16 clients. On the dense Qwen3 model the runtime answers 1.29 times as many requests; on Qwen3.5, where vLLM cannot share these short states between questions, 2.17 times.
Kev · Xeon 6767P, 16 cores, bfloat16 · 16 clients, typical requests · requests per second · higher is better
vLLM · 0.6B
30.7/s
Runtime · 0.6B
78.5/s
vLLM · 0.8B
16.7/s
Runtime · 0.8B
50.5/s
Figure 4 — the same models and load on 16 cores of one memory node: 2.55 times as many requests on Kev-0.6B and 3.03 times on Kev-0.8B.
Machine · modelClientsRuntimevLLMRuntime vs vLLM
RTX 3050 Laptop
Kev-0.6B (Qwen3)
184.1/s · 12 ms34.0/s · 29 ms2.47×
488.4/s · 45 ms52.9/s · 76 ms1.67×
1696.4/s · 166 ms74.6/s · 214 ms1.29×
64102.6/s · 661 ms81.3/s · 787 ms1.26×
RTX 3050 Laptop
Kev-0.8B (Qwen3.5)
158.2/s · 17 ms18.7/s · 58 ms3.11×
465.1/s · 61 ms27.4/s · 146 ms2.38×
1674.3/s · 215 ms34.2/s · 466 ms2.17×
6478.9/s · 860 ms34.0/s · 1.7 s2.32×
Xeon 6767P, 16 cores
Kev-0.6B (Qwen3)
135.2/s · 27 ms13.1/s · 70 ms2.69×
444.7/s · 87 ms21.1/s · 186 ms2.12×
1678.5/s · 203 ms30.7/s · 519 ms2.55×
6488.8/s · 779 ms33.7/s · 1.9 s2.63×
Xeon 6767P, 16 cores
Kev-0.8B (Qwen3.5)
125.9/s · 37 ms8.62/s · 106 ms3.01×
438.6/s · 102 ms13.4/s · 291 ms2.87×
1650.5/s · 311 ms16.7/s · 937 ms3.03×
6452.2/s · 1.2 s20.9/s · 3.0 s2.50×
Xeon 6767P, 16 cores
Kev-8B (Qwen3)
13.50/s · 283 ms1.42/s · 657 ms2.46×
45.00/s · 791 ms2.10/s · 1.9 s2.38×
167.75/s · 2.0 s3.66/s · 4.3 s2.12×
647.78/s · 8.2 s4.12/s · 15.2 s1.89×

Table 3 — typical requests: requests answered per second and the median latency of answered requests, bfloat16 for both engines. The ratio is the runtime's requests per second over vLLM's. No request failed in any run.

Machine · modelClientsRuntimevLLMRuntime vs vLLM
RTX 3050 Laptop
Kev-0.6B (Qwen3)
121.7/s · 46 ms12.4/s · 82 ms1.75×
423.3/s · 172 ms17.4/s · 230 ms1.34×
1625.3/s · 632 ms19.9/s · 802 ms1.27×
6425.3/s · 2.5 s20.1/s · 3.2 s1.26×
RTX 3050 Laptop
Kev-0.8B (Qwen3.5)
117.4/s · 57 ms3.16/s · 318 ms5.51×
418.3/s · 218 ms3.57/s · 1.1 s5.13×
1620.0/s · 801 ms3.56/s · 4.6 s5.61×
6419.9/s · 3.2 s3.56/s · 17.7 s5.60×
Xeon 6767P, 16 cores
Kev-0.6B (Qwen3)
111.4/s · 85 ms6.19/s · 146 ms1.85×
415.2/s · 256 ms8.52/s · 460 ms1.79×
1618.5/s · 850 ms9.86/s · 1.6 s1.87×
6418.3/s · 3.4 s9.96/s · 6.4 s1.84×
Xeon 6767P, 16 cores
Kev-0.8B (Qwen3.5)
111.4/s · 85 ms1.63/s · 592 ms7.00×
412.9/s · 302 ms1.88/s · 2.1 s6.87×
1613.5/s · 1.2 s2.02/s · 7.8 s6.67×
6413.5/s · 4.7 s2.03/s · 30.1 s6.65×
Xeon 6767P, 16 cores
Kev-8B (Qwen3)
11.58/s · 623 ms0.70/s · 1.3 s2.26×
41.91/s · 2.1 s1.02/s · 3.8 s1.87×
161.89/s · 8.4 s1.24/s · 12.6 s1.52×
641.90/s · 32.1 s1.26/s · 44.0 s1.51×

Table 4 — long requests (300 words, 8 questions), same measures as Table 3.

All six Kev models on a CPU

Kev's family runs from 0.5B to 9B parameters on three kinds of Qwen base. On the 16-core Xeon we measured all six; Kev-4B, 8B and 9B do not fit the 4 GB laptop GPU. The runtime is 1.26 to 3.4 times as fast on a new request, 1.9 to 17.6 times as fast on follow-up questions to a document it has seen, and serves 1.46 to 6.1 times as many requests at 16 clients.

Model (base)One typical request2,300-token document, 5 questions5 more questions on it16 clients, typical16 clients, long
Kev-0.5B (Qwen2.5)31 / 59 ms229 / 369 ms46 / 88 ms91.7 / 40.926.2 / 11.4
Kev-0.6B (Qwen3)29 / 72 ms333 / 468 ms54 / 122 ms74.5 / 29.517.5 / 10.0
Kev-0.8B (Qwen3.5)39 / 99 ms0.42 / 1.42 s38 / 641 ms50.4 / 17.112.5 / 2.04
Kev-4B (Qwen3.5)151 / 410 ms1.97 / 5.10 s0.15 / 2.57 s11.4 / 5.142.85 / 0.54
Kev-8B (Qwen3)288 / 656 ms2.34 / 2.94 s277 / 707 ms7.39 / 3.551.77 / 1.21
Kev-9B (Qwen3.5)297 / 715 ms2.66 / 8.72 s0.26 / 4.31 s6.82 / 3.151.69 / 0.32

Table 5 — runtime / vLLM, bfloat16, Xeon 6767P with 16 cores, 15 and 16 November 2026. Latency is the median of one client at a time; the last two columns are requests answered per second. The runtime's memory of answers was off, so nothing here was answered from memory. "5 more questions" asks new questions about a document the engine has seen: the runtime keeps the document's state at any length, while vLLM's prefix cache works only in whole blocks, 640 tokens long on Qwen3.5 on a CPU.

On the GPU, the same comparison

Model (base)One typical request2,300-token document, 5 questions5 more questions on it16 clients, typical16 clients, long256 clients, typical
Kev-0.5B (Qwen2.5)10.7 / 24.7 ms177 / 212 ms15.6 / 42.6 ms120.2 / 88.230.9 / 25.7125.9 / 97.8
Kev-0.6B (Qwen3)11.9 / 30.4 ms250 / 284 ms19.4 / 56 ms96.6 / 73.625.1 / 20.3101.2 / 81.4
Kev-0.8B (Qwen3.5)17.1 / 58.4 ms252 / 340 ms19 / 103 ms74.8 / 34.220.1 / 3.678.4 / 34.1

Table 6 — runtime / vLLM, bfloat16, RTX 3050 Laptop 4 GB, 13 and 14 November 2026, the runtime's memory of answers off.

The one load vLLM wins

Identical requests sent again with the runtime's memory of answers turned off, on the small GPU. vLLM's prefix cache then also holds a repeated prompt's question tokens, while the runtime's state cache holds the state only: on Kev-0.6B at 64 clients over 8 repeated requests, vLLM answers 132.6 requests a second against the runtime's 108.5. On the Xeon the runtime wins this load too, by 3 times. With the memory on, which is the default, the runtime answers repeated requests about 30 to 1,800 times as fast as vLLM.

Longer states, up to each model's full context

The requests above carry states of 50 to 2,600 tokens. Decision models also read whole contracts, long tickets and agent traces, so we sent each engine one new state at a time, from 1K tokens (K = 1,024) up to the model's full context: 32K for the Qwen2.5 and Qwen3 models and 256K (262,144 tokens) for Qwen3.5. Each request asks five questions about its state, and five other questions on the same state follow it.

The states are the benchmark's long reviews, end to end, numbered so that no two are alike. Lengths stopped growing once a new request took more than two minutes; one request at the full context then showed whether the engine could serve it at all. One client at a time, bfloat16, on the same GPU and the same 16 Xeon cores as above.

  • Follow-up questions are faster at every length. The runtime keeps a state's computation at any length, so five more questions cost little. On the RTX 3050 it answers them 2.0 to 2.8 times as fast as vLLM on the dense models and 4.8 to 18 times on Kev-0.8B. On the Xeon it is 6.4 to 27 times as fast on the three Qwen3.5 models and up to 3.5 times on the dense ones, level only on a 4K Kev-0.5B state (62 ms against 63). At 32K on Kev-0.6B on the Xeon the questions take 247 ms against vLLM's 501.
  • New Qwen3.5 states are faster on the Xeon at every length, and at the longest on the GPU. On the Xeon a new Kev-0.8B state is 4.5 times as fast at 1K and 1.69 times at the full 256K (195 s against 329); Kev-4B 4.4 times at 1K and 1.61 times at 256K (617 s against 996); Kev-9B 3.9 and 1.66 times (748 s against 1,245). On the RTX 3050 Kev-0.8B is 2.8 times as fast at 1K, level at 8K, behind by 6 to 10% at 16K and 32K, and 1.74 times as fast at 64K (19.7 s against 34.2).
  • On long dense states, vLLM computes a new state faster. The runtime leads on Qwen2.5 and Qwen3 states up to 2K tokens on the GPU and 4K to 8K on the Xeon. Beyond that vLLM leads, by up to 1.15 times on the GPU (Kev-0.6B at 8K: 1.44 s against 1.66) and 1.11 times on the Xeon (Kev-0.5B at 32K: 10.4 s against 11.6; Kev-8B: 67.9 s against 70.6). At these lengths attention is most of the work, and vLLM's attention kernels are faster than the ones the runtime uses there; on the CPU that is PyTorch's flash attention.
  • Both engines reach the full context on the Xeon. The runtime computes a state longer than 8,192 tokens in segments (4,096 on a GPU of 8 GB or less), each continuing what the earlier ones kept, so its state cache sets the longest state, not the memory of one pass. vLLM computes a long prompt a few thousand tokens at a time. On the 4 GB GPU, Kev-0.8B reached 64K on both engines, Kev-0.5B its full 32K on both, and Kev-0.6B 10K on the runtime and 8K on vLLM.
Machine · modelFull contextLongest served, runtime / vLLMA new state that long, runtime / vLLMWhere it stopped
RTX 3050 Laptop
Kev-0.5B (Qwen2.5)
32K32K / 32K6.6 / 6.4 sFull context on both
RTX 3050 Laptop
Kev-0.6B (Qwen3)
32K10K / 8K2.3 / 1.4 sRuntime: the state cache the card holds (1,200 MB, 112 KB a token). vLLM: 16K refused, longer than the 12,288-token max-model-len the card holds
RTX 3050 Laptop
Kev-0.8B (Qwen3.5)
256K64K / 64K19.7 / 34.2 sRuntime: the state cache the card holds (880 MB). vLLM: 65,536 refused, the most max-model-len the card holds
Xeon 6767P
Kev-0.5B (Qwen2.5)
32K32K / 32K11.6 / 10.4 sFull context on both
Xeon 6767P
Kev-0.6B (Qwen3)
32K32K / 32K17.6 / 16.2 sFull context on both
Xeon 6767P
Kev-0.8B (Qwen3.5)
256K256K / 256K195 / 329 sFull context on both
Xeon 6767P
Kev-4B (Qwen3.5)
256K256K / 256K617 / 996 sFull context on both
Xeon 6767P
Kev-8B (Qwen3)
32K32K / 32K70.6 / 67.9 sFull context on both
Xeon 6767P
Kev-9B (Qwen3.5)
256K256K / 256K748 / 1,245 sFull context on both

Table 7 — how far each engine went, and the time for one new state of that length. On the 4 GB GPU memory sets the limit: vLLM got the longest max-model-len it would accept on the card, and the runtime a state cache within the half of free memory it allows. On the Xeon each run could use 64 GB.

Machine · model1K4K8K16K32K64K128K256K
RTX 3050 Laptop
Kev-0.5B (Qwen2.5)
87 ms
115 ms
396 ms
370 ms
914 ms
853 ms
2.4 s
2.2 s
6.6 s
6.4 s
RTX 3050 Laptop
Kev-0.6B (Qwen3)
115 ms
142 ms
602 ms
553 ms
1.7 s
1.4 s
RTX 3050 Laptop
Kev-0.8B (Qwen3.5)
136 ms
384 ms
523 ms
722 ms
1.1 s
1.2 s
2.7 s
2.6 s
7.0 s
6.4 s
19.7 s
34.2 s
Xeon 6767P
Kev-0.5B (Qwen2.5)
129 ms
231 ms
453 ms
713 ms
1.3 s
1.5 s
3.8 s
3.5 s
11.6 s
10.4 s
Xeon 6767P
Kev-0.6B (Qwen3)
176 ms
315 ms
695 ms
841 ms
2.0 s
1.9 s
5.5 s
5.1 s
17.6 s
16.2 s
Xeon 6767P
Kev-0.8B (Qwen3.5)
240 ms
1.1 s
812 ms
2.1 s
1.9 s
4.2 s
4.0 s
7.8 s
9.3 s
16.0 s
22.8 s
39.2 s
63.2 s
106 s
195 s
329 s
Xeon 6767P
Kev-4B (Qwen3.5)
932 ms
4.1 s
3.6 s
7.8 s
7.3 s
15.2 s
15.8 s
28.1 s
34.3 s
56.6 s
80.1 s
132 s
210 s
–
617 s
996 s
Xeon 6767P
Kev-8B (Qwen3)
1.4 s
2.0 s
4.5 s
5.5 s
11.6 s
11.7 s
27.6 s
26.5 s
70.6 s
67.9 s
Xeon 6767P
Kev-9B (Qwen3.5)
1.8 s
7.1 s
5.2 s
12.9 s
10.4 s
24.5 s
23.0 s
44.6 s
48.6 s
88.4 s
105 s
194 s
263 s
–
748 s
1,245 s

Table 8 — a new state with five questions, one client: the runtime on top, vLLM below in grey. Each figure is the faster of two requests at that length (one beyond 32K); the first request at a new length can carry costs a running server pays once.

Machine · model1K4K8K16K32K64K128K256K
RTX 3050 Laptop
Kev-0.5B (Qwen2.5)
13 ms
36 ms
20 ms
50 ms
30 ms
69 ms
50 ms
114 ms
94 ms
207 ms
RTX 3050 Laptop
Kev-0.6B (Qwen3)
15 ms
30 ms
27 ms
75 ms
41 ms
96 ms
RTX 3050 Laptop
Kev-0.8B (Qwen3.5)
18 ms
314 ms
24 ms
283 ms
26 ms
126 ms
37 ms
191 ms
56 ms
416 ms
97 ms
529 ms
Xeon 6767P
Kev-0.5B (Qwen2.5)
47 ms
81 ms
62 ms
63 ms
67 ms
107 ms
85 ms
174 ms
130 ms
309 ms
Xeon 6767P
Kev-0.6B (Qwen3)
57 ms
103 ms
60 ms
134 ms
84 ms
186 ms
136 ms
298 ms
247 ms
501 ms
Xeon 6767P
Kev-0.8B (Qwen3.5)
52 ms
734 ms
43 ms
655 ms
46 ms
1.1 s
52 ms
1.2 s
71 ms
999 ms
188 ms
2.1 s
331 ms
3.7 s
609 ms
3.9 s
Xeon 6767P
Kev-4B (Qwen3.5)
169 ms
3.2 s
156 ms
2.6 s
161 ms
4.4 s
187 ms
4.2 s
241 ms
2.8 s
448 ms
5.4 s
855 ms
–
Xeon 6767P
Kev-8B (Qwen3)
288 ms
1.0 s
283 ms
788 ms
331 ms
988 ms
450 ms
881 ms
601 ms
1.2 s
Xeon 6767P
Kev-9B (Qwen3.5)
287 ms
5.6 s
270 ms
4.3 s
269 ms
6.4 s
311 ms
6.2 s
332 ms
4.4 s
635 ms
6.5 s
824 ms
–

Table 9 — five more questions on that state: the runtime on top, vLLM below in grey. 32K is 32,512 tokens on the Qwen2.5 and Qwen3 models, which leaves room for the questions in their 32,768; 256K is 261,888 tokens; vLLM's 64K on the GPU is 65,280. A blank cell is a length that engine did not reach or was not asked; on Kev-4B and Kev-9B vLLM went from 64K straight to the full context, because a new request at 64K had taken more than two minutes.

Against Laya's own fast path, on the same GPU

Laya's publisher ships a fast path for CUDA (laya[fast]): TileLang kernels that fuse the multiplies with the activation, norms and rotary positions, run with CUDA graphs. Its published 4.6 ms for one question was measured on an RTX 4070 Ti SUPER. We installed it on our RTX 3050 Laptop and timed both in the same process, on the same requests. The runtime's time covers everything its engine does per request apart from HTTP: tokenising in Rust, the copy to the GPU, the captured graph and the copy back.

RequestRuntimeLaya stock, bf16Laya fast, bf16Runtime vs fast path
1 question, short state6.8 ms23.9 ms9.4 ms1.39×
3 questions, short state16.7 ms40.0 ms23.1 ms1.39×
8 questions, long state290 ms446 ms335 ms1.16×
30 questions, short state122 ms184 ms167 ms1.38×

Table 10 — RTX 3050 Laptop, 4 GB, medians of 40 calls after warm-up. "Short state" is two sentences; "long state" is 28. The fast path in fp16 measured 9.5, 23.3, 341 and 170 ms on the same four requests, within 2% of bf16. It serves one request at a time; the concurrency comparison above uses the publisher's server.

Against PyTorch's compiler, on a CPU

On a CPU the strongest general alternative is PyTorch itself: Hugging Face's ModernBERT-large compiled by inductor with its weights frozen, which is what Intel recommends for CPU inference. We timed Laya's encoder each way on the same threads, in turn, over five rounds, on rows of one length and on a pass of real requests of mixed lengths.

Laya's encoder · one pass of real, mixed-length requests · Xeon 6767P · ms · lower is better
PyTorch eager
329
PyTorch compiled
256
Runtime
161
Figure 5 — eight real requests (states of 30 to 300 words, one to three questions): 12 rows and 1,276 tokens, which the runtime packs and a batch pads to 3,864. bfloat16 on AMX, 16 threads, median of five interleaved rounds. "Compiled" is inductor with frozen weights.

Packing is where the runtime wins. On passes of rows that are all one length, which traffic rarely sends, there is no padding to remove, and the stock compiled model keeps a 5 to 10% lead on the larger ones.

PassPyTorch eagerPyTorch compiledRuntime
Real requests, mixed lengths (1,276 tokens)329.4 ms256.4 ms161.4 ms
64 tokens, one short row49.0 ms17.3 ms16.6 ms
1,024 tokens, eight rows of one length111.7 ms88.3 ms97.6 ms
3,584 tokens, eight 448-token rows309.2 ms243.8 ms257.0 ms

Table 11 — one encoder pass on the Xeon 6767P, 16 threads, bfloat16 on AMX, median of five interleaved rounds.

On every machine

The same engine, at its defaults, on each machine and model family we could reach. On each one it chose its own precision and kernels.

Machine · model, precision chosenShortTypicalLongAnswers changed
RTX 3050 Laptop, 4 GB
Laya, bfloat16
7.4 ms20.4 ms208 ms0.2%
RTX 3050 Laptop
Julia, bfloat16
2.2 ms4.2 ms35 ms1.9%
RTX 3050 Laptop
GLiNER2.5-Decide, bfloat16
7.2 ms11.1 ms48 ms0.3%
RTX 3050 Laptop
Kev, bfloat16
7.2 ms10.3 ms37 ms1.0%
GTX 1650, 4 GB, no tensor cores
Laya, float32
44 ms129 ms1.70 s0%
Xeon 6767P, 16 threads, AMX
Laya, bfloat16
24.6 ms45.8 ms232 ms0.2%
Xeon 6767P, 16 threads
Julia, bfloat16
25.6 ms35.8 ms61 ms2.1%
Core i7-11370H, 4 threads, no AMX
Laya, float32
183 ms494 ms6.3 s0%
Core i7-11370H, 4 threads
Laya, int8 (opt-in)
66 ms206 ms3.3 s0.8%

Table 12 — median latency of one caller at a time, from the decision benchmark. "Answers changed" is the share of 1,800 labelled examples whose winning option differs from the publisher's float32 model; Julia's and GLiNER's references are the engine's own float32, which matches their libraries within 0.0001. Laya's throughput on typical requests was 13,421 tokens a second on the RTX 3050 and 12,322 on the Xeon. The engine also runs on an Apple M5, where it chose bfloat16 on the GPU and float32 on the CPU; we did not benchmark it there.

Against published figures

Laya's publisher reports single-question latency on several devices. None of them is a machine we had, so the table below puts each figure on its own hardware rather than claiming a like-for-like win. Our short request (30 words, 1 question, about 51 tokens) is close to the publisher's 72-token case.

EngineDeviceSourceOne question
Laya fast pathRTX 4070 Ti SUPERpublished4.6 ms
Bud Decision Runtime, bfloat16, over HTTPRTX 3050 Laptop, 4 GBmeasured7.4 ms
Bud Decision Runtime, bfloat16Xeon 6767P, 16 threadsmeasured24.6 ms
Laya, model cardTesla T4published39.5 ms
Bud Decision Runtime, float32GTX 1650, 4 GBmeasured44.2 ms
Bud Decision Runtime, int8Core i7-11370H, 4 threadsmeasured66.3 ms
Bud Decision Runtime, float32Core i7-11370H, 4 threadsmeasured182.8 ms
LayaRyzen 9 6900HX, 8 threadspublished329 ms
Laya, float32EPYC 9R14, 4 corespublished580 ms

Table 13 — median milliseconds for one question on a short state; each row is on its own hardware. Published figures are from Laya's BENCHMARKS.md: the model card's T4 figure, its TileLang fast path on an RTX 4070 Ti SUPER (a card with several times the 3050 Laptop's compute), and CPU figures in float32.

The publisher's 4.6 ms on an RTX 4070 Ti SUPER is lower than our 7.4 ms, on a card with about 3.5 times the memory bandwidth, which is what bounds a single question. Measured on the same card (Table 10), the runtime is 1.39 times as fast as that fast path. On CPUs the gap is wide: a 4-core EPYC 9R14 takes 580 ms in float32, and our 4-thread laptop i7 takes 183 ms in float32 and 66 ms in int8.

On GPUs the best published throughput for a model of this size is text-embeddings-inference on an H100, at about 38% of the card's nominal speed. The runtime turns 74% of the RTX 3050's measured multiply ceiling into Laya tokens. We have not yet run it on an H100.

Accuracy against the original models

A faster engine is only useful if the answers stay the same. On Laya's six labelled sets the runtime in 16 bits scores 80.4% on the RTX 3050 and 80.3% on the Xeon, against 80.4% for the publisher's float32 model, and changes 0.2% of winning options.

We also compared each family against its publisher's own code, on requests none of the benchmark sets contain. In 16 bits the runtime is at least as close to float32 as the publisher's own 16-bit path for Laya, GLiNER2.5-Decide and Kev. Julia is the exception on one measure: its mean change is lower than the library's, but it flips more winners.

Familyfloat32, largest difference16-bit mean change, runtime / libraryWinners changed, runtime / library
Laya0.000050.0018 / 0.00181.7% / 2.2%
Julia0.000050.0090 / 0.00953.0% / 2.5%
GLiNER2.5-Decide0.0000010.0004 / 0.00060.0% / 0.9%
Kev0.000010.0026 / 0.00391.0% / 3.1%

Table 14 — each family against its publisher's own library; probabilities as a client receives them, on an RTX 3050. Julia is the least confident of the four (65.5% on the benchmark), so near-ties flip. On NVIDIA GPUs the runtime gives a request the same bits whatever shares its pass.

The faster CPU path was held to the same rule. On the Xeon, all six Kev models were checked against Kev's own answers recorded on that machine by Kev's library. In float32 every one is within 0.00006 and no winning option changed. In bfloat16 every one is closer to Kev's float32 answers than Kev's own bfloat16 is, in mean and 99th percentile.

Kev model, Xeon 6767Pfloat32, largest differencebfloat16 mean change, runtime / Kev's ownbfloat16 99th percentile, runtime / Kev's own
Kev-0.5B (Qwen2.5)0.0000130.0072 / 0.00970.043 / 0.053
Kev-0.6B (Qwen3)0.0000520.0034 / 0.00550.027 / 0.031
Kev-0.8B (Qwen3.5)0.0000090.0018 / 0.00280.009 / 0.018
Kev-4B (Qwen3.5)0.0000010.0012 / 0.00210.004 / 0.008
Kev-8B (Qwen3)0.0000090.0038 / 0.00820.021 / 0.057
Kev-9B (Qwen3.5)0.0000010.0016 / 0.00280.007 / 0.011

Table 15 — 454 questions for the three smaller models and 244 for the larger ones. One Kev-0.8B question, one that magnifies 16-bit drift, moved by 0.068 in bfloat16 in one run against a bound of 0.055, so on a CPU the engine serves Kev-0.8B in float32 by default, where it fits.

Serve your decision models

Bring the models, the hardware and the latency target. We will show you the runtime on your own machines, at its defaults.

Request a demo

What surprised us

The 16-bit multiply was not correctly rounded

The algorithm search picked split-K kernels for Laya's attention-output and feed-forward shapes, which add partial sums in bfloat16. Measured against float32, that was 4 to 13.5 units in the last place. Excluding split-K brought every shape to correct rounding (0.5 units) and put the runtime's answers closer to float32 than the publisher's own 16-bit path. It also showed that most algorithms without split-K give identical bits, so the engine can pick the fastest of them per pass size without breaking the rule that a row's answer never depends on its pass.

Compute capability does not say which precision is fast

A GTX 1650 reports the same capability as a T4 but has no tensor cores: it multiplies float16 at 0.16× and bfloat16 at 0.88× the speed of float32. The engine times each type at start instead of trusting the number, and runs that card in float32.

Eight-bit multiplies overflow on older x86 CPUs

On x86 processors without VNNI, PyTorch's oneDNN and fbgemm engines sum pairs of eight-bit products in 16 bits, and the sums saturate: 13% error on a Comet Lake i5, against 1.6% with seven-bit inputs. The engine now checks its own eight-bit path at start, picks the input width that is correct, and refuses int8 where it would not be faster than float32.

More CPU executors do not mean more throughput

Four executors of four threads do 1.3 to 2.6 times the work of one executor of sixteen at the same pass size. At the same pass time, which is what a latency target fixes, they do the same work, a single request takes twice as long, and the idle ones slow the busy one by a quarter. One executor is the default.

The right CPU baseline is PyTorch's compiler, not a published figure

We had aimed for about 1,100 tokens a second per core, derived from Intel's published BERT-large figures. Intel's table gives sentences a second but no sequence length, so no per-token figure follows from it. The comparison that holds is the one above: PyTorch's stock compiled model on the same machine.

An answer cache needs load that does not repeat itself

Our load generator seeded each client the same way on every run, and the benchmark's warm-up sent the same requests as the timed run. With answers remembered, half of one run came from memory. Both now draw new requests every time, except the test that is meant to repeat.

A latency cap cut throughput for a large model on a CPU

The engine caps a pass at 300 ms by default so that a request arriving mid-pass does not wait long. Kev-8B on 16 Xeon cores reads its 16 GB of weights once per pass, so a 300 ms pass held 112 tokens, one request: under 16 clients it served 3.2 short requests a second. Passes of 1,024 tokens take 1.2 s but serve 829 tokens a second instead of 392. The engine now takes such passes while a backlog waits, where the curve it measures shows a fifth more throughput, and served 5.9.

The compiler said the loop was vectorised; it was not

The CPU delta rule needs exp on every channel of every token. GCC's report said the loop was vectorised, but the machine code called the scalar expf once per element: 3 ns an element, and 9 ns for a hand-written polynomial. Written on the compiler's vector types at the machine's own width, the same polynomial takes 1 ns. The report is not the code; read the assembly.

vLLM's prefix cache needs whole blocks on Qwen3.5

On Qwen3.5 a block must also hold a page of the delta rule's state, so vLLM's blocks are 544 tokens on the GPU and 640 on the CPU. A state shorter than one block is never cached, which is most decision requests, so vLLM computes it again for every question; a longer one has its last partial block recomputed for every question. The runtime keeps a state by its exact tokens at any length, which is where its lead of 17 times on follow-up questions to a Qwen3.5 model on the CPU comes from.

Measured 10 to 16 November 2026. Load runs used one load generator against each engine's HTTP endpoint, including vLLM 0.32's concurrency sweeps; accuracy and single-caller latency come from the runtime's decision benchmark, whose per-run result files, including example-by-example probabilities, are saved as baselines. Published figures for Laya are from its BENCHMARKS.md; the H100 embedding figure is from RunPod's text-embeddings-inference benchmark.

Get the next one by email

Product releases, benchmarks, and deployment patterns. Monthly, one email, unsubscribe any time.

Unsubscribe any time.

BN
Written by
Bud Newsroom
Bud Ecosystem
Get started with Bud Decision Runtime

Measure it on your hardware.

Every number above is the engine at its defaults. Bring your decision models, your machines and your latency target, and we will run the same benchmark on them.

01 Name the decision models, the traffic and the latency each answer has to meet.
02 The runtime probes your GPUs and CPUs and picks its own precision and kernels.
03 Compare throughput, latency and accuracy against the engine you serve with today.