Performance

LocalMaxxing results

LocalMaxxing times local inference on fixed canonical prompts (reasoning-v1, code-v1) with a fresh nonce per request, one warm-up and five timed runs, and reports the median. Our two RTX 3090 runs were submitted on 2026-10-03 and marked approved and verified. The RTX 5060 Ti runs are local measurements with the same harness and have not been submitted.

Settings: 32K context, thinking on, temperature 0, 320 output tokens. RTX 3090: DFlash2 Q4_K_M drafter with 7 draft tokens and tree verify, BF16 KV, 350 W. RTX 5060 Ti (Windows 11 / WSL 2): DFlash2-TQ3 drafter, tq3 KV. For context: when we submitted, the top RTX 3090 entries were 242.1 (reasoning-v1) and 169.6 (code-v1) tok/s, EXL3 4.0 bpw with a DFlash2 tree; on 2026-10-07 the board's best Qwen3.8-27B entry for the RTX 5060 Ti was 63.2 tok/s. Board positions change, so check the live board.

Qwen3.8-Flash-Next on one RTX 3090

Qwen3.8-Flash-Next (125B, 6B active, mixture of experts) does not fit in 24 GB, so the routed experts live in system RAM and the GPU keeps a cache of expert slots: misses go over PCIe by the copy engine, the next layer's experts are prefetched while the current layer runs, and native MTP drafts up to 5 tokens per step, stopping when the draft is unsure. Measured against Strata 0.2.2 (Niko1221, engine 0.1.47) on the same RTX 3090, same file, one session.

Short chats (5 x 512 tokens, each the first request of a fresh server): tqLLM 91.6 tok/s, Strata 89.6; first token 0.32-0.41 s against 1.07-1.29 s. On a warm server the same chats run at 95.6 tok/s. Up to 8 users at once (--np 8) get the same text as one user alone. Model file for both: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, IQ3_S; tqLLM drafts with mtp-blk48-iq4.gguf, Strata with --spec 4; greedy, reasoning off.

RTX 3090: prefill and decode

Model Qwen3.8-27B-TQ3_4S-v2.gguf, drafter Qwen3.8-27B-DFlash2-TQ3.gguf, tq3 KV cache, greedy, 400 output tokens, context 40960. Prompts are WikiText-2 test text cut to length with a "summarise" instruction; each number is the median of 3 runs, timed by the server.

A second RTX 3090 that also drives a desktop display reproduced decode within 1 % and prefill 4-15 % lower.

RTX 3090: decode up to 256K context

One server with 262,144 tokens of context, one request at a time, a C++ code prompt grown to each size. Decode falls as the attention reads more KV, and stays above 100 tok/s at 256K.

Same file on three GPUs

LocalMaxxing's canonical reasoning and code prompts, one file and drafter on every card. Not submitted to the LocalMaxxing board.

Reasoning prompts

Drafter speed follows how often its guesses are accepted. Summaries of encyclopedia text accept about 35-40 % of the drafted tokens; reasoning text accepts more.

Long prompts: --draft-retrieve 32 lets the drafter also read 32 far passages of the prompt. On an RTX 3090 at 64K context, 31K / 40K / 51K-token prompts go from 158 / 143 / 137 to 176 / 156 / 148 tok/s.

Long context on 16 GB (TqCache)

RTX 5060 Ti 16 GB, Qwen3.8-27B-TQ_XS.gguf + DFlash2 drafter, tq3 KV cache. With the default context the memory fit caps the context at 204,800 on this card and keeps all KV on the GPU. With --context 262144, TqCache keeps the oldest positions in system RAM, and the output stays exactly the same.

Reading KV over PCIe costs speed, so ask for more context than the default only when the prompts need it.

MoE on RTX 3090: MiniCPM-V 4.7 vs llama.cpp

MiniCPM-V 4.7 (35B-A3B, mixture of experts), supported from 1.0.0. The Q5_K_M file is 24.7 GB and does not fit in the card's 24 GB. tqLLM keeps the least-routed experts in pinned host memory (--moe-host-mb 2000, about 0.15 % of the routing) and the output is identical with and without the split. llama.cpp's best setting that survives 32K context is -ncmoe 2 (-ncmoe 1 crashes at 32K). The IQ3_S file fits, so both engines keep every expert in VRAM.

MoE on DGX Spark: MiniCPM-V 4.7 vs llama.cpp

MiniCPM-V 4.7 (35B-A3B, mixture of experts), supported from 1.0.0. GB10 (DGX Spark), target MiniCPM-V-4_7-35B-A3B-IQ3_S.gguf, drafter dspark-MiniCPM-V-4_7-35B-A3B-F16.gguf (openbmb's DSpark), greedy, 512 output tokens, thinking off, both engines in the same session. llama.cpp DSpark uses its default 3 drafts (5 and 8 were slower); tqLLM uses chain verify with 3 drafts, and its output is identical to its own no-drafter run on all three prompts.

Context: 32K to 1M on one RTX 3090

Qwen3.8-27B runs every context size from 32K to 1M on one 24 GB RTX 3090, with exact recall of planted facts: the memory fit makes room step by step and TqCache keeps the oldest KV in system RAM only past 512K. Decode: 104.9 tok/s at 32K, 75.2 at 256K, 49.6 at 512K, 21.4 at 768K, 7.7 at 1M (exact recall). Full charts, what the fit did at each size, and the 1M command: Features: long context on one GPU.

By GPU

RTX 5060 Ti 16 GB (WSL 2), drafter on, one run each, 16K context chosen by the memory fit: 2K prompt 777 tok/s prefill / 94.5 tok/s decode, 8K prompt 821 / 76.9.

Why the drafter is exact

Speculative decoding is only worth it if it never changes the answer. In tqLLM, plain decoding and every verify path (token tree, single chain, MTP) share one attention arithmetic, so greedy output is identical token for token. This is checked by the test-tree, test-dflash-exact and test-native-mtp tests and on long prompts up to 51K tokens.

Reproduce

sh bench/get_wikitext.sh                       # bench/data/wiki.test.raw (checksum-verified)
# native build: starts tqserve itself, without and with the drafter
python3 bench/relbench.py --bin build/tqserve --gpu "RTX 3090" --wiki bench/data/wiki.test.raw --out results.jsonl \
  --model v2=models/Qwen3.8-27B-TQ3_4S-v2.gguf \
  --drafter models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf --vocab models/mtp-vocab/atx_65536.txt
# or against a running Docker server (start it with -e KV_FORMAT=tq3 to match)
python3 bench/relbench.py --url http://localhost:8086 --mode dflash2 --gpu "RTX 3090" \
  --wiki bench/data/wiki.test.raw --out results.jsonl --model v2=models/Qwen3.8-27B-TQ3_4S-v2.gguf

Each line of results.jsonl records the GPU, the model file, the settings and the engine commit the server reports in GET /health.