Features - tqLLM

Long context on one GPU: memory fit + TqCache

One 24 GB card, one model, every context size from 32K to 1M, with exact recall of facts planted at the start and the middle. Two parts of tqLLM make that work without any manual tuning:

The memory fit measures free VRAM at start and makes room step by step: it moves the token embeddings or the LM head to system RAM, shrinks prefill buffers and switches to a compressed KV cache, and moves things back to the GPU when there is room again. It prints every step it took.
TqCache takes over when even the compressed KV no longer fits: the oldest positions live in system RAM and are read over PCIe, with identical output, instead of the server refusing the context.

From 256K we use the trellis-coded KV cache (--kv tcq3/tcq2, 3-bit keys and 2-bit values); past the trained 256K the context is stretched with YaRN, the method Qwen recommends.

Share of the KV cache (%), higher is better

32K - 512K

On the GPU (%)100

In system RAM via TqCache (%)0

768K

On the GPU (%)81

In system RAM via TqCache (%)19

1M

On the GPU (%)57

In system RAM via TqCache (%)43

On the GPU (%)
In system RAM via TqCache (%)

From the engine's own start-up lines: up to 512K the memory fit keeps all KV on the GPU (moving embeddings or the LM head to RAM first when needed); at 768K TqCache keeps the oldest 149,504 of 786,432 positions in RAM, at 1M the oldest 453,632 of 1,048,576. Output stays identical.

Decode tok/s, higher is better

tqLLM + DFlash2 drafter

One RTX 3090 24 GB, Qwen3.8-27B-TQ3_4S-v2.gguf, DFlash2-TQ3 drafter, a prompt filling ~90 % of the context (WikiText with planted facts), 256 output tokens, greedy. Up to 128K the memory fit's default KV; from 256K the trellis-coded KV (--kv tcq3/tcq2); past 256K YaRN (--rope-scaling yarn). Exact recall at every size.

Prefill tok/s, higher is better

tqLLM

Same runs as the decode chart: prefill speed on the prompt that fills ~90 % of each context.

Context	Prompt tokens	Prefill tok/s	Decode tok/s	Recall (start / middle)	What the fit and TqCache did
32K	31,077	1,955	104.9	yes / yes	nothing needed
64K	60,380	1,642	101.9	yes / yes	embeddings to RAM (1 step)
128K	120,656	838	87.3	yes / yes	6 steps incl. KV q4/tcq2; LM head back on the GPU
256K	236,319	554	75.2	yes / yes	trellis KV, all on the GPU
384K	356,171	434	54.6	yes / yes	trellis KV + YaRN ×1.5, all on the GPU
512K	475,498	357	49.6	yes / yes	trellis KV + YaRN ×2; embeddings to RAM
768K	712,193	261	21.4	yes / yes	7 fit steps, then TqCache: 19 % of the KV (1.6 GB) in RAM
1M	947,699	204	7.7	yes / yes	7 fit steps, then TqCache: 43 % of the KV (5.0 GB) in RAM

Up to 512K all KV stays on the GPU. Past that, TqCache keeps the oldest positions in system RAM with identical output; this test PC is a Ryzen 5 3600 with DDR4 and a PCIe 3.0 link to the GPU, the slow case for those reads (decode drops from 49.6 at 512K to 21.4 at 768K and 7.7 at 1M). A PCIe 4.0 or 5.0 board reads them 2-4× faster.

For 1M: --context 1048576 --kv tcq3/tcq2 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 262144. Longer context costs decode speed (attention reads more KV); ask for the context your prompts need.

Exact speculative decoding

A small drafter (DFlash2 for Qwen3.8-27B, DSpark for MiniCPM-V 4.7, or the model's own MTP head) proposes several tokens and the full model checks them in one pass, keeping only what it would have written itself. Tree, chain and MTP verify share one attention arithmetic with plain decoding, so greedy output is identical token for token. On one RTX 3090 that is 273 tok/s on the LocalMaxxing reasoning prompt. How it stays exact.

MoE models larger than the GPU

Mixture-of-experts models such as MiniCPM-V 4.7 35B-A3B run even when the file is bigger than VRAM: --moe-host-mb N keeps the least-used experts in system RAM (chosen from a routing profile), the rest runs from VRAM. On one RTX 3090 the 24.7 GB Q5_K_M file decodes at 177 / 182 / 168 tok/s in image 1.0.0 against llama.cpp's 124 (-ncmoe 2), output identical; the next image adds a VRAM cache of recently used experts (--moe-cache-mb) and the shared-expert fusion: 187 / 193 / 178. MoE numbers.

TurboQuant weights

TQ3, TQ4 and TQ3_4S rotate each block of weights before rounding, so a small codebook fits every block: Qwen3.8-27B fits in 11.85 GB (TQ_XS) for 16 GB cards, or 14.72 GB (TQ_L) closest to the original. Standard GGUF files (Q4_K_M, Q5_K_M, Q8_0, 2-bit Unsloth UD) load too. TQ formats, explained.

Coding agents and APIs

The server speaks the OpenAI Chat Completions API, and from image 1.2.0 also the OpenAI Responses API (Codex) and the Anthropic Messages API (Claude Code) on the same port. tqllm launch opencode | codex | claude starts the server with the right model and drafter for your GPU, then the agent. Launch.

Several users

--np N serves several requests at once with batched verify; the context splits across the slots, and a setting that does not fit is refused at start with the largest one that does.