Run big models fast on the GPU you already have.
tqLLM is a C++/CUDA inference engine for open-weight models and TurboQuant GGUF weights. One Docker image, an OpenAI-compatible API, and speculative decoding that never changes the answer.
Built for one person and one GPU
Most engines are tuned for data-centre batches. tqLLM is tuned for the case you actually have: one card, one user, long prompts, and answers you can trust.
Exact speculative decoding
A DFlash2 drafter proposes several tokens; the model checks them in one pass and keeps only what it would have written itself. Tree, chain and MTP verify share one attention arithmetic with plain decoding, so greedy output is identical token for token.
TurboQuant weights
TQ3, TQ4 and TQ3_4S rotate weights before quantising, so a small codebook fits every block. The 27B fits in 11.85 GB as TQ3, and TQ4 sits much closer to the original at a similar size.
Long context, small cards
The memory fit sizes everything at start and says what it chose. Ask for more context than the GPU holds and TqCache keeps the oldest KV positions in system RAM, exactly, instead of refusing.
OpenAI-compatible API
/v1/chat/completions, /v1/completions and /v1/models on
localhost. Point any OpenAI client, coding agent or chat UI at it.
Thinking, done right
Reasoning on or off, effort levels, thinking budgets with an answer reserve, and the model's own Jinja chat template, all as server options.
Many users when you need it
--np N serves several requests at once with batched verify. The context splits
evenly across them, and an --np that does not fit is refused at start with the largest that does.
Up and running in three commands
Pull the image, download a model, start the server. The engine picks its kernels for your GPU at
start, and the API listens on http://localhost:8086.
# 1. models
hf download YTan2000/Qwen3.8-27B-TQ3_4S Qwen3.8-27B-TQ3_4S-v2.gguf --local-dir models
hf download YTan2000/Qwen3.8-27B-TQ drafters/Qwen3.8-27B-DFlash2-TQ3.gguf mtp-vocab/atx_65536.txt --local-dir models
# 2. server
docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/models:/models:ro" \
-e MODEL_FILE=/models/Qwen3.8-27B-TQ3_4S-v2.gguf \
-e DRAFT_MODEL_FILE=/models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf \
-e DRAFT_VOCAB_FILE=/models/mtp-vocab/atx_65536.txt \
-e CONTEXT=32768 -e REASONING=on -e MAX_TOKENS=4096 \
ghcr.io/turbo-tan/tqllm:1.2.3
# 3. ask
curl http://localhost:8086/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Explain a hash map."}],"max_tokens":256}'Measured, not promised
Every number names its GPU, files and settings, and the benchmark script ships in the repository.
| GPU | Model | Decode, plain | Decode, drafter | Prompt |
|---|---|---|---|---|
| RTX 3090 24 GB | Qwen3.8-27B TQ3_4S-v2 | 55.5 | 155.2 | 2K summary |
| RTX 3090 24 GB | Qwen3.8-27B TQ3_4S-v2 | 50.6 | 131.0 | 32K summary |
| RTX 3090 24 GB | Qwen3.8-27B TQ3_4S-v2 | - | 256 | LocalMaxxing reasoning |
| RTX 5060 Ti 16 GB | Qwen3.8-27B TQ_XS | - | 113 | 200K tokens |
See it run
tqLLM against llama.cpp, and screen recordings from TQ Arena. Each caption names the GPU, the file and the drafter.
IQ3_S): 201.8 vs 147.3 tok/s. Q5_K_M (24.7 GB, larger than the card):
--moe-host-mb 2000 146.8 vs -ncmoe 3 131.0 tok/s, the smallest offload that fits a 3090
that also drives a desktop (-ncmoe 2 runs out of memory there).Qwen3.8-27B-TQ3_4S-v2.gguf on the 3090 at 278.9 tok/s and a
12.55 GB pre-release TQ file on the 5060 Ti at 150.7 tok/s, both with the DFlash2 drafter, LocalMaxxing
reasoning prompt, thinking on.Qwen3.8-27B-TQ3_4S-v2.gguf + DFlash2 drafter. New
writing is slower.TQ Arena: compare engines side by side
Send one prompt to several engines or GPUs at once and watch each stream with live tokens per second, time to first token and totals; HTML answers open next to each other when they finish. Not released yet.
Qwen3.8-27B-TQ3_4S-v2.gguf) and 93 tok/s on GB10 (an earlier TQ4 file), both DFlash2 Q4_K_M,
thinking off.Support tqLLM
tqLLM is free and open source, built and tested on our own GPUs. Donations pay for test hardware and the time to keep it fast on the cards people actually own.