HTTP API
OpenAI-compatible, on 127.0.0.1:8086 by default.
| Endpoint | What it does |
|---|
POST /v1/chat/completions | chat, with streaming, reasoning and images (with --mmproj) |
POST /v1/completions | plain text completion |
GET /v1/models | the served model |
GET /health | status, engine commit and the memory fit's choices (fit) |
GET / POST /admin/slots | read or change the live number of parallel requests |
Each response reports its own timing (prompt and decode tok/s, drafted and accepted tokens).
top_k accepts 0 to INT_MAX (sampling uses at most 16 candidates); seed accepts 0 to
9007199254740991, and negative seeds are rejected with HTTP 400.
Docker variables
| Variable | Default | Sets |
|---|
MODEL_FILE | | the GGUF to serve |
TOKENIZER_FILE | from the GGUF | tokenizer override |
DRAFT_MODEL_FILE, DRAFT_VOCAB_FILE | | DFlash2 drafter and its token list |
KV_FORMAT | bf16 | KV cache format, e.g. tq3 |
CONTEXT | 16384 | context, split across parallel slots |
PARALLEL | 1 | requests served at once |
MAX_TOKENS | 1024 | default output cap |
REASONING | off | thinking mode |
Model and serving
| Option | Default | What it does |
|---|
--model FILE / --model-dir DIR | | the GGUF (or a safetensors directory) to serve |
--tokenizer FILE | the GGUF itself | only for safetensors or a deliberate override |
--host IPv4, --port N | 127.0.0.1, 8086 | where the API listens |
--context N | 2048 | context per server, split across --np slots; the memory fit may lower it |
--max-tokens N | 512 | default output cap per request |
--np N | 1 | requests served at once (batched verify) |
--name NAME, --system TEXT | | model name in the API; a default system prompt |
--api-key KEY / --api-key-file FILE | | require a bearer key |
--mmproj FILE | | vision projector for image input (--image-min-tokens, --image-max-tokens) |
--repeat-limit N | 256 | end a request after N identical tokens in a row (0 off) |
Reasoning and chat templates
| Option | Default | What it does |
|---|
--reasoning on|off | off | thinking mode |
--reasoning-effort low|medium|xhigh | | effort hint for the template |
--reasoning-budget N | -1 | thinking token budget (-1 unlimited, 0 end now) |
--reasoning-answer-reserve N | 8192 | tokens kept for the answer after a budgeted think |
--jinja / --no-jinja | jinja | the model's Jinja chat template, or the native Qwen layout |
--chat-template-file FILE, --chat-template-kwargs JSON | | override the template or pass template variables |
Speculative decoding
| Option | Default | What it does |
|---|
-md FILE, --draft-vocab FILE | | DFlash2 drafter and its token list; the output equals decoding without it |
--draft-max N | 7 | drafted tokens per step |
--no-tree | tree on | verify a single draft chain instead of a token tree |
--draft-window N | GGUF value | drafter attention window (0 = whole context) |
--draft-retrieve N | 0 | N far chunks of the prompt in the drafter window; faster on long prompts, ~20 KB per token of memory |
--lookup L [--lookup-first K] | off | prompt-lookup drafts next to the tree |
--mtp | off | draft with the model's own MTP head (text-only, greedy, one user) |
--mtp-from N | 0 | with -md and --mtp: prompts of N+ tokens draft with MTP |
KV cache and long context
| Option | Default | What it does |
|---|
--kv FORMAT / -ctk K -ctv V | bf16 | KV format; --list-kv lists them (tq3, trellis tcq3 / tcq2, q8, q4, splits) |
--kv-host N | memory fit | KV positions kept in system RAM (exact) |
--kv-stage on|off | on | stream the RAM part back layer by layer while decoding |
--kv-compact R | off | compress old KV R-fold (not exact; one slot) |
--prompt-cache on|off | off | keep the previous turn's KV for follow-ups (one slot) |
--rope-scaling none|linear|yarn | none | context past the trained length (--rope-scale, --yarn-orig-ctx) |
Memory fit and prefill
You normally pass only --context (and --np). If that does not fit at full settings, the
server picks what fits and prints one line, for example:
tqserve: fit: to fit 1 slot(s) x 131072 in GPU memory: KV cache q4/tcq2 (asked bf16), prefill buffers -ub 512,
prefill FP16 weight blocks of 2048 rows (choose yourself with --kv, -ub, --gemm-rows or a smaller --context)
| Option | Default | What it does |
|---|
-fit on|off | on | load, measure, and step down until the margin is free |
-fitt MiB | 768 (256 under WSL) | free memory to leave on the GPU |
-fitc N | 4096 | smallest context the fit may choose |
-fitl on|off | on | on: pick compact settings and say so; off: refuse instead |
--gemm-rows N | 0 (fit) | prefill FP16 weight blocks of N rows (less VRAM) |
-ub N | automatic | prefill chunk size |
--prefill-int8 on|off | on for RTX 30 / 40 / 50 | int8 prefill GEMMs |
--prefill-fp8 on|off | off | FP8 prefill GEMMs (sm_89+); faster prefill on GB10, small accuracy cost |
--slot-reserve MiB | 128 | free memory per parallel slot; an --np that does not fit is refused |
--graph-cache N | 256 | captured CUDA graphs kept (~12 MB each, least recently used evicted) |
| You want | Add |
|---|
| full-quality BF16 KV, accept a shorter context | --kv bf16 |
| bigger prompt buffers (faster prefill, more memory) | -ub 2048 |
| exactly what you asked for, or an error | -fitl off |
| several users at a long context (4 x 32K) | --np 4 --context 131072 |
| more free memory for other programs | -fitt 2048 |
Diagnostics
| Option / variable | What it does |
|---|
--list-kv, --list-flags | KV formats; every environment flag with its status |
TQLLM_DEBUG=key,... | memlog, profile, steps, stageprof, matvec, tune, prefill, draft, tree_chain, ... |
TQLLM_MATVEC_RULE=BITS:KERNEL[:PF],... | override the decode matvec kernel per format |
Tokenizer conversion
GGUF models need no tokenizer file: tqserve reads it from the GGUF metadata. Only safetensors checkpoints, or a
deliberate --tokenizer FILE override, need a tokenizer.tqtok, built from
tokenizer.json and tokenizer_config.json of the same model family. It needs Python 3, not
PyTorch:
python3 tools/tqconvert.py --source /path/to/hf-tokenizer --out-dir models --tokenizer-only