Install

Requirements

Get the image

The prebuilt image is ghcr.io/turbo-tan/tqllm:1.2.3. One tag covers both platforms; docker pull picks the right one and the engine picks its kernels for your GPU at start.

docker pull ghcr.io/turbo-tan/tqllm:1.2.3
# or from Docker Hub
docker pull ytan2000/tqllm:1.2.3

Download a model

Model files are on Hugging Face. The 27B with its DFlash2 drafter is the recommended start; the drafter is what makes decoding fast, and it never changes the output.

hf download YTan2000/Qwen3.8-27B-TQ3_4S Qwen3.8-27B-TQ3_4S-v2.gguf --local-dir models
hf download YTan2000/Qwen3.8-27B-TQ drafters/Qwen3.8-27B-DFlash2-TQ3.gguf mtp-vocab/atx_65536.txt --local-dir models

Other supported files are listed on the Models page. GGUF files carry their tokenizer, so nothing else is needed.

Docker quick start

1. Start the server. This is the RTX 3090 / 4090 setting: 32K context, thinking on, drafter on.

   docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/models:/models:ro" \
     -e MODEL_FILE=/models/Qwen3.8-27B-TQ3_4S-v2.gguf \
     -e DRAFT_MODEL_FILE=/models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf \
     -e DRAFT_VOCAB_FILE=/models/mtp-vocab/atx_65536.txt \
     -e CONTEXT=32768 -e REASONING=on -e MAX_TOKENS=4096 \
     ghcr.io/turbo-tan/tqllm:1.2.3

2. Wait for the fit line. At start the server measures free memory, sizes the KV cache and buffers, and prints what it chose. It then listens on http://localhost:8086.
3. Send a request, as in First request. The server reports tok/s for every request.

Models are mounted read-only. Any server option can go after the image name, for example docker run ... ghcr.io/turbo-tan/tqllm:1.2.3 --prefill-int8 on.

Settings for your GPU

docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/models:/models:ro" \
  -e MODEL_FILE=/models/Qwen3.8-27B-TQ3_4S-v2.gguf \
  -e DRAFT_MODEL_FILE=/models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf \
  -e DRAFT_VOCAB_FILE=/models/mtp-vocab/atx_65536.txt \
  -e CONTEXT=16384 -e KV_FORMAT=tq3 \
  ghcr.io/turbo-tan/tqllm:1.2.3

• REASONING=on for thinking tasks, with MAX_TOKENS large enough for the thinking plus the answer (4096 or more).
• PARALLEL=N serves N requests at once; the context is split evenly between them.
• No drafter file? Add --mtp after the image name to draft with the model's own prediction head (one user, greedy, same output): 155 tok/s at 1.6K tokens on an RTX 3090, against 57 without drafting.

First request

curl http://localhost:8086/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Explain a hash map."}],"temperature":0,"max_tokens":128}'

Any OpenAI-compatible client works. With the official Python SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8086/v1", api_key="none")
reply = client.chat.completions.create(
    model="tqllm",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
)
print(reply.choices[0].message.content)

If you start the server with --api-key KEY, pass the same key as api_key.

Docker Compose

The repository ships a compose.yaml. Place the model at models/model.gguf, or name your files with variables:

docker compose up            # uses the prebuilt image
docker compose up --build    # or builds the image from source

MODEL_NAME=Qwen3.8-27B-TQ3_4S-v2.gguf DRAFT_NAME=drafters/Qwen3.8-27B-DFlash2-TQ3.gguf \
VOCAB_NAME=mtp-vocab/atx_65536.txt CONTEXT=32768 REASONING=on docker compose up

Use MODEL_DIR=/path/to/models to point at an existing models directory.

Build from source

git clone https://github.com/turbo-tan/tqllm.git
cd tqllm
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=native
cmake --build build --target tqserve -j4
./build/tqserve --model models/Qwen3.8-27B-TQ3_4S-v2.gguf \
  -md models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf --draft-vocab models/mtp-vocab/atx_65536.txt \
  --context 32768 --port 8086

./build/tqserve --help prints every option. Safetensors checkpoints need a tokenizer file; see Tokenizer conversion.

Windows (WSL 2)

1. Install WSL 2 and a Linux distribution (wsl --install in an administrator PowerShell), and the current NVIDIA driver for Windows. The driver provides CUDA inside WSL; do not install a Linux GPU driver in the distribution.
2. Install Docker (Docker Desktop with the WSL 2 backend, or Docker Engine inside the distribution with the NVIDIA Container Toolkit). Check with docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu24.04 nvidia-smi.
3. Follow the Docker quick start from the WSL shell. Keep the models inside the Linux file system (for example ~/models), not under /mnt/c, for fast loading.

Under WSL the memory fit leaves 256 MiB of the GPU free by default (768 on Linux), since Windows manages display memory itself. Measured on an RTX 5060 Ti 16 GB under WSL 2: 94.5 tok/s decode on a 2K prompt with the drafter.

Troubleshooting

The server chose a smaller context or a compact KV cache

That is the memory fit. It moves the token embeddings and LM head to host memory first, then shrinks buffers, the KV format and finally the context, and prints one line saying what it did (also in GET /health). Your own settings always win: --kv bf16, -ub 2048, or -fitl off to get exactly what you asked for or an error.

Other programs need GPU memory

Keep memory free with -fitt MiB (default 768, 256 under WSL), for example -fitt 2048.

My GGUF does not load

Supported tensor types are listed on the Models page. Standard Q4_K_M, Q5_K_M, Q6_K and Q8_0 files load, and so do 2-bit Unsloth UD quants (Q2_K, IQ2_*, IQ1_S); only IQ1_M tensors do not load yet.

Decode is slower than the tables

Drafter speed depends on how often its guesses are accepted: predictable text (reasoning, prose) runs fastest, code and long contexts slower. Compare against the same prompt type in Performance.