Launch and connect

Quick start: tqllm launch

tools/launch/tqllm.py in the repository starts a local server and runs a coding agent against it, like ollama launch. It needs Python 3, Docker with the NVIDIA Container Toolkit (or a native tqserve build) and the agent itself installed.

git clone https://github.com/turbo-tan/tqllm && cd tqllm
python3 tools/launch/tqllm.py --list                     # model catalog (local copies are marked)
python3 tools/launch/tqllm.py launch opencode            # the default model for this GPU, downloaded if missing
python3 tools/launch/tqllm.py launch codex --model qwen3.8-27b-xs
python3 tools/launch/tqllm.py launch claude --model models/Qwen3.8-27B-TQ_S.gguf \
  --drafter models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf --draft-vocab models/mtp-vocab/atx_65536.txt
python3 tools/launch/tqllm.py serve --model qwen3.8-27b   # just the server, until Ctrl+C

• Agents: opencode, codex and claude (Claude Code, through the server's Anthropic Messages API). Arguments after -- go to the agent, e.g. launch opencode -- run "fix the failing test".
• Models: --model takes a GGUF file, a catalog name or nothing (the default for your GPU's memory). Catalog: qwen3.8-27b (TQ3_4S-v2, the 24 GB default), qwen3.8-27b-s (TQ_S, the 16 GB default), qwen3.8-27b-xs and qwen3.8-27b-l. Files are looked up in ./models, ~/models and $TQLLM_MODELS; missing ones are offered for download from Hugging Face with a size and sha256 check (--yes skips the question).
• Server: the Docker image (ghcr.io/turbo-tan/tqllm:1.2.3) by default, or --native build/tqserve. Context defaults to 65536 because agent prompts are long; --context, --port, --reasoning and --server-arg (repeatable) pass through. A server already answering on the port is reused and left running; one the command started is stopped when the agent exits or on Ctrl+C.
• The agents' own configuration files are never written; each gets its provider through its environment or command line.

Prefer to run the server yourself, or use another tool? The manual steps follow.

Manual: 1. Start the server

Download a model as in Install, then start the image. Coding agents send long prompts, so give them more context than chat needs; the memory fit lowers it, and says so, if the card cannot hold it.

docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/models:/models:ro" \
  -e MODEL_FILE=/models/Qwen3.8-27B-TQ3_4S-v2.gguf \
  -e DRAFT_MODEL_FILE=/models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf \
  -e DRAFT_VOCAB_FILE=/models/mtp-vocab/atx_65536.txt \
  -e CONTEXT=65536 -e REASONING=on -e MAX_TOKENS=8192 \
  ghcr.io/turbo-tan/tqllm:1.2.3

On a 16 GB card use Qwen3.8-27B-TQ_XS.gguf and add -e KV_FORMAT=tq3. The server listens on 127.0.0.1:8086 only; it is not reachable from other machines unless you change the port mapping.

Manual: 2. Check it answers

curl -s http://localhost:8086/health
curl -s http://localhost:8086/v1/models

/health reports the engine commit and the model; /v1/models lists the served model (tqllm).

opencode

Add a provider to opencode.json in your project (or ~/.config/opencode/opencode.json):

{
  "$schema": "https://opencode.ai/config.json",
  "model": "tqllm/tqllm",
  "provider": {
    "tqllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "tqLLM (local)",
      "options": { "baseURL": "http://localhost:8086/v1" },
      "models": { "tqllm": { "name": "Qwen3.8-27B (tqLLM)" } }
    }
  }
}

Then run opencode in the project, or opencode run "your task" for one task.

Codex CLI

The server speaks the OpenAI Responses API, which Codex uses, so Codex connects with its standard settings and no key. Add this to ~/.codex/config.toml:

model = "tqllm"
model_provider = "tqllm"

[model_providers.tqllm]
name = "tqLLM"
base_url = "http://localhost:8086/v1"
wire_api = "responses"

Then run codex in the project, or codex exec "your task" for one task. Without editing the file:

codex -c model_providers.tqllm.name=tqLLM \
      -c model_providers.tqllm.base_url=http://localhost:8086/v1 \
      -c model_providers.tqllm.wire_api=responses \
      -c model_provider=tqllm -m tqllm

Needs a server with /v1/responses: image 1.2.0 or newer, or a native build of the repository (launch codex --native build/tqserve). Codex 0.16x no longer accepts the older wire_api = "chat". Tested with Codex 0.162.1.

Claude Code

The server also speaks the Anthropic Messages API, so Claude Code points straight at it:

ANTHROPIC_BASE_URL=http://localhost:8086 \
ANTHROPIC_AUTH_TOKEN=local \
ANTHROPIC_MODEL=tqllm ANTHROPIC_SMALL_FAST_MODEL=tqllm \
claude

Needs image 1.2.0 or newer.

Continue (VS Code / JetBrains)

In Continue's config.yaml:

models:
  - name: tqLLM
    provider: openai
    model: tqllm
    apiBase: http://localhost:8086/v1
    apiKey: none

Open WebUI

Because the server listens on the host's loopback, run Open WebUI on the host network (Linux) and open http://localhost:8080:

docker run -d --network host -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL=http://127.0.0.1:8086/v1 -e OPENAI_API_KEY=none \
  --name open-webui ghcr.io/open-webui/open-webui:main

Or add the connection later under Admin settings, Connections, OpenAI API.

OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8086/v1", api_key="none")
stream = client.chat.completions.create(
    model="tqllm",
    messages=[{"role": "user", "content": "Write a binary search in C."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Any client that speaks the OpenAI Chat Completions API works the same way: set the base URL to http://localhost:8086/v1 and the model to tqllm.

Settings for agents

• Context: 65536 is a good start for agents; their system prompts and tool definitions are long.
• Reasoning: REASONING=on is better for planning; turn it off for quick edits.
• Output cap: MAX_TOKENS limits each answer; agents that write whole files need 8192 or more.
• Several agents at once: add --np 2 (or more) after the image name; the context splits across them.

All options: Options and API.