quack

From the roadmap. Results files are in evals/results/.

Why the harness matters

FrontierHarness Eval held one model, Kimi K3, fixed across 30 tasks and changed only the harness. Pass rates ran from 50% to 67%, and the median cost of a task from $1.05 to $18.34.

quack hasn’t been run on it yet. When it has, the result goes here, whatever it is.

frontierharness.org

The numbers behind the chart

Harness	Pass rate	Median cost	Median cache hit
Codex	66.7%	$3.47	88.0%
DSH Creator	63.3%	$3.28	84.3%
Claude Code	63.3%	$18.34	67.8%
Pi	60.0%	$2.43	79.4%
DSH PTC	60.0%	$4.58	87.2%
DSH Standard	60.0%	$3.46	86.5%
Oh My Pi	56.7%	$4.75	82.2%
Kimi Code	56.7%	$3.65	88.0%
DSH Minimal	56.7%	$4.72	84.6%
Exo Harness	53.3%	$1.05	70.3%
OpenCode	50.0%	$3.24	78.4%
Hermes	50.0%	$2.90	85.9%

1. 1

   Install the plucked duck

   The whole agent (prompts, sub-agents, MCP, ACP, language servers) minus the hosted-service providers, so nothing in it points off the machine. Offline, use --archive, or the .deb and .rpm packages.

   shell

   $ curl -fsSL https://github.com/paulhovey/quack/releases/latest/download/install.sh \
     | sh -s -- --plucked

2. 2

   Describe your server

   Anything OpenAI-compatible: vLLM, SGLang, llama.cpp. base_url is the only required line; leave out models and they’re listed from the server.

   providers.d/corp.toml

   # /etc/quack/providers.d/corp.toml
   name = "Corp LLM"
   base_url = "https://llm.internal.example:8000/v1"
   api_key = { env = "CORP_LLM_KEY" }   # leave out if no key
   models = [
     { id = "qwen3-coder-480b", alias = "coder" },
     { id = "qwen3-coder-30b", alias = "fast" },
   ]

3. 3

   Lock it down

   The system layer’s policy holds against user files, a repo’s .quack/, environment variables and the command line.

   config.toml

   # /etc/quack/config.toml
   [model]
   default = "corp/coder"
   small = "corp/fast"
   
   [network]
   offline = true
   
   [sandbox]
   mode = "workspace-write"     # commands: no network
   
   [permissions]
   deny = ["shell:curl *", "shell:wget *"]
   
   [policy]
   locked = [
     "model.default", "network",
     "sandbox.mode", "permissions.deny",
   ]
   max_permission_mode = "full-auto"   # no yolo
   project_config = "trusted-only"
   
   [policy.providers]
   allowed = ["corp"]
   hosts = ["llm.internal.example"]
   
   [policy.mcp]
   hosts = ["*.internal.example"]

4. 4

   Check it

   There’s nothing else to install. quack doctor makes no model call; a user’s provider pointing elsewhere shows as blocked, and the only host listed is yours.

   terminal

   $ quack doctor
   ok    defaults   /usr/share/quack/defaults
   warn  providers  1 of 2 usable
   ok      corp     2 declared models
   warn    rogue    not allowed by policy
   ok    model      corp/coder
   ok    agents     explore (on corp/fast), general
   ok    sandbox    workspace-write · Landlock + seccomp
   info  network    llm.internal.example · policy allows only
                    llm.internal.example, *.internal.example

What’s guaranteed

Only what the files name

No telemetry, update checks or model downloads. A recording-proxy test and a CI run in a loopback-only network namespace check every build.

Hosts are a policy, not a hope

With [policy.providers] hosts set, a provider pointing anywhere else doesn’t load, whoever wrote it: a user, a project or a bundle.

Locks hold everywhere

User files, a repo’s .quack/, environment variables, -c and /config can’t change a locked key. QUACK_SYSTEM_DIR is ignored while the real system folder holds a policy.

Commands stay offline too

The model’s shell commands run in the OS sandbox with no network on Linux and macOS. Windows has no sandbox yet, so there, deny rules are the only control on commands.

Where it lives

Each layer overrides the one before. Every layer’s folder has the same shape, one file per item, and a file with the same name in a higher layer replaces the lower one. quack config paths prints the folders for this machine.

config.toml          settings
providers.d/*.toml   model servers and services
mcp.d/*.toml         MCP servers
lsp.d/*.toml         language servers
hooks.d/*.toml       programs that run on agent events
agents.d/*.toml      sub-agents for the task tool
profiles.d/<id>/     prompts and tool shapes per model family
commands/*.md        your slash commands
skills/<name>/       SKILL.md skills
bundles/<name>/      groups of the above
AGENTS.md            instructions for every project

Every option

Any subset works; everything left out keeps its default. quack config check validates every file against its JSON schema, /config edits settings in the UI, and the README has the full list.