Troubleshooting
Start with the local doctor from the repository checkout:
gitmoot doctor --repo .
gitmoot status --repo owner/repo
gitmoot daemon status
Most Gitmoot failures come from one of four places: the installed binary, GitHub CLI auth, runtime/plugin discovery, or a local daemon/job/lock state.
Local Review Lead Is Refused
Symptom: gitmoot agent review or a review-resolved gitmoot agent run refuses
before creating a job because the lead is missing, cannot access the repo,
lacks implement, or has a non-write autonomy policy.
Likely cause: a review-only agent was dispatched without --lead, or the named
implementer is not dispatchable from its agents-database row.
Fix: keep the reviewer read-only and name a separate registered implementer:
gitmoot agent review reviewer --repo owner/repo --pr 12 --lead implementer "Review this PR."
gitmoot agent show implementer
gitmoot agent repos implementer
The implementer must be allowed on the repo, hold implement, and use
workspace-write or danger-full-access. Do not grant implementation access to
the reviewer merely to bypass the preflight.
Install Script Failed
Symptom: curl -fsSL https://gitmoot.io/install.sh | sh exits before
installing gitmoot.
Likely cause: network failure, unsupported platform, missing shell tools, or a release artifact that is not available for the current OS/architecture.
Check:
uname -s
uname -m
curl -fsSL https://gitmoot.io/install.sh -o /tmp/gitmoot-install.sh
sh -n /tmp/gitmoot-install.sh
Fix: retry the installer or use the direct binary fallback from the install page. Verify the artifact checksum before running it.
Binary Not On PATH
Symptom: gitmoot: command not found after install.
Likely cause: the install directory is not on PATH, or the shell has not been
restarted after pipx ensurepath or installer profile changes.
Check:
command -v gitmoot
echo "$PATH"
ls -l ~/.local/bin/gitmoot
Fix: add the install directory to PATH, restart the shell, or move the binary
to a directory already on PATH.
Checksum Mismatch
Symptom: the local SHA256 does not match the release checksum.
Likely cause: partial download, wrong artifact, stale checksum, or a tampered download.
Check:
sha256sum <artifact>
shasum -a 256 <artifact>
Fix: delete the file, download the artifact again from GitHub Releases, and compare against the checksum for that exact release and platform.
GitHub CLI Auth Fails
Symptom: PR comments, issue comments, review publication, status checks, or merge actions fail.
Likely cause: gh is not installed, is authenticated as the wrong account, or
does not have access to the repo.
Check:
gh auth status
gh repo view owner/repo --json nameWithOwner
gh pr list --repo owner/repo --state open
Fix: authenticate gh for the account that can read and write the repository,
then retry the Gitmoot operation.
Send A Gitmoot Bug Report
Symptom: a Gitmoot job failed, blocked, or was cancelled, and you want to send a useful report without exposing raw runtime output.
Likely cause: the failing job has local context that matters for debugging: repo, agent, runtime, action, task, selected error, result summary, and recent events.
Check:
gitmoot job show <job-id>
gitmoot report bug --job <job-id> --preview
Fix: preview first. The report is redacted, omits raw runtime output by default,
adds the gitmoot-dashboard-report and bug labels, and includes a
fingerprint marker so open duplicates can be reused.
Create the GitHub issue only when you intend to file it:
gitmoot report bug --job <job-id> --create --yes
The command prints either created issue: ... or existing issue: ...; use that
URL when sharing status.
Plugin Doctor Fails
Symptom: Codex or Claude Code does not discover Gitmoot, or
gitmoot plugin doctor reports missing package or runtime state.
Likely cause: plugin package was not generated, runtime CLI is missing, runtime uses a different home directory, or the package cache is stale.
Check:
gitmoot plugin doctor
gitmoot plugin doctor codex
gitmoot plugin doctor claude
gitmoot plugin path codex
gitmoot plugin path claude
Fix:
gitmoot plugin install codex --force
gitmoot plugin install claude --force
gitmoot plugin doctor
The plugin is discovery and guidance. It does not replace gitmoot, GitHub
CLI auth, or runtime/model credentials.
Runtime Session Not Found
Symptom: gitmoot agent doctor <name> cannot validate a Codex, Claude, or Kimi
session, or a job resumes the wrong session.
Likely cause: a last reference changed, the runtime home changed, or the
session id is stale. For a Kimi agent, the runtime reference must be a Kimi
session id (session_<uuid>) or empty.
Check:
gitmoot agent list
gitmoot agent show <agent>
gitmoot agent doctor <agent>
codex exec resume --help
claude --help
kimi --help
Fix: prefer explicit session UUIDs or thread names over last, then
re-subscribe the agent with the correct session reference — or, for a
registered agent whose session is genuinely dead or stranded, rebind it in
place without re-registering:
gitmoot agent restart <agent>
agent restart abandons the old runtime session and starts a fresh one for the
same agent; it refuses while the session is live or the agent has in-flight
jobs (finish or cancel those first).
Note that some session failures self-heal without your intervention: a dead
Claude --resume target is retried on a fresh session and re-pinned (#443),
and a transient 401 ("socket connection closed unexpectedly") under sustained
concurrency is retried with backoff (#487/#509). A job whose events show one of
these errors followed by a success worked as designed.
Daemon Not Running
Symptom: queued jobs do not move, PR comments are not consumed, or dashboard shows the daemon as down.
Likely cause: daemon was never started, exited, or is running with the wrong repo/home.
Check:
gitmoot daemon status
gitmoot daemon logs
gitmoot status --repo owner/repo
If daemon status warns that the advertised log is missing or was last written
before the running daemon started, the path is not receiving current output.
gitmoot doctor reports the same confirmed condition as a non-fatal
daemon log check. If the daemon runs under systemd --user, follow the likely
live stream instead:
journalctl --user -u gitmoot-daemon -f
The warning does not assert how the daemon was launched. It is omitted when the daemon is stopped, the log is current, or the comparison cannot be made.
Fix:
gitmoot daemon start --poll 30s --workers 1
Use gitmoot daemon run only when you intentionally want a foreground process.
--repo owner/repo scopes the daemon to a single repo: it polls only that
repo's PRs and claims only that repo's queued jobs. Omit --repo to supervise
every enabled registered repo from one daemon (#581). If queued jobs for one
repo never move while another repo's jobs do, check whether the daemon was
started with --repo scoping it to a different repo, or with a --session <root-job-id> (alias --root) filter: a daemon started with --session runs
only jobs whose root_job_id matches that orchestration run plus the root
coordinator job itself. Restart it without --repo/--session to drain
unrelated jobs.
Also check the repo is enabled in gitmoot repo list.
Claude auth is independent of daemon restarts. Rotate the owner-only
runtime-auth.env with gitmoot auth set claude; the next delivery observes
it. Inspect masked sources with gitmoot auth status and validate them with
gitmoot auth probe claude or gitmoot doctor.
Daemon Already Running
Symptom: gitmoot daemon start/run refuses with daemon already running with pid ….
Likely cause: a daemon is already supervising this Gitmoot home. One daemon per home is enforced with a pidfile plus a flock backstop (#550/#556); starting a second one is refused by design (a stale pidfile whose owner is dead is liveness-checked and recovered automatically, so restarts after a crash work).
Fix: use the running daemon — it supervises all subscribed repos. To change its
settings, send kill -HUP <pid> for a live [daemon] config reload (#577), or
use gitmoot daemon restart. Scripts that start daemons should treat this
refusal as success, not an error.
Job Stuck Or Failed
Symptom: a job is queued, blocked, failed, or no longer changing state.
Likely cause: runtime delivery failed, worker is read-only, GitHub auth failed, or another lock is active.
Check — read the stuck reason first:
gitmoot job list --repo owner/repo # WHY: column on queued/blocked jobs
gitmoot job show <job-id> # why_stuck: / next_retry_at: lines
gitmoot job events <job-id>
gitmoot agent show <agent>
gitmoot lock list --repo owner/repo
A job that is simply WAITING now says so in its own record (#1553). Every
dispatch pass that examines a queued job and declines to claim it writes a
dispatch_held_back event naming the reason: an admission-budget refusal, a
checkout held by an in-flight job, a runtime session lock and its holder, or -
when the pass cannot attribute the wait - the bare fact that the job was not
selected this pass. Before this, that reason went only to daemon stdout, so
gitmoot job events could not tell a job waiting two hours from one about to
start. The event is throttled to one row per job per distinct reason every five
minutes, so a long wait produces a readable handful of rows rather than one per
tick.
gitmoot job list appends a WHY: column and gitmoot job show prints a
why_stuck: line for queued/blocked jobs (#552) — e.g. a runtime-session lock
wait (naming the holder), blocked: awaiting human, auth failing: …,
throttled: …, retrying: …, or a blocked-operational: <class> deferral with
the attempt schedule. A deferral that needs a human (dirty/wrong-head checkout)
also prints a suggested_action naming the fix.
An omp Unknown model: <provider/id> refusal is class model_unavailable
(#2469): a routed review moves straight to the next usable pool model with no
hold; with none left the job stays failed, since waiting cannot make omp know
the model.
Deferred jobs recover on their own (#532): a delivery failure classified as a
retryable operational blocker — runtime_auth, runtime_quota,
network_outage, or checkout_contention — is re-queued with a bounded retry
budget instead of failing terminally. job show --json carries the
blocker_class, attempt count, and suggested_action. A runtime_auth deferral
only re-dispatches once a live doctor-style credential probe passes (a failing
probe extends the hold without spending a retry), and over [events] the
deferral is a first-class job.deferred emitted instead of job.failed. A job
that "failed then reappeared as queued" is the deferral working; only act when
the retry budget is spent and the job stays failed.
A Claude read-only seat (every review/ask job under the read-only autonomy
policy) does not rely only on its staged credential snapshot. It stages the
runtime config dir (payload.runtime_config_dir, else the daemon's
CLAUDE_CONFIG_DIR, else ~/.claude) and also carries the resolved
runtime-auth.env overlay. When that snapshot is already expired the job records
one readonly_seat_credential_expired event naming the expiry, the refresh-token
state and whether an overlay was available; read it with gitmoot job events <job-id> before treating the runtime's own "OAuth session expired and could not
be refreshed" wording as an account problem. It never refuses the job.
gitmoot doctor and gitmoot auth probe claude both report that seat credential
beside the ambient one, and doctor FAILS (non-zero exit) when it is expired with
no refresh token, since every read-only seat job on that runtime will fail until
the account is re-logged in.
An OMP read-only seat uses no operator profile or ambient provider key. Set
OMP_AUTH_BROKER_URL to an HTTPS or loopback HTTP origin and point
OMP_AUTH_BROKER_TOKEN_FILE at an owner-only regular file; the daemon refuses
an upstream OMP_AUTH_BROKER_TOKEN because every seat must read /proc for
runtime startup. The job's effective model must
be provider-qualified, such as kimi-code/k3. Gitmoot writes a random
job-local token and loopback URL only into the seat's minimal config.yml; no
broker secret enters the child environment. The proxy exposes only that
provider, forwards refreshes only, and keeps broker writes local to the
disposable client. The daemon-staged OMP binary receives job-private
PI_CONFIG_DIR and PI_CODING_AGENT_DIR roots. Missing or unsafe broker
configuration refuses before staging; an unqualified effective model refuses
before OMP launches. The state and proxy are removed at the job boundary.
When broker and clients share an OS account, run auth-broker serve with a
dedicated relative PI_CONFIG_DIR and point OMP_AUTH_BROKER_TOKEN_FILE at that
directory's auth-broker.token. Otherwise both roles use
~/.omp/auth-broker.token, so a client token can invalidate the service.
A kimi credential that goes blank mid-session is REPORTED, not repaired (#1856).
Kimi's credential (~/.kimi-code/credentials/kimi-code.json) has been observed
blanked IN PLACE after a failed refresh: still-valid JSON, 136 bytes, with
access_token and refresh_token empty strings and expires_at 0, while
scope and token_type stay populated. Gitmoot writes that file NOWHERE — the
only writer is the vendor CLI — so it cannot make the write atomic, and a safety
net whose failure mode is "gitmoot corrupted the owner's credential" would be
worse than the defect. Instead the two paths that run a kimi child against the
LIVE profile — a non-seat delivery, and gitmoot agent doctor's health check —
read the credential immediately before and after the child and report a
degradation: the delivery records one kimi_credential_degraded job event (on
success and failure alike, because the measured case ran to success), and doctor
prints the same observation on stderr since a command has no job row. It fires
only on a credential that demonstrably carried a token before and demonstrably
carries none after, so a host that is simply logged out stays silent, and a
reading gitmoot cannot interpret (unreadable file, unmeasured schema) is never
treated as evidence in either direction. Read it with gitmoot job events <job-id>. Attribution is inferred, not confirmed: the event records that a
kimi child ran while the file changed, never that this child wrote it. A
read-only seat is deliberately NOT observed — it reads a staged clone inside its
own writable cache root, which the child may legitimately rewrite, and gitmoot
never grants a seat read or write access to the live profile.
sandbox-exec: sandbox read path "…": no such file or directory means a path
granted to the sandbox does not exist on disk. Explicit read grants are required,
not skipped — check readable_paths in gitmoot job show <job-id> --json and
create (or stop granting) the named path.
A job stuck in running is recovered automatically once it shows no lease
progress past the staleness window (default 30m; tune with the
GITMOOT_STALE_RUNNING_AFTER environment variable; the smallest honored value
is 1m — below-1m, malformed, or non-positive values are rejected in favor of the
30m default rather than clamped, #560). The window is a same-boot crash backstop,
not a timeout: a job holding a runtime session lock whose lease has not elapsed is
left running regardless of the window. After a reboot there is no wait at
all — the kernel boot id changes, so on its next startup and every tick the daemon
immediately requeues every job claimed on the previous boot and reclaims its
stranded runtime session lock, regardless of any unexpired lease (#651).
Boot-aware recovery is Linux only; elsewhere recovery falls back to the lease/age
window above.
Fix: resolve the underlying runtime/auth/lock issue, then retry when safe:
gitmoot job retry <job-id>
Cancel (abandon) only when the job should not continue. Cancel now dismisses a
blocked job — one paused awaiting a human — as well as a queued/running one
(#631); a dismissed job is not lost, since gitmoot job retry accepts a cancelled
job and resurrects it:
gitmoot job cancel <job-id>
A backlog of blocked jobs never clears on its own. Clear a stale batch with the
bulk form, which is a dry-run by default (it prints id/agent/repo/age and
cancels nothing) until you pass --yes:
gitmoot job cancel --state blocked --older-than 7d # preview the selection
gitmoot job cancel --state blocked --older-than 7d --yes # cancel it
Only blocked is accepted for --state; narrow the selection with --older-than
(a Go duration like 168h, or a <N>d days suffix), --repo owner/repo, and
--agent name. gitmoot doctor warns when blocked jobs older than 30d have piled
up and prints the exact command to dismiss them.
To sweep the backlog automatically, set [orchestrate].blocked_ttl to a positive
Go duration (e.g. blocked_ttl = "168h"): the daemon then dismisses any blocked
job idle longer than the TTL through the same cancel path, recording a
blocked_ttl_expired job event. It is off by default (empty or 0s disables
it; a negative value is rejected), because a blocked job is a human-awaiting
decision that is never auto-discarded unless you opt in. This is the single-job
counterpart of [orchestrate].escalation_ttl, which auto-finalizes a whole paused
delegation tree and is on by default (24h).
Stale Lock
Symptom: implementation or merge work is blocked by a lock whose owner is gone.
Likely cause: a worker died or the daemon stopped before cleanup.
Check:
gitmoot lock list --repo owner/repo
gitmoot lock show owner/repo <branch>
Fix: let a running daemon reclaim stale resource locks automatically. Task lane
locks are released only after they have been unchanged for at least 24 hours and
no non-terminal task or job references the same repository and branch. Human-
resumable blocked, awaiting_human_merge, and awaiting_human tasks retain
their locks. Release a branch lock manually only after confirming the owner is
no longer working:
gitmoot lock release owner/repo <branch> --owner <agent>
Merge Gate Deferred By An Active Branch Job
Symptom: a ready PR remains unmerged and the gate reason says an active job is in flight on its branch.
Likely cause: an ask, review, or implement job targeting that PR branch is
still queued or running. Gitmoot treats this as a transient deferral so it cannot
squash-merge, delete the source branch, and strand an in-progress fix. The task
stays ready_to_merge rather than entering a blocked state or emitting a
merge-gate error.
Check:
gitmoot job list --repo owner/repo
gitmoot job show <job-id>
gitmoot job events <job-id>
Fix: let the job settle or cancel it deliberately. The daemon re-evaluates the unchanged policy merge path on its next tick; do not release its branch lock while the job is active.
Gate Status Says A Head Was Never Cleared
Symptom: a pull request carries gitmoot/merge-gate = pending with the
description Gitmoot merge gate has not cleared this head.
Likely cause: this is the intended signal, not a failure. GitHub's
mergeStateStatus reports the absence of blockers, so a head Gitmoot has never
evaluated used to read exactly like one it approved. Gitmoot now marks an active
managed head as unjudged until its own gate produces a verdict.
Check:
gh pr view <number> --repo owner/repo --json statusCheckRollup,mergeStateStatus
gitmoot task list --repo owner/repo
Fix: nothing to do. A later gate evaluation replaces the marker with the
specific pending, failure, or success verdict for the same head. Gitmoot marks
heads only while it owns the merge decision: with [merge_gate] auto_merge = false, with GITMOOT_DISABLE_NATIVE_MERGE_GATE=1, or once a task reaches
awaiting_human_merge, dismissed, superseded, stranded or merged, it
replaces only its own generic marker with Gitmoot merge gate is not applied to this head and leaves any real gate verdict untouched. A blocked or
awaiting_human task keeps the marker, because that head genuinely has not been
cleared and Gitmoot can still resolve it when the task resumes. A draft pull
request keeps the marker until it is undrafted.
Reviewed PR Is Behind Main
Symptom: a reviewed pull request is behind its base. Gitmoot either merges the reviewed head without a branch update or blocks the merge while preserving that head.
Likely cause: both outcomes protect the exact-head review. Gitmoot merges the reviewed head as it stands only when GitHub reports it mergeable and the base explicitly permits behind heads. A branch update would create a new head and invalidate the approval.
Check:
gh pr view <number> --repo owner/repo --json headRefOid,mergeable
gh api repos/owner/repo/branches/<base>/protection --jq '.required_status_checks.strict'
Fix: if the base requires an up-to-date head, mergeability is conflicting or
unknown, or protection cannot be determined, update the branch explicitly and
obtain a new exact-head review. /gitmoot merge bypasses the automatic-merge
kill switch, but it never updates a reviewed head.
Live Docs Or LLM Context Stale
Symptom: gitmoot.io/docs or /llms.txt does not show current source docs.
Likely cause: docs were changed but not rebuilt/deployed, or stale deployed files were not deleted.
Check:
cd website
npm run build
curl -fsS https://gitmoot.io/docs/reference/cli | rg 'gitmoot dashboard'
curl -fsS https://gitmoot.io/llms.txt | rg 'Dashboard|Release Notes'
Fix: deploy the current static build with delete semantics:
cd /root/gitmoot/website
npm run build
rsync -a --delete build/ /var/www/gitmoot-docs/
Worktrees consume too much disk
Every five minutes, the daemon checks task-owned worktrees whose task is
merged, dismissed, superseded, or stranded. Age alone never qualifies a
task worktree. Gitmoot retains it unless the recorded path is deterministic,
there is no non-final job or branch lock, the /proc/<pid>/cwd scan is
conclusive with no live process inside it, the worktree HEAD is reachable from
the recorded task branch, and git status --porcelain --ignored reports no
tracked, untracked, or ignored content. Removal is non-force and preserves the
branch.
A malformed payload on any non-final job also pins task worktrees because Gitmoot cannot prove that the job owns some other path. The bounded retention log names one malformed job responsible for this global safety pin.
A worktree registered to an older checkout root is removed through the owner in
its .git pointer. If that owner cannot inspect or remove it, Gitmoot records
terminal_worktree_unremovable once instead of retrying every daemon tick.
Retention decisions log three times per path and classification before
identical messages are suppressed.
gitmoot doctor reports delegation worktrees only; ordinary task worktrees
appear in the bounded terminal task worktree retained daemon lines instead.
Doctor separates reclaimable final delegation owners, pinned non-final owners,
and unproven directories. /api/health exposes the same delegation metric and
quarantined cleanup count in its top-level worktrees field.
[workflow].delegation_worktree_ttl = "72h" supplies the default aged-cleanup
grace period; "0" disables aged cleanup. Ordinary and aged cleanup share the
same producer safety gate. Terminal state alone is insufficient: blocked,
resumable or undispatched consumers, unknown liveness or
ownership, and preservation failures retain the resource. Succeeded owners need
a stored result; failed/cancelled owners additionally need a proven
reconstruction path (currently a pinned native read-only PR review). Cleanup
never cancels consumers or treats an operational failure as permission to
delete.
Delegation worktree reclaim (the skipped and aged passes) runs on its own
background loop in the daemon, beside job dispatch, so a slow capture or removal
on a network mount no longer delays any queued job (#2517). While an attempt
removes a worktree it reserves that worktree's checkout and the repo's registered
checkout: no job starts on either until the attempt ends, and an attempt waits
while a job holds either. The terminal task worktree pass still runs in the
worker tick. Under the store-wide cleanup lock a detached worktree is only
moved aside (git worktree move into a sibling .gitmoot-cleanup-trash-*
directory, about half a second), and its files are deleted afterwards without
that lock, so job starts no longer pause for a slow deletion (#2524). A trash
directory whose deletion did not finish (a crash, a failed deletion) is recorded
as pending and deleted by the background loop later. A worktree with its own
branch is still removed in place.
Each kind of reclaim pass is bounded (#2517): it spends at most about 10
seconds per pass starting candidates across all repos, and the rest wait for a
later tick. The first repo turned away goes first on the next tick, so one repo
with slow cleanup cannot starve the others. A worktree whose preservation
evidence could not be captured (for example a git status/git diff that
timed out on a slow network mount) is retried after a quarter of the time its
cleanup has been pending, between one minute and six hours; any other reason to
keep a worktree is rechecked after a minute. Before running git, the capture
compares every tracked file's content with the index in parallel, and git then
runs with core.checkStat=minimal: on network mounts such as sshfs, inode
numbers are unstable, which otherwise makes git re-read every file one at a
time. A file that does not match is compared by git as usual.
No new fix clone is created — #2203 removed the
auto-fix leg that allocated one — and no gitmoot path deletes one that already
exists. It is a standalone object
database, and Linux has no inode-conditional unlink that can make deletion match
a preceding proof. Commit and nested-repository checks can diagnose obvious
retention reasons, but they do not close over every blob, tree, annotated tag,
pack, or concurrent write. Passing them therefore records
delegation_worktree_retained_unproved, never a proved-disposable handoff.
Terminal cleanup leaves the managed clone in place. Allocation recovery and
enqueue failures rename it to a .ttl-reclaiming-orphaned-* sibling so retries
can allocate without destroying bytes. Cleanup obligations remain open.
Managed-path absence is not treated as removal, even when no sibling is found,
and dangling symlinks remain visible through lstat.
For an explicitly authorized manual cleanup, use producer-owned inspection and a generation-bound token:
gitmoot job cleanup inspect --json <job-id>
gitmoot job cleanup apply --resource <resource-id> --token <eligibility-token> --json <job-id>
Inspection is advisory. Apply derives the path from metadata and revalidates consumers, generation, policy, physical identity and liveness under the lifecycle fence, then the checkout lock. Retry/resume and enqueue use the same fence. No arbitrary path, orphan or shared-checkout deletion is supported. Inspect itself holds the same database's exclusive lifecycle fence while reading Git and process state; concurrent lifecycle mutations wait. It does not create obligations or receipts, but its fence sidecar must be writable. A read-only database connection does not make a read-only-mounted home supported.
path_referenced retains a workspace named by any other job, including an old
terminal owner. The aged selector can discover both owners without authorizing
either removal. Investigate the ownership overlap; do not bypass the shared gate.
retry_disposition_unknown means the failed/cancelled owner has no proven
reconstruction path. Only exact-head native read-only PR reviews currently have
that proof. Failed/cancelled implement legs and every other unproven owner
retain their workspaces and owned branches indefinitely, even after TTL expiry.
This is intentional: TTL is not a maximum retention or disk-cap guarantee.
Repeated aged scans may continue to discover these retained jobs; a skip event
does not imply future removal. Neither apply nor reopen overrides the refusal,
and no force-disposal command is provided. Inspect the job and escalate disk
capacity to its owner; retry only when the work should genuinely resume. Do not
retry just to discard evidence or bypass the gate with direct deletion.
Receipts remain in the producer database outside disposable paths.
A completed token replays as already_removed without touching a new tree at
the same path. An unfinished historical episode replaced by retry becomes
superseded, not successful removal. Accounting-only recovery can finish after
retention changes; remaining destructive effects still require fresh admission.
Apply exits 0 for removed/already-removed, 3 for another completed decision,
1 for operational errors and 2 for invalid arguments.
Accounting recovery also reports checkout-lock release failures as operational
errors. Inspect the durable receipt to distinguish completed removal from
unfinished accounting before retrying.
approved_size_budget: null means unknown, never zero or unlimited, and the
budget-gated consumer path refuses with size_budget_unknown (schema v2, #2360).
To allow consumer cleanup, an owner adds a complete [cleanup_budget] section
(see the CLI reference). size_measurement_incomplete means the allocated-bytes
walk crossed a mount, met an external hardlink or an unreadable/changing entry, or
hit a sizing bound; size_over_budget means the tree is larger than the cap.
Refused applies are recorded as durable refusal receipts. Do not substitute direct path
deletion or a consumer-side size scan for the producer contract.
gitmoot doctor and /api/health discover the canonical fixes/ directory
structurally, including set-asides created before a job row exists. Directory
discovery and logical-size accounting each stop after 4096 entries; the API sets
truncated=true, the summary marks counts and bytes as lower bounds, and doctor
warns. Removal remains a manual operator decision after inspecting the working
tree and object database.
The daemon queries at most 256 due pending or aged delegation owners host-wide.
Attempted candidates and selected rows skipped by repository, session,
lifecycle, or checkout-liveness filters persist a later next-attempt time, so
fairness survives restarts rather than depending on an in-memory cursor. The
aged pass also attempts at most eight candidates per repository per tick.
Candidate-local failures skip only that worktree, and later candidates continue.
The five-minute pass cadence advances after every attempt so a failed cleanup
cannot hot-loop. Cleanup obligations retry once per minute and stop in
quarantined after the third failure. Inspect them with gitmoot job cleanup list --state quarantined and reopen a repaired target with
gitmoot job cleanup reopen <resource-id>.
Checkout contention defers both ordinary and aged cleanup without spending the
failure budget. Persisted error text uses the existing credential-redaction
policy. Aged selection uses the current generation's recorded terminal time,
and an older generation's completion cannot suppress cleanup of its replacement.
Repeated terminal-task failures log three times per path before identical messages are suppressed. The terminal-task pass proves at most eight candidates per tick, because each proof takes the checkout mutation lock and walks the ignored tree twice; the window rotates through the candidate list so a permanently retained worktree cannot starve the ones behind it.
For immediate relief, list candidate directories and prove ownership before removing anything:
find "$HOME/.gitmoot/worktrees" -type d -path '*/delegations/*/*' -prune -print
sqlite3 "$HOME/.gitmoot/gitmoot.db" -header -column '
SELECT id, state, updated_at, json_extract(payload, "$.worktree_path") AS path
FROM jobs
WHERE state IN ("succeeded", "failed", "cancelled")
AND datetime(updated_at) <= datetime("now", "-72 hours")
AND json_extract(payload, "$.worktree_path") <> ""
AND (json_extract(payload, "$.delegation_id") <> ""
OR json_extract(payload, "$.read_only_worktree") = 1);
'
gitmoot job show <job-id> --json
git -C <registered-checkout> worktree remove --force <verified-worktree-path>
git -C <registered-checkout> worktree prune
The sqlite3 command is an optional operator aid, not a Gitmoot dependency.
Verify that job show still reports succeeded, failed, or cancelled and
the same payload.worktree_path. Never remove a worktree for a blocked, queued,
or running job; settle it first.
Read-Only Review Seat Cannot Run Go Or Read Prior Verdicts
A read-only review seat is Landlock-confined. Two symptoms have the same root cause - it is reaching for something outside its grants - and neither needs a wider grant to fix.
Permission denied, exit 126, running Go
A read-only seat is now given a Go toolchain automatically, and the old
bootstrap-and-download workaround is retired. The daemon enumerates the Go
installations on its PATH, resolves the effective go.work plus checkout-root
go.mod requirement, and selects the lowest installed release that satisfies
both. It stages an immutable copy under the gitmoot home and sets GOROOT,
PATH, GOTOOLCHAIN=local, and a deterministic GOWORK. A seat should simply
run go.
The copy exists because a seat cannot be granted the operator's own installation safely: three review rounds on that shape each produced an escape, most recently a member symlink that walked a grant outside containment with no privilege required. A copy the daemon owns has no outside root to contain.
What to expect:
go versionworks and reports the selected satisfying release.GOTOOLCHAIN=localis correct here because staging already selected a release new enough for the active workspace and module; there is nothing to fetch.- The copy is per selected installation, materialised on first use, and reused. It is keyed on content, so replacing one installed version in place produces a new copy rather than a stale compiler.
GOWORKmatches selection.offstays off; unset orautosearches from the checkout; an explicit absolute path is honored. A selected workfile outside the checkout receives a file-only read grant, never a parent-directory grant.- The staged copy is read-only to the seat, always. If it were writable a seat
could rewrite its own
gobinary, which is the defect the copy exists to remove. CGO_ENABLED=0is still required. cgo reads/usr/include, which is outside the seat's grants, so-raceremains unavailable in a seat.TMPDIRis a private, short directory,/tmp/gmr-<8 hex>. Each seat gets its own, mode0700, granted to that seat only and deleted when the job ends.TMPandTEMPpoint at it too.GOCACHEandHOMEstay under the seat's own cache root. Writes anywhere else under/tmpstill returnEACCES.
If a seat gets exit 126 running go, no usable toolchain was staged. The command
is an engine-owned failure stub, not a fallthrough to an operator installation;
it prints gitmoot: runtime unavailable: no daemon-staged artifact exists.
Host-specific causes also print gitmoot: read-only seat toolchain: on the
daemon's stderr. A declared real review of a Go checkout (a go.mod or
go.work anywhere in the tree, or a go.work Go would pick up from a parent
directory) also records seat_toolchain_unavailable with that diagnostic and
blocks. A review of a repository with no Go module is not refused for a missing
Go toolchain (#2513); its seat still gets the exit-126 go.
- No Go installation is on the daemon's
PATH. This normal case has no daemon diagnostic, so theseat_toolchain_unavailableevent is the only recorded reason and invokinggostill fails explicitly through the stub. - A candidate is not a recognizable Go installation. It must have an
executable
bin/goand aVERSIONfile naming a Go release, both real files rather than symlinks. - Every installation prefix is copied.
/opt,/usr/local,/nix/store,/snap, and profile paths follow the same staging path; none receives a recursive host-root grant. - No installed release satisfies the active workspace and module. The diagnostic names the required release and every rejected candidate.
- The active
go.workis unsafe or invalid. Relative explicitGOWORKvalues, symlinks, non-regular files, oversized files, and files without a validgodirective fail closed before a mismatched compiler can run. - Free space is below the floor. Staging refuses up front rather than filling the filesystem, and the message names both the free bytes and the 4 GiB floor.
- The source contains a symlink. A symlinked name is refused anywhere in the copied set, because following one copies something the daemon never validated. The check also proves the file it opened is the file it inspected, and staging walks a directory through a held handle rather than re-resolving its path, so a source whose name is redirected mid-copy is refused rather than followed - including when the directory containing it is the thing replaced.
Two limits of that, stated rather than left implied. First, it proves the file
did not change between being inspected and being opened. A hardlink does not
change the file - it is the same underlying object - so replacing a source file
with a hardlink to another file is not detected. Two things bound this rather
than excuse it: with the kernel default fs.protected_hardlinks=1, creating such
a link already requires read and write access to the target, so it gains nothing
an attacker did not already have; and the selected source installation is
expected to be operator-owned and not writable by a seat, so a seat cannot
create one. Second, staging reads the source twice, once to
fingerprint it and once to copy it, and only executables are reconciled between
those two reads: a non-executable that differs between them is not detected. Each
read is internally consistent, so this is a gap between the passes rather than
inside one. If you point staging at a directory that untrusted processes can
write, these are the gaps to know about - the intended source is an installation
only the operator can modify.
To recover, fix the cause and restart the daemon, or run the seat's job again: staging is attempted per launch and a previously failed stage is retried rather than remembered. A staged copy that has become corrupt is detected on reuse and re-staged automatically, so deleting it by hand is not required. Staged copies are retained by current pin: a version still pinned is never collected, and a version nothing references is removed.
One limit, stated rather than implied: the identity covers every executable in
the copy, so a substituted compiler is always detected. It does not cover
non-executable members such as src or doc, because re-hashing the whole 221.6
MiB tree on every seat launch would cost more than the copy itself. The
consequence of a torn non-executable member is a visibly broken toolchain and a
build error, not a silently wrong compiler.
Seat temp dir, Go module cache, and other toolchains
Temp dir. A seat's TMPDIR is /tmp/gmr-<8 hex>, 17 bytes, so a test that
binds a Unix socket under t.TempDir() stays inside the 108-byte socket path
limit. It used to be <cache root>/tmp, which was 57+ bytes on a real host, so
socket tests failed for reasons unrelated to the code under review. When the
daemon's own TMPDIR is a clean absolute path of at most 24 bytes, it is used
as the parent instead of /tmp; a longer one is ignored. That is also how
gitmoot run inside a seat (a reviewer running gitmoot's own tests) nests its
seats: in the outer seat's /tmp/gmr-<8 hex>. If no short parent is writable,
as inside a seat of a daemon from before this change, the seat falls back to
<cache root>/tmp and logs no short writable temp parent; using cache-root temp; it works, but its socket paths are long again. The directory is mode
0700, owned by the daemon user, and only that seat is granted it: the parent
directory, other seats' gmr-* directories, the checkout and the gitmoot home
stay read-only. The daemon records each directory in
<gitmoot-home>/cache/seat-tmp/ with its owner's pid and removes it when the
job ends. If the daemon crashes first, the next daemon start (and the next seat
launch) removes every directory whose owner process is gone. The owner is the
process that made the directory (the daemon or a foreground job run), so a
seat process that outlives it loses its temp dir. A record that names a
gmr-* directory under a parent no seat could have used (anything but an
absolute path of at most 24 bytes), or the sweeping process's own TMPDIR or
a directory holding it, is kept and reported, never removed. The cache-root
fallback has no record; it goes with the cache root.
Go module cache. Offline go build and go test work in a seat. Before
the seat starts, the daemon runs go mod download for the checkout's modules
into <gitmoot-home>/cache/seat-go-mod/<repo hash>/, one cache per repository,
and grants the seat that cache read-only with GOPROXY=off. Repeat reviews of
the same repository reuse it. The download runs under its own Landlock
sandbox, with the checkout read-only and only the cache writable, the daemon's
GOPROXY minus direct and off, GOVCS=*:off and GOTOOLCHAIN=local, so no
version-control tool runs against a URL the checkout chose. A module the download could not fetch
fails in the seat as module lookup disabled by GOPROXY=off; the cause is on
the daemon's stderr as gitmoot: read-only seat Go modules:. A seat never
writes this cache, so one review cannot plant a module for the next.
Before #2314 each seat's GOMODCACHE was a new, empty directory keyed on the
checkout path, so every offline build failed. Pointing a seat at a cache
populated for another checkout path does not work either: the Landlock grant
does not cover it.
Python and other toolchains are not available in a local seat. pytest,
ruff and uv fail with Permission denied because they live in
operator-owned trees (/usr/local, ~/.local/bin, Python's
dist-packages) that a seat is not granted. Granting those trees is the
operator-tree grant #1878 rejected, and copying them is unbounded: the
interpreter's packages are hundreds of megabytes, include editable installs
that point into operator checkouts, and a project's own dependencies are often
not installed on the host at all. Rust, Node, Flutter/Dart and similar
toolchains need their own staged toolchain and writable caches. These checks
move to disposable VMs (sandboxd/E2B) under #2315. A local seat that cannot run
them reports evidence: static_only with a checks_blocked reason such as
toolchain. Terminal devices (/dev/ptmx) stay
closed: seats run as root, and a terminal device would let a reviewer type into
other agents' terminals.
Read-only seat runtime executable is unavailable
Claude, Kimi, and Codex read-only seats run a copied runtime artifact from
<gitmoot-home>/runtimes, never a recursive read grant over the operator's
profile or package root. That directory is a SIBLING of toolchains/, not a
child: the toolchain collector removes anything under toolchains/ that is not
a pinned toolchain, which would delete staged runtimes mid-launch.
Every source read uses openat2 with RESOLVE_NO_SYMLINKS and
RESOLVE_BENEATH, so no component of a source path may be a symlink and nothing
resolves outside the boundary. os.Root is not sufficient here: it refuses
escapes out of a root but follows symlinks that stay inside one, so a member
replaced by a link to a sibling credential was previously copied.
An AMBIGUOUS SOURCE TREE IS REFUSED WHOLE rather than filtered, because file mode bits cannot distinguish a secret from a payload - a 0640 file under a 0750 or setgid directory reads as group-readable, and a POSIX ACL shows up in the group bits. Staging therefore refuses, and publishes the runtime unavailable, if the boundary contains a symlink, a non-regular member, a setuid or setgid member, any POSIX ACL, a directory the world cannot traverse, or a member that is not world-readable. The resolved entrypoint itself is exempt from the world-readable rule only, so a runtime installed mode 0700 still runs.
A SCRIPT RUNTIME GETS AN EXPLICIT LAUNCHER. Codex resolves to codex.js with
#!/usr/bin/env node, so the engine stages that interpreter too and writes a
launcher that execs the STAGED interpreter with the STAGED entrypoint. The
kernel never resolves the entrypoint's original shebang, which would reach the
operator's copy. A binary runtime gets a relative symlink instead, needing no
shell.
THE ENGINE'S OWN LAUNCHER DIRECTORY IS EXCLUDED FROM REVALIDATION, and the
exclusion is a traversal rule rather than a filter applied afterwards (#1974).
A published tree is re-proved on every reuse by recomputing the digest of its
members with the same symlink-refusing traversal, and .bin/ never contributed
to that digest - the recorded value is the digest of the SOURCE members, which
cannot contain a launcher the publish created after reading them. Collecting it
and dropping it afterwards still had to OPEN it, so a binary runtime's symlink
launcher failed the reuse with too many levels of symbolic links and the
runtime was republished as the exit-126 shim on every subsequent staging. Only
the TOP-LEVEL .bin is excluded; a .bin directory that came from a source
package, which is the ordinary shape of a node-packaged runtime, is payload and
is still hashed.
Every runtime class the engine can dispatch is staged, not only the seat's own.
An absent runtime, an unresolvable interpreter, an ambiguous tree, or a kernel
without openat2 all resolve to engine-owned commands that print
gitmoot: runtime unavailable: no daemon-staged artifact exists and exit 126.
This preserves ordinary system commands through the inherited PATH without
permitting a runtime name to fall through to its host copy. Failures also appear
as gitmoot: read-only seat runtime: on daemon stderr.
Reclaiming old copies. Each distinct content fingerprint publishes a new
directory, so every provider upgrade leaves one behind. The daemon removes a copy
once a day when two things are both true: no job has staged it for 7 days, and
no live process uses it. "Uses" means running from it, having it as the working
directory, naming it on the command line, or mapping a file from it. Staging
refreshes a last-used stamp under a lock that the collector takes exclusively,
so a copy being handed to a job is never removed. A removed copy is staged again
the next time a job needs it. Run gitmoot runtime prune --dry-run to see what
would be removed, and gitmoot runtime prune to do it now (--min-idle changes
the 7 days). If the process scan cannot finish, nothing is removed.
Unavailable-runtime commands are never removed. toolchain.Collect still has no
production caller, so staged Go toolchains are not reclaimed.
Codex reviews report zero executed checks (bwrap: setting up uid map)
A codex read-only seat used to fail EVERY command before running it, so its reviews came back static-only with an executed-check count of zero. Codex's own sandbox is bwrap, which needs a user namespace that gitmoot's Landlock domain denies, so two sandboxes were fighting.
Measured on this box, one command with only the domain differing:
bwrap --dev-bind / / --unshare-user -- /bin/true
outside the Landlock domain -> rc=0
inside the domain -> rc=1 bwrap: setting up uid map: Permission denied
inside + /proc writable -> rc=1 bwrap: Failed to make / slave
inside + /proc + /dev writable -> rc=1 bwrap: Failed to make / slave
plain /bin/true inside the domain -> rc=0
So the domain executes fine and NESTING was the cause; granting more only moved the error from the uid map to mount propagation, which is why widening grants could not settle it.
A codex seat therefore now runs with its own sandbox OFF and relies on the Landlock domain, which is the boundary gitmoot builds and tests. End to end in that seat, same grants either way:
- before:
bwrap: setting up uid map: Permission denied, nothing executed - after:
/bin/bash -lc 'echo PROBE_RAN' … succeeded in 0ms: PROBE_RAN - writing outside the granted roots is still refused -
sh: cannot create /root/…: Permission denied, and no file appears on the host
If a codex seat still reports zero executed checks, the seat's own state
directory is the usual cause: CODEX_HOME must be WRITABLE (it is staged
inside the seat's cache root). A read-only CODEX_HOME fails earlier and
differently, with failed to initialize in-process app-server client: Permission denied.
A review reporting it could not read prior verdicts
The seat is given a RENDERED, REPO-SCOPED list of prior verdicts inside its own
cache root, and GITMOOT_PRIOR_VERDICTS names the file. It states its own
as_of time and scope, so a frozen list cannot be mistaken for a live query.
Rendered rather than a copy of the database, for two measured reasons. Granting
the live store as read-only files does not work at all: SQLite opens the -wal
and -shm sidecars O_RDWR|O_CREAT, which a read-only grant refuses, so the
open fails before any verdict is read - and journal_mode=wal persists in the
header, so a quiescent database behaves the same. And a whole-database snapshot
is far too wide: it would hand every repo's jobs, prompts and results, plus
lock and fencing tokens, to a runtime that forwards what it reads to a
third-party model API. The rendered list carries only verdict fields, so
everything else is structurally absent rather than merely unmentioned.
If a seat has no single repo scope, no list is staged and the reason is recorded
in the job's dropped diagnostics rather than passing silently.
A review on the remote execution backend gets the same list at
/home/user/.gitmoot/runtime/evidence/prior-verdicts.json, also named by
GITMOOT_PRIOR_VERDICTS; when it cannot be rendered the job records a
remote_review_evidence_unavailable event with the reason.
Jobs refused with worktree root unavailable
A [worktrees] root is configured (#2375) and its preflight failed: the root
directory is missing, the mount point is not mounted, or it is mounted from a
different source or with a different filesystem type. Gitmoot refuses rather
than place worktrees on the local disk. Check findmnt <mount_point> against
mount_source / mount_fstype in config.toml, remount, and retry the job.
Worktrees already under <home>/worktrees are unaffected. gitmoot doctor
shows the unavailable root in the worktrees check.
Jobs Stay Queued With janitor_pressure_held
[janitor_capacity] is enabled and workspace-janitor does not clear a host disk
the job would write to. The event names the device and janitor's state or the
reason the answer could not be trusted (stale sample, uncovered device, other
policy hash, janitor failing). Check janitor --format json status under
data.capacity. The job starts by itself on the first dispatch pass after
janitor reports target_recovered (or cleanup_needed with
admission_hold: false); nothing needs retrying. To turn the consumer off,
set enabled = false or delete the section; it is read on every pass.
Read-Only Reviewer Seat Refuses To Start
A read-only seat runs the runtime against an ISOLATED home rather than yours, so anything the runtime reads at startup must be staged into that home first. When a startup input is missing, Gitmoot now fails BY NAME instead of letting the runtime report something unrelated.
Symptoms:
read-only seat requires runtime input "config.toml", and <path> does not exist- the host profile has no such file. This is a HARD PREREQUISITE for kimi:~/.kimi-code/config.tomlmust exist on the host, or the seat cannot start. Previously this surfaced as "No model configured" behind an auth message that pointed atkimi login, which cannot fix it.runtime input "<path>" must be a regular file- the path is a directory, socket, device or fifo. A SYMLINK is fine and is followed, so a stow/chezmoi-managed profile works; a symlink whose target is missing counts as missing, and a symlink to a directory is refused.read-only seat credential <path> is unusable: claudeAiOauth expired at <time> and carries no refreshToken- the profile stages and parses but cannot authenticate. Re-authenticate the host claude profile. An expired token WITH a refresh token is accepted, because refreshing it is the runtime's job.narrow runtime input "<path>": codex config.toml has a section header that is not readable- the file has a section header that never closes, so Gitmoot cannot locate the credentials it must strip. It refuses rather than stage a file it cannot classify. Fix the TOML or remove it from the host state dir.refuse launch: fd <n> is a <type> ...; a managed runtime's stdin/stdout/stderr must be a pipe, /dev/null or a terminal- since #2360 a managed launch (sandbox-exec, read-only and no-execution seats) passes only checked stdio to the runtime and marks every other inherited descriptor close-on-exec. A stdio that is a regular file, directory or socket could hand the runtime write access it was not granted, so the launch refuses. Daemon-run jobs always use pipes; for a hand-run command, pipe the output (... | tee log) instead of redirecting it to a file.
What the seat stages, per runtime:
| runtime | staged from | inputs |
|---|---|---|
| claude | ~/.claude | .credentials.json, narrowed to claudeAiOauth and checked for usability. Nothing is staged in gateway mode, where the gateway supplies the credential. |
| codex | ~/.codex | auth.json, plus config.toml when present - NARROWED, see below |
| kimi | ~/.kimi-code | config.toml (REQUIRED), credentials/kimi-code.json |
| omp | none | No operator profile input is staged. The daemon reads its upstream token from the owner-only file named by OMP_AUTH_BROKER_TOKEN_FILE. A minimal job-private config.yml holds the random token and loopback URL for a proxy filtered to the provider in the required provider/model selector. PI_CONFIG_DIR and PI_CODING_AGENT_DIR both resolve inside the disposable state root. |
Why the staged codex config.toml is not a copy
The seat's staged state lives inside the one writable path granted to the
sandbox, so a reviewer can read anything placed there. A codex config.toml
routinely carries credentials that have nothing to do with running the model, so
Gitmoot narrows it before staging:
[mcp_servers.*]is dropped ENTIRELY. Itsenvtable holds tokens,argscan hold them just as easily, and a read-only reviewer seat has no business spawning third-party servers. MCP servers are optional to codex, so dropping them cannot stop the seat starting.[model_providers.*]KEEPS its structure and loses onlyapi_keyandhttp_headers. Dropping the section would look stricter and would break a custommodel, because removingbase_urlandwire_apimakes the provider unresolvable.env_keyandenv_http_headersname environment variables rather than hold values, so they stay.- Everything else is kept: those are the model and sandbox settings the file is staged for.
Dotted and quoted forms are classified the same way, so
mcp_servers.github.env.TOKEN = "..." at the top level and
[mcp_servers."my server".env] are both dropped.
The staged kimi config.toml is narrowed too
kimi's config.toml is REQUIRED, and it carries more than codex's does:
measured on a live host, api_key under [services.*] and a key under
[services.*.oauth] alongside [providers.*]. For kimi the staged copy lands
directly inside the seat's writable root, so it is narrowed on the same rule:
[services.*]is dropped ENTIRELY. It configures optional tooling (moonshot search and fetch) and carries those tools' credentials.[providers.*]keeps its structure and keepsapi_keyand itsenvsub-table ONLY for the providerdefault_modelresolves to - the model's segment before the first/matched against the provider name's segment after the last:, sodefault_model = "kimi-code/k3"selects[providers."managed:kimi-code"]. Dropping every provider credential is not an option: kimi refuses to start when bothapi_keyand theenvsub-table are absent, so the seat legitimately needs exactly one.- If NO provider can be resolved, or two match ambiguously, every provider credential is withheld. A credential nobody can prove is needed is not staged.
The same rule now applies to codex: [model_providers.*] keeps api_key and
http_headers for the provider model_provider selects and strips every
other. Stripping the selected provider's key too left the seat unable to
authenticate at all, because env_key names a variable the sandbox's
environment allowlist does not pass through.
Finding out what was withheld
Narrowing is not silent. When anything is withheld the job records a
read_only_seat_config_narrowed event naming it, so a reviewer whose MCP tool
is missing - or a seat that cannot reach a provider - can find out why:
gitmoot job events <job-id> | grep read_only_seat_config_narrowed
Two more refusals, and what gateway mode withholds
Narrowing reads the file with ONE scanner shared by narrowing and provider selection, so escapes, comments, multi-line strings and unbalanced brackets are interpreted identically everywhere. Two shapes are refused rather than staged, because in both cases part of the file could not be classified, and staging an unclassified tail is the leak the narrowing exists to prevent:
config has an array or inline table that never closes- an unbalanced[or{reached the end of the file.config has a multi-line string that never closes- as before.
A triple delimiter inside a COMMENT or inside a single-line string is not a multi-line string, and a backslash-escaped quote is valid TOML that stages unchanged.
One provider shape is refused for a different reason:
codex model_provider "<name>" authenticates only through env_key, and a read-only seat's environment allowlist does not pass that variable through- set an inlineapi_keyfor that provider, or pointmodel_providerat one the seat can reach. Previously this staged cleanly and the seat failed later with the runtime's own message.
In GATEWAY mode the gateway supplies the credential, so a gateway seat stages
NO credential file for any runtime - claude's .credentials.json, codex's
auth.json and kimi's credentials/kimi-code.json are all withheld. Model
settings are still staged, narrowed as above.
Comments, and what a seat does when a config cannot be narrowed
Narrowing classifies the COMMENT-FREE part of each line and stages the line
unchanged, so an ordinary TOML comment cannot change what is withheld. Both
directions were defects: [services] # see [docs] used to have the comment's
] swallowed into the section name, staging that section's secrets verbatim,
and default_model = "kimi-code/k3" # pinned used to make the comment part of
the provider name, withholding the one credential the seat needs. Your comments
survive into the staged file.
A key/value pair is split on the first = OUTSIDE quotes, so a quoted key
containing = is kept rather than dropped.
A config that cannot be narrowed is treated by whether the runtime needs it:
- codex's
config.tomlis OPTIONAL. If it cannot be narrowed - an unbalanced[or{at end of file, an unreadable section header, or a selected provider that can only authenticate throughenv_key- the seat starts WITHOUT it and falls back to the runtime default. The file is not staged and the reason is named in theread_only_seat_config_narrowedjob event. - kimi's
config.tomlis REQUIRED, so the same refusal fails the seat by name. kimi cannot start without it, and a silent fallback would surface later as "No model configured".
In GATEWAY mode no runtime stages a credential file, and the policy the seat computes says so rather than naming files it then withholds.
Isolated worktrees duplicate gigabytes of tool cache
An isolated-worktree job re-materializing its own uv/go/npm/pip cache
inside its worktree (rather than reusing a shared one) is the largest driver of
gitmoot doctor's worktree disk figure once a fleet has run for a while — one
worktree can carry several gigabytes of immutable, content-addressed packages
that duplicate what every other job already downloaded.
Gitmoot points UV_CACHE_DIR, PIP_CACHE_DIR, npm_config_cache, GOCACHE,
and GOMODCACHE at one shared, host-level directory (default
<home>/cache/tools) for isolated-worktree jobs, so package caches are reused
across jobs instead of duplicated per worktree. This is on by default; set
[cache] enabled = false in config.toml to opt out, or [cache] dir = "/absolute/path" to relocate the shared directory (must be absolute).
A read-only/auto codex job does not get the redirect — codex never grants it
writable-path access, so pointing its tools at the shared directory would break
rather than help. Those jobs keep their pre-existing (in-worktree) cache
behavior.
If an existing worktree already carries a large in-worktree cache from before this took effect, it is cleaned up the same way as any other worktree content: by the delegation-worktree reclaim pass above, or manually once the owning job is terminal.