# Problem codes

Canonical page: https://zoomies.sh/problem-codes/

Zoomies never reports a problem without a code. The code is stable across
releases and across languages of complaint (it is what you search for, alert
on, and quote in a bug report) while the sentence beside it is written for the
person reading it and may improve over time.

Problems come from two places and behave differently:

* **The startup validator** reads `zoomies.yaml` and the `ZOOMIES_*`
  environment before anything else happens. Its findings are printed at startup
  and shown in the UI's problems panel for as long as the setting stands.
* **The controller** raises the rest while running, after every reconcile pass.
  These come and go with the fleet: a host that starts answering again clears
  `host.unhealthy` by itself.

Each code also has an **audience**, which decides who is shown it on an
instance where one team operates the controller and another uses the fleet:

* The **platform's** are what the process itself is doing wrong (its lease,
  its loops, its backups, the release it could be running) together with every
  finding from the startup validator, because each of those names a setting
  that says what the process binds, trusts, stores or logs.
* The **fleet's** are what its own pools, hosts, runners and jobs are doing
  wrong. These reach everyone who can read the list.
* One code is **both**: `controller.problems_partial`, because a list that
  could not be fully gathered has to say so to whoever is reading it.

On a single-team instance the account that installed it holds the platform
role, so the list is undivided and this changes nothing. The audience of each
code is recorded once, in `problemAudience` in
`internal/controller/problems.go`, and a test fails if a code is raised without
one; it is deliberately not repeated in the table below, because a second
copy of an answer is a second copy to get out of step.

Both use the same three severities:

| Severity | What it means |
| --- | --- |
| **Error** | Startup stops, or the fleet cannot do the thing being asked of it. A configuration error is fatal; a runtime error is work that is not happening. |
| **Warning** | Nothing stops. Something is weaker or slower than the default, and the entry says what that costs. |
| **Info** | Neither wrong nor risky, but worth knowing, usually a default that surprises people. |

A few codes change severity with the circumstances, and each says so below.
`security.disable_auth` is the clearest case: on loopback it is a warning, and
on a public address the same setting is an error.

## Configuration: the listener

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `bind.empty` | error | `server.bind` | Give it a `host:port`. Nothing can start without one. |
| `bind.malformed` | error | `server.bind` | It is not a `host:port` address. A bare port needs the colon: `:8080`. |
| `bind.public_no_tls` | warning | `server.bind` | Sessions and API tokens cross the network in the clear. Put TLS in front of it, or bind to loopback and tunnel. |
| `proxy.untrusted` | info | `server.trusted_proxies` | Client IPs in the audit log are the socket's, not the header's. Correct behind nothing; wrong behind a proxy, which is what the setting is for. |
| `proxy.bad_cidr` | error | `server.trusted_proxies` | An entry is not an IP address or a CIDR block. |
| `proxy.trust_everyone` | warning | `server.trusted_proxies` | Any client can claim any address, so the audit log's IPs mean nothing. Name your proxy's range instead. |
| `origins.any` | warning | `server.allowed_origins` | Any website can act with a signed-in operator's session. Name the origins you serve from. |
| `origins.insecure` | warning | `server.allowed_origins` | A plaintext origin is allowed to act on this controller. |
| `indexing.allowed` | warning | `server.allow_indexing` | Search engines are invited to index a fleet controller. Deliberate for a demo, rarely otherwise. |
| `tls.mode_unknown` | error | `server.tls.mode` | Not a TLS mode. The modes are `off`, `self_signed` and `files`. |
| `tls.files_missing` | error | `server.tls` | `mode: files` needs both a certificate and a key. |
| `tls.file_unreadable` | error | `server.tls.*` | The file is named but cannot be read. Usually ownership after an install as another user. |
| `tls.self_signed` | info | `server.tls.mode` | Browsers and agents will not trust it without being told to. Fine for a private network, not for anything else. |
| `external_url.missing` | warning | `server.external_url` | GitHub cannot be told where to deliver webhooks, so scaling falls back to the poller and reacts in tens of seconds. |
| `external_url.malformed` | error | `server.external_url` | Not an absolute URL. |
| `external_url.insecure` | warning | `server.external_url` | GitHub will deliver webhooks over plaintext, so the payloads and their signatures cross the internet unencrypted. |

## Configuration: GitHub

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `github.api_base_missing` | error | `github.api_base_url` | Empty. Leave it unset for github.com, or give GitHub Enterprise Server's API base. |
| `github.api_base_malformed` | error | `github.api_base_url` | Not an absolute URL. |
| `poll.disabled` | warning | `github.poll_fallback` | If a webhook delivery is lost or misconfigured, jobs queue for ever with nothing to notice. Leave it on unless you monitor delivery yourself. |
| `poll.too_fast` | warning | `github.poll_interval` | Polling this often will consume the App's API rate limit. The poller is a safety net, not the primary path. |

## Configuration: storage and secrets

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `db.path_missing` | error | `database.path` | Empty. |
| `db.parent_not_dir` | error | `database.path` | The parent exists and is not a directory. |
| `crypto.no_key` | warning | `security.encryption_key_file` | One will be generated on first start, and it is the only copy. Back it up: without it the stored GitHub App private key and webhook secrets cannot be decrypted. |
| `crypto.key_in_config` | warning | `security.encryption_key` | The key is in the config file, so anything that reads the file (a backup, a support bundle) reads every stored secret. Point at a file instead. |

## Configuration: the settings in the database

These are raised while assembling the configuration, after the database is open
and before anything is built from it. They are about the rows themselves rather
than about what any one setting says.

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `settings.stored_invalid` | warning | the setting named | A stored value will not parse, so the layer underneath it is in force; the configuration file, or the built-in default. Set it again on the settings page, or clear it with `zoomies config unset <key>`. |
| `settings.stored_unknown` | info | - | Settings are stored that this version does not have, usually because a newer one set them and this is a rollback. They are kept untouched, so upgrading again picks them up where it left off. |
| `settings.stored_unreadable` | error | - | A stored credential was sealed with a different encryption key. Starting anyway would run the fleet with credentials silently missing. Restore the key it was sealed with, or clear the setting and set it again. |
| `settings.imported_from_file` | info | - | The settings your `zoomies.yaml` spells have been copied into the database, once, on the first start after upgrading. Nothing about what the controller runs has changed; the file is still the layer underneath them. |

## Configuration: authentication

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `auth.disabled` | **error wherever the controller is reachable, warning on loopback with nothing in front** | `security.disable_auth` | Every request is treated as an administrator. Acceptable for local development; never on a host others can reach. An external URL or a trusted proxy counts as reachable, because a loopback bind behind a proxy is not private. |
| `auth.public_view` | warning | `security.public_view` | Anyone who can reach this controller reads its fleet without signing in: pools, hosts, runners, jobs, usage and runner logs. The audit log, accounts, tokens, settings, AI context and Eli stay behind sign-in, and every change still needs somebody signed in. Right for a public demo or a project that shows its CI fleet on purpose; turn `security.public_view` off anywhere else. See [Public view](https://zoomies.sh/public-view/). |
| `dind.expected` | info | `security.docker_in_docker_expected` | This fleet has said that a pool giving its jobs their own Docker daemon is expected, so that pool no longer raises `pool.dangerous` for the privileged container the daemon runs in. Nothing about the runners changed; a warning a fleet has already decided about, repeated for every such pool on every pass, is what teaches an operator to stop reading the list. The host Docker socket and persistent runners still warn. Turn the setting off to be told again. |
| `auth.cookie_insecure` | warning | `security.cookie_secure` | Session cookies go out without `Secure`, so one plain-HTTP request to this host hands over a live session. Set an https `server.external_url`, or `security.cookie_secure` if TLS is terminated in front. |
| `auth.no_login_limit` | warning | `security.rate_limit_logins` | Password guessing is not rate limited. |
| `two_step.required` | info | `security.require_two_step` | Every account that signs in with a password is asked for an authenticator code, and one without an authenticator sets it up at its next sign-in. It says what it does not cover (sessions that already exist, API tokens, and single sign-on accounts, whose second factor belongs to the identity provider) because those are what an operator turning it on is most likely to assume it does. Nothing to change. |
| `mcp_oauth.enabled` | info | `security.mcp_oauth` | Claude and other MCP clients can sign in to `/mcp` through the browser, and it names the address to give them and the `claude mcp add` line for Claude Code. It says when `server.external_url` is unset, because the addresses advertised are then taken from each request's Host header, and when `security.mcp_open_registration` is off, because only an administrator's clients can connect. Nothing to change; see [Connect Claude to Zoomies](https://zoomies.sh/connect-claude/). |
| `mcp_oauth.plain_http` | warning | `security.mcp_oauth` | MCP sign-in has been turned on where the controller is reached over plain HTTP, so authorisation codes and the tokens they buy cross the network readable. Serve the controller over https, or set `security.mcp_oauth` to false and give MCP clients an API token. |
| `mcp_oauth.no_external_url` | error | `security.mcp_oauth` | MCP sign-in has been turned on where the controller listens off this machine, and `server.external_url` is not set, so the issuer, the endpoints and the resource a token is bound to would be read from each request's `Host` header, which whoever sends the request chooses. Set `server.external_url` to the https address people use, turn `security.mcp_oauth` off, or bind to loopback behind a proxy. Left unset, `security.mcp_oauth` does not turn itself on in this shape, so an instance that started before goes on starting. On loopback nothing is refused. |
| `bootstrap.incomplete` | error | - | `ZOOMIES_BOOTSTRAP_ADMIN` needs exactly one of `ZOOMIES_BOOTSTRAP_PASSWORD_FILE` or `ZOOMIES_BOOTSTRAP_TOKEN_FILE`, and neither file means anything without it. Set all that is missing, or unset all three and use the setup token. |
| `bootstrap.ignored` | warning | - | The bootstrap variables are set on an instance that already has accounts, so this start ignored them; they only ever create the first account. The file they name is a credential kept for nothing: remove the variables and delete it. |
| `auth.session_ttl` | error | `security.session_ttl` | Must be positive. |
| `auth.session_ttl_long` | warning | `security.session_ttl` | A stolen session cookie stays useful for this long. |
| `oidc.incomplete` | error | `oidc` | Single sign-on is enabled with no issuer or no client id. |
| `oidc.no_redirect` | error | `oidc.redirect_url` | Needed, and it must be an address the identity provider can reach. |
| `oidc.insecure_issuer` | warning | `oidc.issuer` | The token exchange happens in the clear. |
| `oidc.link_by_username` | warning | `oidc.link_by_username` | A first single sign-on login can take over an existing password account with the same name. |
| `oidc.open_signup` | warning | `oidc.allow_signup` | Anyone your identity provider authenticates gets an account here. Narrow it at the provider, or turn signup off and create accounts yourself. |
| `oidc.password_login_hidden_without_sso` | warning | `oidc.hide_password_login` | The password form is set to be hidden, but single sign-on is off, so the setting does nothing yet, and the day single sign-on is turned on, everybody below administrator loses password sign-in. Turn the setting off, or configure single sign-on and check its button works first. |
| `egress.private_target` | warning | the URL's own key | A URL the controller dials, `oidc.issuer` (when OIDC is on), `github.api_base_url`, `capacity_demand.destination_url`, `agent.runner_download_url` or a backup remote's endpoint in the file, names this machine, a link-local address such as the cloud metadata service at `169.254.169.254`, or a private range (RFC 1918, carrier-grade NAT, IPv6 unique-local), in any spelling that reaches one. At startup it is a warning and never stops the controller: that value came from whoever runs the process, and an upgrade must not stop an install that works. Written through the API instead, `PATCH /settings`, a settings import (the preview marks the row), a backup remote, a direct provider or an installation's own API base URL, the same sentence is a 422 on that field. Use the service's public address, or, when it really lives on a network you own, set `security.allow_private_egress`. See [security](https://zoomies.sh/security/#securityallow_private_egress-true). |
| `egress.private_allowed` | warning | `security.allow_private_egress` | The switch that lets every URL the controller dials name this machine, a link-local address or a private network is on, so nothing checks them any more and `egress.private_target` stays silent. It is a warning, not information, because it removes the one guard between somebody with settings rights and this machine's neighbourhood; on a LAN install that needs it, the warning is the acknowledgement, and it never stops startup. Turn the setting off unless the identity provider, Enterprise Server, provider or backup remote really lives on a network you own. See [security](https://zoomies.sh/security/#securityallow_private_egress-true). |
| `egress.private_provider_allowed` | warning | `assistant.allow_private_provider` | The switch that lets an assistant provider's address name this machine, a link-local address or a private network is on. It opens only the assistant's providers, which is where a local model lives, so it is the acknowledgement a local model needs; turn it off if every provider is hosted. |

## Configuration: the agent

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `agent.backend_missing` | error | `agent.backend` | No runner backend selected. |
| `agent.backend_unknown` | error | `agent.backend` | Not a backend. They are `docker`, `podman` and `process`. |
| `agent.capacity` | error | `agent.capacity` | Must be at least 1, or the host can never take a runner. |
| `agent.workdir` | error | `agent.work_dir` | Empty. |
| `agent.process_backend` | warning | `agent.backend` | The process backend gives jobs no container isolation: a job can read and write anything the agent user can. |
| `agent.process_root` | warning | `agent.backend` | The process backend is running as root, so every job is root on the host. |
| `agent.root` | warning | `agent` | The agent process is running as root. Raised only where an agent actually runs. |
| `agent.insecure_tls` | warning | `agent.insecure_skip_verify` | The agent does not verify the controller's certificate, so anything on the path can impersonate it. |
| `agent.insecure_http` | warning | `agent.allow_insecure_http` | The agent talks to the controller over plain HTTP, so its token and every runner's credentials cross the network in the clear. |
| `agent.unverified_runner_download` | warning | `agent.allow_unverified_runner_download` | The process backend may install a runner archive whose checksum it could not confirm. |
| `agent.extra_ca_invalid` | error | `agent.extra_ca_file` | Not an absolute path to a readable PEM file with at least one certificate. Point it at your organisation's root CA in PEM form, or clear it. |
| `agent.extra_ca` | info | `agent.extra_ca_file` | Container runners on this host trust an extra CA as well as the image's own roots. Right for your proxy's CA; clear it otherwise. |
| `agent.docker_build_cache_mb` | error | `agent.docker_build_cache_mb` | Must be between 0 and 1048576 MiB; 0 disables automatic Docker builder-cache cleanup. |
| `agent.bootstrap_cpu_grace_short` | warning | `agent.bootstrap_cpu_grace` | Less than 2m of normal CPU quota before pressure throttling; registration can slow under load. |
| `runners.docker_wait_short` | warning | `runners.docker_wait` | Less than the recommended 3m for a loaded DinD daemon to become ready. |
| `agent.bootstrap_cpu_grace` | error | `agent.bootstrap_cpu_grace` | Must be between 0s and 10m; 0s applies pressure throttling immediately. |
| `agent.finished_retention` | error | `agent.finished_retention` | Cannot be negative. |
| `agent.finished_retention_long` | warning | `agent.finished_retention` | Finished runners stay on the host this long, holding disk and, for a non-ephemeral pool, whatever the job left behind. |
| `agent.heartbeat_interval_long` | warning | `agent.heartbeat_interval` | Hosts heartbeat less often than the controller's timeout, so a healthy host will be counted lost. |
| `agent.none` | info | `agent.embedded` | This controller hosts no runners itself, so at least one standalone agent has to join it. |

## Configuration: the scheduler and the rest

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `scheduler.history_sizing` | error | `scheduler.history_sizing` | Use `off`, `shadow` or `on`. |
| `security.auto_apply_remedies` | error | `security.auto_apply_remedies` | Use `off`, `shadow` or `on` (`false` and `true` are still read as `off` and `on`). |
| `scheduler.host_order` | error | `scheduler.host_order` | Use `headroom`, `largest_standard` or `best_fit`. |
| `scheduler.placement_mode` | error | `scheduler.placement_mode` | Use `headroom`, `shadow` or `readiness`. |
| `scheduler.size_routing` | error | `scheduler.size_routing` | Use `off`, `shadow` or `on`. |
| `scheduler.auto_pools` | error | `scheduler.auto_pools` | Use `off`, `shadow` or `on`. |
| `scheduler.auto_pools_docker_mode` | error | `scheduler.auto_pools_docker_mode` | Use `none` or `dind`. The host's own Docker socket is not offered to a pool the controller makes. |
| `scheduler.size_default_class` | error | `scheduler.size_default_class` | Use `small`, `medium` or `large`. |
| `scheduler.size_class_limits` | error | `scheduler.size_small_max_cpus` | The limits between the size classes are not positive, or the medium ones are below the small ones, which would leave a class empty. Set `scheduler.size_small_max_*` no larger than `scheduler.size_medium_max_*`; the defaults are 4 CPUs and 16 GB, and 12 CPUs and 48 GB. |
| `runners.class_sizes` | error | `runners.small_cpus` | The runner sizes of the classes are not positive and growing. A job is classed as the smallest runner that holds it, so a larger class with a smaller runner would never be chosen. The defaults are 1 CPU and 2 GB, 2 CPUs and 4 GB, and 4 CPUs and 8 GB. |
| `scheduler.auto_pools_host_grace` | error | `scheduler.auto_pools_host_grace` | The grace before a silent host stops counting towards its pool is not longer than the five minutes after which the fleet gives its runners up. A pool would shrink while the host's jobs were still being counted. Set it above 5m; the default is 10m. |
| `scheduler.size_wait_negative` | error | `scheduler.size_class_hold` | A size routing wait is negative. Use a duration, or 0 for none. |
| `scheduler.size_routing_without_pools` | info | `scheduler.auto_pools` | Size routing is on and automatic pools are off, so no pool is kept for a class and a job is only answered by a pool of yours that carries its size label. Turn `scheduler.auto_pools` on, or label your own pools `zoomies-small`, `zoomies-medium` and `zoomies-large`. |
| `scheduler.interval` | error | `scheduler.interval` | Must be positive. |
| `scheduler.burst` | error | `scheduler.max_creates_per_tick` | Must be at least 1. |
| `scheduler.lifetime_short` | warning | `scheduler.max_runner_lifetime` | Idle runners are recycled sooner than a long job takes, so work may be interrupted. |
| `scheduler.provision_timeout_short` | warning | `scheduler.provision_timeout` | Runners are failed sooner than they may legitimately take to start. An agent allows itself fifteen minutes for a create, because a cold image pull on a slow link is minutes rather than seconds, and a runner on a pool that provides Docker then waits for that daemon before it registers. A timeout inside the two together condemns runners that are still coming up, and the replacement pulls the same image over the same link. Set it above both, such as `20m`. |
| `scheduler.default_runner_limits_off` | warning | `scheduler.default_runner_limits` | A pool that sets no `cpus` or `memory_mb` gets runners with no cgroup limit at all, so a host's worth of them can each take every core and all of the memory; the shape that stops Docker answering. Leave it on, or set both on every pool. |
| `scheduler.auto_rerun_on` | warning | `scheduler.auto_rerun` | A job whose runner died under it is sent back to GitHub automatically, without anybody asking. That spends the installation's GitHub minutes, and a job that got as far as running may have had side effects its author expected to happen once. Turn it off to leave the re-run to the button on the job, or lower `scheduler.auto_rerun_limit`. |
| `scheduler.auto_rerun_limit` | error | `scheduler.auto_rerun_limit` | The bound on automatic re-runs is outside 1–5. Zero would make `scheduler.auto_rerun` read as on while doing nothing; a large one turns a fault the fleet causes every time into a bill. |
| `scheduler.host_throttling_off` | warning | `scheduler.host_throttling` | An overwhelmed host is never stepped down: the pressure holds still refuse new starts while CPU or memory is acutely short, but nothing outlasts a sample, and the runners already on the host are never slowed. Leave it on unless something outside Zoomies manages the hosts' load. |
| `runners.docker_wait` | error | `runners.docker_wait` | The runner image accepts a wait of one second to one hour and exits with a configuration error for anything else, so no runner on a Docker pool would take a job. Use a duration in that range, or 0 to leave the image's default. |
| `runners.env_reserved` | error | `runners.env` | It names a variable the controller writes for each runner individually; its JIT configuration, name, labels, group or credentials. One value for the whole fleet is wrong for every runner in it; remove it. |
| `updates.interval_negative` | error | `updates.check_interval` | Must not be negative. Use a duration, or 0 to never ask. |
| `updates.interval_too_fast` | warning | `updates.check_interval` | Releases are published far less often than this, and the check is unauthenticated. |
| `updates.mode` | error | `updates.mode` | Not a release update mode. Use `off`, which says that a release exists and nothing more; `manual`, which adds the Update buttons and moves nothing without a click; or `auto`, which takes a release once it has been public for `updates.soak`. Auto updates a machine only through its own update helper: while a release is due and the helper beside the controller is not installed, auto waits and updates no host either (where that helper cannot be installed, the controller is left to a person and hosts follow the release it runs). Hosts go one at a time, nothing starts while the controller is fenced, and a failed update halts the rollout until an administrator resumes or cancels it on the Hosts page (or with zoomies updates resume or cancel), where a running rollout can be cancelled too. |
| `updates.soak_negative` | error | `updates.soak` | Must not be negative. Use a duration such as `24h`, or 0 for no wait. |
| `updates.mode_without_check` | warning | `updates.mode` | The mode is `manual` or `auto` and `updates.check_interval` is 0, the air-gap switch, so this controller never learns of a newer release and is never updated from here. In `auto`, hosts whose update helper is installed are still taken to the release the controller runs, once you upgrade it by hand. Give the check a duration such as `24h`, or set the mode to `off`. |
| `updates.auto` | info | `updates.mode` | The update mode is `auto`, which is meant to install new releases without anyone asking: the controller takes the newest once it has been public for `updates.soak`, and the hosts that have opted in then follow it, one at a time. There is no automatic rollback, because a migration is one way. Nothing to change if that is what you want; `manual` keeps the decision with a person. Auto updates a machine only through its own update helper: while a release is due and the helper beside the controller is not installed, auto waits and updates no host either (where that helper cannot be installed, the controller is left to a person and hosts follow the release it runs). Hosts go one at a time, nothing starts while the controller is fenced, and a failed update halts the rollout until an administrator resumes or cancels it on the Hosts page (or with zoomies updates resume or cancel), where a running rollout can be cancelled too. |
| `updates.auto_without_soak` | warning | `updates.soak` | The mode is `auto` and the soak is 0, which removes the only wait between a release being published and every host running it. A release that turns out to be broken, or is replaced by a fix within hours, would be taken before anyone has had the chance to notice. Use `24h`, the default, or set the mode to `manual`. |
| `github.toolchain_scan_negative` | error | `github.toolchain_scan_interval` | Must not be negative. Use a duration such as `24h`, or 0 to scan only when asked. |
| `github.toolchain_scan_too_fast` | warning | `github.toolchain_scan_interval` | Every workflow in every repository is read more often than hourly. Each scan spends calls from the GitHub quota the scheduler uses to find queued jobs, and workflows change far less often than that. |
| `images.refresh_negative` | error | `images.refresh_interval` | Must not be negative. Use a duration, or 0 to leave images alone. |
| `images.refresh_too_fast` | warning | `images.refresh_interval` | Every pool's image is checked on every host far more often than an image is built. |
| `images.refresh_off` | info | `images.refresh_interval` | Nothing refreshes runner images, so a pool naming a moving tag keeps whatever its hosts pulled first. Expected on an air-gapped fleet, or one that pins every pool to a digest. |
| `retention.jobs_short` | info | `retention.jobs` | Job history is kept for less than the week the sizing advice counts. Advice to give a pool a smaller runner, or a host smaller runners, rests on what a week of jobs used, so it is not given. Keep a week or more. |
| `retention.audit_renamed` | info | `retention.audit` | The key was renamed to `retention.scaling_events`, which is all it ever bounded; audit rows are never pruned. The value is still honoured. Rename it. |
| `backup.remote_incomplete` | error | `backup.remotes` | A remote has no endpoint or no bucket, so nothing would be copied to it and nothing would say so. Finish it, or set `disabled: true` until it is ready. |
| `backup.remote_endpoint` | error | `backup.remotes` | A remote's endpoint is not an HTTP URL. Write the service's own, such as `https://s3.eu-west-2.amazonaws.com` or `http://minio:9000`; the scheme is what decides whether the connection is encrypted. |
| `backup.remote_credentials` | error | `backup.remotes` | A remote has no access key or no secret key, so every request to it would be refused and the copies would pile up unsent. |
| `backup.remote_name` | error | `backup.remotes` | A remote's name is not usable: it is a path component in the API and a word in a log line, so it is lower-case letters, digits and dashes. |
| `backup.remote_duplicate` | error | `backup.remotes` | Two remotes share a name, so one of them would be unaddressable by the Backups tab, the log and the problems drawer alike. |
| `backup.remote_insecure` | warning | `backup.remotes` | A destination is reached over plain HTTP, so the access key, the signature and the backup itself cross the network in the clear, anyone on the path can read the fleet and write to the bucket afterwards. Use `https://` unless the endpoint is on this host or a network you own end to end. Raised by the validator for a destination in the file and by the controller for one added on the Backups page; loopback is not warned about. |
| `backup.remote_plaintext` | warning | `backup.remotes` | A destination is sent the backup unencrypted. A backup is the whole fleet (every repository and job it has seen, every account, and the sealed GitHub App credentials) and in a bucket it is a file anyone who can read the bucket can open. Set a passphrase on it and keep that wherever you keep the encryption key; nothing here can recover a lost one. Raised about a destination in the file and about one added on the page alike. |
| `backup.remote_passphrase_short` | warning | `backup.remotes` | A remote's passphrase is shorter than the eight characters the Backups tab's encrypted download accepts. The archive is only as private as this, and it is typed once, into a file. |
| `backup.no_remote` | info | `backup.remotes` | Backups are taken on a schedule and nothing in `zoomies.yaml` says where a copy goes. That is a backup against a mistake, not against the disk, the machine or the datacentre. Add a destination on the Backups page or under `backup.remotes`, or keep shipping the directory yourself; the point of the entry is that one of the three is somebody's job. It is raised from the file, so the Configuration page drops it once the fleet has a destination stored from the Backups page, which the validator cannot see. |
| `capacity_demand.url` | error | `capacity_demand.destination_url` | Not an absolute HTTP URL. |
| `capacity_demand.secret` | error | `capacity_demand.signing_secret` | Empty, so deliveries could not be signed and a receiver could not tell them from anyone else's. |
| `capacity_demand.cooldown` | error | `capacity_demand.cooldown` | Must be positive. |
| `capacity_demand.timeout` | error | `capacity_demand.timeout` | Must be positive. |
| `metrics.public` | warning | `metrics.public` | The metrics endpoint answers without authentication, so anyone who can reach it learns the shape of the fleet. |
| `status.public` | warning | `status.mode` | The fleet status at `/status`, `/status.svg` and `/api/v1/status` answers without an account. It names nothing, but anyone who can reach the controller learns that the fleet exists, which release it runs, whether it is blocked, roughly how busy it is and the codes of its current problems. Set `status.mode` to `authenticated` if only people with an account should see it. |
| `status.public_no_tls` | error | `status.mode` | `status.mode` is `public` on a listener that is not on loopback and does not terminate TLS, so the address it invites people to would carry every sign-in in the clear. Set `server.tls.mode` to `self-signed` or `files`, bind `server.bind` to loopback behind a proxy that terminates TLS, or set `status.mode` to `authenticated`. |
| `status.mode` | error | `status.mode` | Not a mode. They are `off`, `authenticated` and `public`. |
| `log.level` | error | `log.level` | Not a level. They are `debug`, `info`, `warn` and `error`. |
| `log.format` | error | `log.format` | Not a format. They are `text` and `json`. |
| `log.debug` | info | `log.level` | Debug logging is on, which is loud and includes request detail. |
| `limits.loopback` | warning | the `limits.*` key named | A ceiling is set on a controller only this machine can reach. It guards an instance many people reach against one caller spending what the others need; with nobody else able to reach this one it can only refuse you. Set it back to 0, or leave it if the controller is about to go behind a proxy or get an external URL. |
| `limits.negative` | error | the `limits.*` key named | A ceiling cannot be negative. Set it to 0 for no ceiling, or to the most this instance should hold. |
| `kennel.scope` | error | `kennel.scope` | Not a scope Kennel Club knows. They are `served`, the repositories this fleet has run a job for, and `installation`, every repository the GitHub App can see. |
| `kennel.refresh_interval` | error | `kennel.refresh_interval` | Shorter than an hour, or zero. Every refresh spends requests from a limit shared with everything else the installation does, and what is read changes over days, so a shorter interval buys nothing and zero would never refresh. Use `24h`, the default, or anything of an hour or more. |
| `kennel.api_budget` | error | `kennel.api_budget_percent` | Outside 5 to 50. Below 5% a refresh would not finish before the next was due; above 50% Kennel Club would compete with the scheduler for the requests it needs to place a runner. The default is 20. |
| `kennel.unknown_check` | error | `kennel.disabled_checks` | A name that is neither a check's code nor an area. It would turn nothing off, so the check you meant to turn off would go on running. The finding lists the names that are valid. |
| `ui.capacity_map.layout` | error | `ui.capacity_map.overview_layout`, `ui.capacity_map.hosts_layout` | Not a layout the host capacity map can open in. They are `overlay`, every host on one chart, and `split`, a chart for each. |

## Configuration: infrastructure providers

Every code here is silent while `provider.enabled` is false, which is the
default. A deployment that does not rent machines is not told how to bound
something it is not doing. The one exception is the runtime code
`provider.renting_off`, which exists to say that a configured provider is
being held back by that very switch.

| Code | Severity | Setting | What to do |
| --- | --- | --- | --- |
| `provider.interval` | error | `provider.interval` | Must be positive. It is how often machines are reconciled. |
| `provider.timeouts` | error | `provider.create_timeout` | A provider operation has no bound, or `provider.ambiguity_timeout` is not longer than `provider.create_timeout`, which would quarantine machines that are merely still being built. |
| `provider.enrol_timeout` | error | `provider.enrol_timeout` | Shorter than the silence that loses a host, so machines that did arrive would be given up on. |
| `provider.no_ceiling` | warning | `provider.max_machines` | Providers are on but the ceiling is none, so nothing will be rented however much work queues. A maximum of none is none, as it is for a pool's `max_runners`. |
| `provider.paused` | info | `provider.paused` | New machines are held by configuration. Draining, deleting, recovery and ownership checks all continue. |
| `provider.delete_grace_short` | warning | `provider.delete_grace` | A machine whose host goes briefly quiet would be destroyed mid-job. Ninety seconds of silence only makes a host unhealthy; the runners on it are not given up until five minutes, and until they are the fleet still believes they are running jobs. Set it well above that, such as `10m`. |
| `provider.idle_timeout_short` | warning | `provider.idle_timeout` | A quiet host carries no runner anything will place, so it reads as idle from the moment it falls silent. Set at or below the five minutes its runners are given, a machine is drained for a network blip rather than for being unwanted. Set it above that, such as `15m`. |
| `provider.scale_down_fast` | warning | `provider.scale_down_cooldown` | Machines are removed sooner than one idle period, so the quiet between two bursts pays the creation cost again. |

## Runtime: hosts and installations

| Code | Severity | What it means | How to see it worked |
| --- | --- | --- | --- |
| `host.unhealthy` | **error with runners on it, warning without** | The host has stopped heartbeating. With runners recorded on it their state is unknown, which is worse than a spare host being down. | The host's card turns green and the entry leaves the problems list on the next heartbeat; `zoomies hosts list` shows it connected. |
| `host.os_health` | **error when a counted check could not run, warning otherwise** | A connected host's latest OS report has a finding in the checks `zoomies doctor` counts by default; the safe tier, without optional checks: file watches, Docker log rotation, free disk and the like, or a check that could not run properly. The entry names up to three and says how many more. The aggressive and dedicated tiers are choices rather than faults, so they are suggestions on the host's page and never raise this. A pending reboot is not counted here: it has its own entry, `host.reboot_pending`, so a host whose only finding is a reboot raises that and not this. One entry per host. Zoomies does not change the operating system from here: `sudo zoomies doctor --verbose` on the host says what each check found (from another machine, the host's page has a command for `zoomies doctor --host` with a short-lived token in it), and `sudo zoomies doctor --interactive` on the host offers each fix. Nothing is raised for a host that has sent no report, one that is offline (`host.unhealthy` speaks for it) or a container's partial report. It never reaches the public status page. | The host page's health pill clears, or shows the check as accepted, once the next report arrives; `sudo zoomies doctor` on the host counts no finding. |
| `host.health_stale` | warning | A connected host that has sent an OS report before has sent none for ten minutes, so what its page shows may no longer be true. Its collector has most likely stopped, or its clock is wrong, because the age is judged by the report's own timestamp. The host page greys a badge after three minutes; this is raised only after ten. A container's partial report is produced afresh every minute and is not judged. It clears with the next report. It never reaches the public status page. | The host page's report age drops below three minutes and the entry leaves the list with the next report. |
| `host.reboot_pending` | info | An update installed on a connected host takes effect only after it restarts, usually a newer kernel. It stays until the host has rebooted, whether or not anything is running on it. The entry says when there is no runner on the host, so it can be rebooted now. Otherwise it says to cordon the host and reboot it when `zoomies runners list --host <host-id> --state busy` shows nothing, because a runner kept warm for a pool waits for work and never finishes by itself, and a count of runners cannot tell it from one holding a job. Zoomies never reboots a host. It never reaches the public status page. | The host page stops showing a pending reboot after the host comes back, and the entry leaves the list. |
| `host.update_failed` | error | The latest update of a host from here ended as failed or timed out, and its agent still runs the release it had, placing work as normal on it. On a controller built from `main` it also ends as failed when the dev channel served the host a different build from the controller's, because the helper can only name the channel, and the host's card says so. The entry names the host and the release it was asked to take, and never carries the reason itself: the update helper's sentence can name a path on the host, so it is on the host's card for the `platform` role, and every other role reads a fixed sentence there. It offers no one-click fix. Look at `journalctl -u zoomies-update` on the host, where `sudo zoomies updates helper status` says what the helper did last, then update it again from the Hosts page, or by hand with the command on its card. It never reaches the public status page. | The entry leaves the list when a new attempt for the host opens, or when the host reports the release it was asked to take, however it got there. Builds from `main` cannot be ordered against anything, so when the host was asked to take one, or this controller is built from `main`, it also leaves when the host reports the build this controller would take it to now. |
| `host.update_helper_stuck` | error | The update helper on a host had stopped reading requests when an update was handed to it, so the attempt failed before anything ran and the host still runs the release it had, placing work as normal. The agent found `request.json` waiting in the update folder and looked at why: the helper's path unit disabled, inactive or failed (systemd stops it when the service starts too often), or a request older than a healthy helper takes. The entry names the host, the release it was asked to take, the attempt, how long the request had waited and what systemd says of the two units, and carries no path, so every role reads it. The agent cannot touch the root-owned units, so it asks root for a repair through the update folder: the repair clears the units' failed state, rewrites and starts the path unit, and only then removes a request or `upgrade.lock` that is older than 15 minutes with no helper or upgrade running. It does nothing while the helper or an upgrade is running, and repairs at most three times a day and ten minutes apart. The entry appears when the repair did not put the helper right, was held back by those limits, or was not answered, which happens on a host whose repair units were never installed. Run `sudo zoomies updates helper install` on the host, which does the same by hand and installs the repair units, then `sudo zoomies updates helper status`, then update again from the Hosts page. A repair that worked is not a problem: it is in the audit as `update.helper_repair`. It offers no one-click fix and never reaches the public status page. | The entry leaves the list when a new attempt for the host opens, or when the host reports the release it was asked to take, however it got there. |
| `host.update_unavailable` | info | `updates.mode` is `manual` or `auto` on a controller built from a release, and a host runs an older release than the controller but its agent does not offer to update itself, which it does only once the update helper is installed on the host. The agent inside the controller, a host whose agent speaks another protocol (it has a louder entry of its own) and a host on a build that cannot be ordered against a release are left out. Nothing is wrong meanwhile. Run `sudo zoomies updates helper install` on the host so it can be updated from the Hosts page, or update it by hand with the command on its card. Where the helper can never be installed on the host (it is not Linux, systemd does not run it, or its agent runs in a container under a rootless runtime, as the agent reports and the host's card says), the entry says which, and its fix is the command on the card alone. It never reaches the public status page. | The host's card offers its Update button, or the host reports the controller's release, and the entry leaves the list. |
| `host.cordoned_with_work` | warning | A cordoned host could run jobs that are queued. Cordoning is deliberate, so this is a reminder rather than a fault. | The entry leaves the list once the host is uncordoned or the queued jobs have been taken by another host. |
| `host.duplicate_agent` | warning | Two agent sessions have used this host's credentials in turn. An agent takes a new session each time it starts and never returns to an old one, so alternation means a second agent holds a copy of the token, usually a cloned VM or a copied state directory. Neither is refused: both are running real jobs, and picking one would end the other's. | After the copy is stopped and the token rotated, the host's sessions stop alternating and the entry leaves the list within a few heartbeats. |
| `host.runtime_recovering` | warning | A host's agent reports that its container runtime failed (it could not be reached, or it did not answer in time) and it is holding new starts until one recovery attempt. Running jobs continue. The entry names which failure in a row this is, when the agent reported it and when the attempt is due, on the controller's clock, with the backend's own error, shown as prose, so a backtick in what the agent sent reads as an apostrophe and cannot offer a command to copy. For a runtime that cannot be reached, start it or give the agent's user its socket (for Docker, `sudo systemctl start docker`); for one that is slow, look at its load and disk and lower the host's capacity if it carries more than the machine can. It is kept on the host row, so a controller restart does not lose it, and clears on the next heartbeat after a start succeeds. | The entry leaves the list once the agent's recovery attempt succeeds; the host's card no longer shows a held start. |
| `host.image_pull_failed` | warning | A runner start or prewarm on a host failed because its pool's image could not be made ready. The entry names the host, the pool, the image and the registry it comes from (`docker.io` for a bare name, otherwise the reference's host with its port) so a host whose egress blocks `ghcr.io` says so instead of showing runners that never register. Check the host can reach the registry and is logged in to it for a private image (`docker pull` on the host gives the daemon's own answer), and that the tag exists. The command is offered only for an image that is shaped like one, and the error and the names in the entry are shown as prose, so text an agent reported cannot put a command of its own in front of you. Cleared by the next start or prewarm of the same pool that succeeds on that host. | The next runner start on the host succeeds: `zoomies runners list --host <host-id>` shows one past `provisioning`, and the entry leaves the list. |
| `host.throttled` | warning | One or more hosts have been stepped down after sustained pressure: CPU pinned with a runner no limit binds, a load average past twice the cores, or memory at the reserve. The entry carries each host's own sentence, how many of its slots it is left, which measurement did it, what the running jobs are getting (never less than half their CPU allocation), and that it lifts one step after five minutes of calm. Wait for it, lower the host's capacity or the pools' limits so its runners fit the machine, or lift it from the host card (**Lift the throttle**) once the cause is fixed; a cleared throttle comes back on the next heartbeat if the pressure is still there. A host that keeps being throttled has too many slots for its machine. | The host's card no longer shows it stepped down and `zoomies hosts list` shows its full slot count once pressure has eased. |
| `host.slots_below_capacity` | info | A host's capacity says it takes more runners than its machine holds of the standard runner size it was given (set to three slots, holding one because a runner is six of its seven CPUs) while a pool that reaches it has had its typical job wait at least 30 seconds for a runner, over at least five jobs in the last six hours, read from the jobs themselves, so a restart does not forget it. Without that it is silent: big runners may be exactly what was wanted, and a capacity above what they allow is only a ceiling. It is also silent for a host the controller has stepped down (`host.throttled` already says what to do), because that host is held back by load and changing its runner size would lift the step-down in the same write. The size is also withheld, with the reason in the detail, when a pool that takes its size from the host has had a job killed for memory in the last week, or its jobs used more than the smaller runner would hold with half as much again to spare: a smaller runner holds more of them, which is no use to a job that needed the memory. The detail names the machine, the capacity, the runner size and which pools waited. When the controller can price a smaller size it carries a remedy, applied with one click or `zoomies problems apply`: it asks every pool that reaches the host how many runners it would have room for there before and after, and proposes the size only if none loses any and the host gains some. The effect says how many runners the host holds, not the pools' rooms added together. Rooms are counted on what the machines can hold: a host that is reporting, uncordoned and compatible counts, and a temporary hold on new starts does not take it out. Otherwise the detail says what stands in the way; the size is below the host's own smallest runner or below a pool's, or a pool's minimum charges each runner more than the standard, in which case lower that pool's smallest runner. Give each runner less in the host's Runner sizes, or lower the capacity if the big runners are what you want. | After the size or the capacity changes, the host's card shows the slots its machine holds and the entry leaves the list on the next pass. |
| `pool.minimum_overcharges` | info | A Docker-in-Docker pool sized by its hosts charges every runner far more memory than its jobs have used. The pool's smallest runner is a figure per container, so a thin sidecar multiplies it: a 1.5 GB minimum with the sidecar holding 15% of the memory is a request for a 10 GB slot, because only that gives the thinner half 1.5 GB. The machines pay for it, holding fewer runners than they could. Raised only when the pool's typical job has waited at least 30 seconds for a runner in the last six hours, over at least 20 jobs with a measured peak in the last week, none killed for memory (a pool that has been killed for it is never advised to shrink) and the most any job used, with half as much again, is well under the charge (by a quarter or more). A job's peak is its two containers added together, but each has a limit of its own, so the floor is also sized from the most each half used in any job of the last week, kept per job as its heartbeats arrive, and from the last few hours of heartbeats under the same coverage rule as the sidecar share (60 samples, three runners, half an hour, none more than half), whichever is larger: a thin sidecar that has used most of what it is given, in a weekly build as much as in a recent one, keeps the floor where it is. The advice waits until at least 20 jobs have been measured that way, which after an upgrade is a week of jobs, and with no such samples it waits too. Memory only: it is a limit a job is killed at, so what jobs used is evidence of what a slot has to hold, while a build's CPU peak is a burst the elastic valve lends for. When the controller can price it, it carries a remedy that lowers the smallest runner to what keeps every slot at 1.5 times the most a job has used, proposed only if the pool's hosts then have room for more runners; otherwise the detail says lowering it would not help and the limit is elsewhere. | After the smallest runner is lowered, the pool's hosts show room for more runners and the entry leaves the list on the next pass. |
| `pool.minimum_held_by_job` | info | One kind of job holds a Docker-in-Docker pool's floor where it is. The pool's evidence is by pool, so a single heavy job (a weekly build, a release) is all it says, and every runner carries a slot sized for it; `pool.minimum_overcharges` stays silent because the floor has to hold that job. This notice is raised when the same conditions hold (jobs have waited, at least 20 measured jobs in the last week, none killed for memory, the sidecar evidence covers the pool) and, with the heaviest job name set aside, the rest of the pool's jobs used at most a figure that would put the slot at least a quarter lower and give the pool's hosts room for more runners. It names the job, its peak and how often it ran, says what the rest would fit in, and **proposes nothing to apply**: lowering the smallest runner before that job is somewhere else is what gets it killed. Move it first (a pool of its own, with a label only that job asks for, is the certain way; with `scheduler.size_routing` on, a size label or `zoomies size pin` sends it to a larger class of the pool's own runners; `scheduler.history_sizing=on` gives it a larger runner from what it used before, though only after a run or two) and then lower the smallest runner. Silent when there are too many job names to be sure which holds it. | Once the heavy job is pinned to a larger class or given its own pool, the entry leaves the list and `pool.minimum_overcharges` can speak for the rest. |
| `pool.queue_wait_high` | info | A pool that keeps no runner warm (`min_runners` 0) has jobs that waited five minutes or more at the 95th percentile for a runner in the last six hours, over at least 10 jobs, while the median is short: a burst of jobs waits for the runners ahead of it to be made. Not raised for a pool the controller keeps, whose minimum is worked out from its hosts, or for one that already has a minimum. It carries a remedy that keeps two runners warm (fewer if the pool's maximum or its hosts' room is smaller), which holds idle capacity for the pool; if no host has room for a runner of the pool, the detail says so and there is no remedy. | The p95 wait on the pool's page falls below five minutes over the next six hours once a runner is kept warm. |
| `jobs.workflow_failing` | info | A job failed at the same step three or more times in the last six hours and did not pass once in that time, on this fleet's runners or on GitHub's hosted ones. It names the step and where the job ran, and has no remedy: the fix is in the workflow or the repository's settings, and the fleet only saw it. Each pattern is one problem, newest failure first. | The job passes at the step it was failing on: `zoomies jobs list --workflow <name> --failed` shows no new failures and the entry leaves the list. |
| `jobs.duration_regressed` | info | A job's typical run over the last day took at least half as long again, and at least a minute more, than over the seven days before, with at least ten measured runs in each window, on jobs this fleet ran. It carries no remedy: the detail says to compare a run with an older one and to group the job's runs by host and controller version, which tells a slow workflow from a slow machine. The check reads eight days of jobs, so it reads them at most once a minute, and an entry can lag the jobs by that long. | The job's typical duration on the Jobs page falls back towards the seven-day figure and the entry leaves the list after the next day of runs. |
| `host.work_concentrated` | info | A host has been throttled after sustained pressure while it is running runners, and one or more other hosts are available, uncordoned, running nothing and under 30% CPU, and a pool that runs on the busy host could run on them (selector, backend and platform). Raised only while `scheduler.host_order` is the default, `headroom`, which prefers the host with the most left afterwards: the largest machine (often the one that also runs the controller) is the first choice every time a job needs a runner, even throttled, because it still has more left than a small idle one. No change is proposed, because what helps is a decision that depends on what the large host is for: lower its capacity so it takes fewer runners at once. Changing `scheduler.host_order` is not the answer on its own: `best_fit` packs the fullest host first and `largest_standard` prefers the host where a pool's runner is biggest, and both keep choosing a large host that `headroom` would have left. It clears as soon as the host is no longer throttled or no other host is idle. | New runners land on the idle hosts: `zoomies runners list` shows them spread, and the entry leaves the list once the busy host is no longer throttled. |
| `host.memory_pool_exhausted` | warning | A runner on this host, of a pool set to lend, needed more memory than it was created with and the [memory valve](https://zoomies.sh/elastic-memory/) had none to lend: either everything no runner's guarantee needs was already lent, or what was left is the free memory the valve keeps back for the host itself (its reserve and a twentieth of its RAM). The entry says how much is lent and how much is held back. The runner was left at the limit it had, as it would be with the valve off, so this is a sizing signal and not a fault: lower the host's capacity, give its runners a smaller share, or put the memory-hungry pool on a larger host. It clears fifteen minutes after the last refusal, whether or not the runner that met it is still there. A pool that only observes never raises it: nothing was lent, so a runner that would have been refused is evidence on the runner and in the decision counts. | The next run of the job on the host completes without a kill, and the host page's memory valve shows headroom. |
| `host.overprovisioned` | warning | A measured host has more slots than its machine can carry: more than it has allocatable CPUs, or more than it has 2 GB of allocatable memory for. Where every pool that can place on the host has a minimum below a core or 2 GB, that minimum is the size a slot is judged against instead, because those runners are accepted at it. A slot is two containers wherever a `docker_mode: dind` pool places on the host, whether or not it typed its own CPU and memory; a pool that typed them is given those figures a second time for the daemon its builds run in, and a pool sized by its host still splits one slot between a runner and that same daemon, so either way the machine has to be twice the size per slot and the detail names the pool that made it so and whether it typed its limits. The detail names the machine, the capacity and, with default limits on, the share each runner is being given, which is what a job there crawls on; with them off it says nothing limits the runners at all. The share is only given where the host's daemon can enforce it and the pool runs on a container backend; elsewhere nothing limits the runners either. The fix names the largest capacity that fits, never below one, or says the machine is too small to run a runner well when even one slot does not. Lower the capacity on the host card or with `PATCH /api/v1/hosts/{id}` (a host's capacity is decided once when it joins and a heartbeat never rewrites it, so `--capacity` on a fresh join token applies only at the next join, and `agent.capacity` only sets the figure when the embedded host first enrols, not afterwards) or add a host. A `dind` pool sized by its host that cannot give both containers a comfortable share at all is refused the host outright rather than merely counted here, see [hosts and pools](https://zoomies.sh/hosts-and-pools/#default-allocations). | After the capacity is lowered or the pools' minimums raised, the host's card shows slots its machine can carry and the entry leaves the list. |
| `host.limits_unenforceable` | warning | A host's Docker or Podman daemon has said it cannot apply a CPU quota, a memory limit or a pids limit. A CPU quota it cannot apply is refused at create, so a pool that sets one fails every runner it starts there; a memory limit is dropped, so the pool's limit binds nothing; a pids limit is ignored. No default is given on that field either way. On a rootless daemon the fix is to delegate the controllers to its user (`Delegate=cpu cpuset io memory pids` in a drop-in for the user slice); on a root daemon it is cgroup v2, or a kernel built with the controller that is missing. | Runners start on the host with their limits applied: a runner's resource sample on its page names a quota, and the entry leaves the list. |
| `host.limits_unverified` | info | A host's agent predates the limits probe, so the controller does not know whether its daemon can apply a limit and gives its runners no default CPU or memory, because a limit sent to a daemon that cannot apply it fails the create. Pools that set their own limits are unaffected. Upgrade the agent: the probe arrives with the next heartbeat, with no re-join, and defaults start on the next runner. Raised only while `scheduler.default_runner_limits` is on, since with it off an unverified probe changes nothing. | After the agent is upgraded, the host page reports whether its daemon applies limits and the entry leaves the list. |
| `host.shared_folder_unmounted` | warning | A host cannot give its runners a kept tool cache, for one of two reasons the detail names. The first: a containerised agent (the controller's embedded one, or an agent in a container) runs runners, but its container does not mount the [shared folder](https://zoomies.sh/configuration/#the-shared-folder) from the host at its own path. The folder it would use is inside its own data volume, where the host's Docker daemon, which binds runners' caches by that path, cannot see it; handing it out would give every runner an empty cache folder owned by root that no setup action can write to. So the agent keeps no tool cache instead, and the pools that keep one, named in the detail, run their jobs without it. The second: a tool cache folder in the shared folder exists but the runner (uid 1001) cannot write to it (typically one the daemon created as root before the mount was fixed) which would fail every setup action that downloads with `EACCES`; runners start without it instead. Raised only while such a pool could run on the host. Add the mount, the detail gives the line, or run `zoomies upgrade --yes` on the host, which offers it; open or delete the unwritable folder as the detail says. The next heartbeat after the fix clears it. | After the mount or the agent's volume is added, the host page shows the tool cache as kept and the entry leaves the list. |
| `host.scratch_unmeasured` | info | A host's agent cannot tell how full its runners' in-memory folders get, so full-folder detection is off there: no `jobs.scratch_full` appears for a job that fills one, and the job's explanation cannot say a folder was why. The agent measures a folder by running `df` in the runner's container through the container engine's API, which needs nothing the agent does not already have, and it says why it could not once the host's runners have been sampled for two minutes and none of them has had a folder measured. The reason decides the change. `exec_refused`: the engine, or a socket proxy or authorisation plugin in front of it, refuses to run a command in a container; allow exec. `no_df`: the runner image has no `df`; use one with coreutils or busybox, as the stock image has. `unreadable`: `df` printed something that is not the table the agent reads; run `docker exec <runner container> df -P -k /tmp` on the host to see what. `timed_out`: the engine did not answer in time because it is overloaded or stalled, and the host's other entries say how. Raised only for a host with runners that keep a folder in memory, and not for a cordoned or incompatible host. | The next sample after the change measures a folder, and the entry leaves the list. |
| `host.resources_unknown` | **warning with default limits on, note otherwise** | One or more hosts have reported no CPUs, no memory and no disk. They are placed by slot count alone, which is how every host was placed before agents learnt to measure themselves, so nothing is broken, but a pool's resource limits cannot be fitted against a machine nobody has measured, every figure on the host's card is missing, and with `scheduler.default_runner_limits` on its runners get no default either, because their share of an unmeasured machine cannot be computed. A pool with no limits of its own therefore runs unlimited there, which is the shape defaults exist to stop, and that is what makes it a warning. Upgrading the agent fixes it on the next heartbeat, with no re-join. | After the agent is upgraded, the host's card shows CPUs, memory and disk and the entry leaves the list. |
| `installation.unhealthy` | error | The GitHub App installation is not usable; the App was uninstalled, its key was rotated, or its permissions were changed. Nothing can register until it is fixed. An App that is merely not subscribed to `workflow_job` is *not* this: the fleet works on the fallback poller, more slowly, and `webhook.never_received` is the entry for it. | |
| `webhook.rejected` | warning | Deliveries arrived and were refused, almost always a signing-secret mismatch. | |
| `webhook.never_received` | warning | No webhook has ever arrived, so scaling is running entirely on the poller. | |
| `tailcat.unavailable` | warning | The controller's private-connection listener has no Tailcat relay it can reach, so hosts enrolled with a [private connection](https://zoomies.sh/private-hosts/) cannot heartbeat or take work; direct hosts are unaffected. The entry carries the last attempt's reason. It is the platform's, because the fix is the controller's own outbound access. Nothing needs restarting: the controller retries every twenty seconds, tries the relay its identity was sealed with first and another only while that one does not answer, and the entry clears once one does. A listener that cannot start no longer stops the controller starting. | |

## Runtime: pools, jobs and runners

| Code | Severity | What it means | How to see it worked |
| --- | --- | --- | --- |
| `pool.no_capacity` | **error with jobs waiting, warning without, and a warning even with jobs waiting when every host that could run the pool is merely full** | The pool cannot start the runners it wants. The entry carries the scheduler's own reason: no host matches its selector, every host that could run it is full, or every host is cordoned. | A runner starts for the pool: `zoomies runners list --pool <pool-id>` shows one past `provisioning`, and the entry leaves the list. |
| `pool.runners_failing` | **error with jobs waiting, warning without** | Runners are being created and dying before they register. The fix is the category of the most recent failure (an image that cannot be pulled, a registration GitHub refused, a container backend that is not answering) and the usual causes only when nothing narrowed it. This is the failure with no failed job behind it: the jobs stay queued, nothing is marked failed, and every other count reads as a fleet that is merely busy. | Runners of the pool reach `idle`: `zoomies runners list --pool <pool-id> --state idle` is not empty, and the entry leaves the list. |
| `pool.github_rate_limited` | warning | The pool has jobs waiting and its installation is inside a GitHub rate-limit backoff, so the scheduler is creating nothing for it rather than spending another call on a quota that is already gone. It lifts on its own when `poller.paused` does. | The entry leaves the list when `poller.stale` clears and the pool's runners start again; the pool's page no longer names a backoff. |
| `scheduler.registration_throttled` | warning | The last scheduling pass had a host chosen for a runner and did not create it, because the installation was already at `scheduler.registration_concurrency` credential requests in flight. Nothing fails: the demand is kept and a later pass takes it, so it shows as runners appearing slowly. It is listed because it is the one limit that used to bind silently; the pool would report jobs waiting, and an operator reading that adds hosts, which cannot help when hosts are not what ran out. Raise the setting if the installation's GitHub quota has room; leave it if the fleet is deliberately gentle with that quota. | |
| `controller.lease_lost` | **error** | Another controller has taken this database's lease, so two are running against it. Both are scheduling, and they will mint runners against each other and remove each other's workloads. Stop one; the survivor restarts with `--takeover`. | |
| `pool.runner_group_unresolved` | warning | A pool asked for a runner group its target does not offer, or GitHub would not say which groups exist, so its runners registered in Default instead of the isolation boundary the pool requested. | The pool's next runner registers in the group the pool names, which its page and GitHub's runner list both show. |
| `pool.runner_group_public_repositories_blocked` | warning | GitHub reports the pool's runner group as unavailable to public repositories. The runners can register, connect and appear idle, but GitHub will leave matching public-repository jobs queued. Enable **Allow public repositories** for that runner group under the organisation's **Settings > Actions > Runner groups**, or use a repository-scoped installation. | A queued public-repository job is picked up by the pool's next idle runner, and the entry leaves the list. |
| `pool.repository_scale_up_deferred` | warning | The per-repository creation limit held runners back. Expected under a burst; standing means the limit is too tight. | The entry leaves the list once runners for the repository are created without being held back; the pool's page shows no deferred starts. |
| `jobs.unmatched` | warning | Jobs are queued that no enabled pool here will run. Usually their labels match no pool: they may belong to another runner provider, or a pool may be missing a label. The entry says so instead when the cause is the GitHub target rather than the labels (a pool advertising exactly those labels but belonging to another installation, or a repository no installation here covers) because those need the installation changed, not the workflow. | `zoomies jobs list --unmatched` is empty and the entry leaves the list once the labels match a pool or the installation covers the repository. |
| `jobs.label_advice` | info | Jobs whose measured runs call for something other than what their `runs-on` asks for, counted by kind, in one entry however many there are: a job that names a size class too small for what it uses (it can only ever run on hosts that are too small, and is killed when it needs more than they have), a job that names none and needs more than the default class (it is routed there best effort, which is not a promise, because GitHub decides which waiting job a runner takes), and a job that names a class larger than it uses (it occupies a host another job needs). Only a job with at least five measured runs (`AdviceMinRuns`) is advised on, and one with fewer is listed as a `not_enough_data` row with its count rather than counted here; a job an operator has pinned is left out, and there is none while `scheduler.size_routing` is `off`. The advice for each job is on the Jobs page, in `GET /api/v1/label-advice` and in `zoomies jobs advice`, with the figures it rests on and what to write instead. | The Jobs page lists no advice for the job after its next five measured runs, and `zoomies jobs advice` leaves it out. |
| `jobs.scratch_full` | warning | A job failed on a runner whose in-memory folder (its work folder, `/tmp` or the Docker image store, kept in memory by its pool) was at least 90% full at a sample before the job ended. A job that writes past a folder's ceiling fails a step with "no space left on device"; GitHub records an ordinary failure, the runner's own output never carries a step's words, and a compiler deletes its work directory as the link fails, so the agent's samples are the only evidence and the fill itself falls between them, which is why the job's explanation is sure to medium. The entry is per pool, because the folder's size, and whether it is in memory at all, is the pool's setting: give the folder room by raising the pool's memory per runner (the folder is fitted larger), typing a larger size for it, or keeping it on disk (Placement: Auto, or `--tmpfs-tmp=false` for `/tmp`). See [Keeping the work folder in memory](https://zoomies.sh/hosts-and-pools/#keeping-the-work-folder-in-memory). | The next run of the job on the pool completes, its page shows every folder under 90%, and the entry leaves the list after an hour. |
| `jobs.runner_lost` | warning | A job's runner stopped under it, so the failure is the fleet's rather than the workflow's. When most of them share one category, the detail says so and the fix is that category's. | The next run of the job finishes on a runner that outlives it: `zoomies jobs list --ours --since 1h` shows nothing new, and the entry leaves the list. |
| `jobs.oom_killed` | warning | The kernel killed a job's runner, one of its steps (exit 137), or in a Docker-in-Docker pool the sidecar its builds ran in, for its memory limit on a host in the last hour. GitHub records it as an ordinary failure; this is the fleet owning up. The entry names the job, the host and the most the job was measured using. With `scheduler.history_sizing: on` the next run of that job is placed on a host with room for half as much again; otherwise give the pool a larger minimum memory, or raise `memory_mb`. | The next run of the job completes without exit 137: its page shows a peak below its limit, and the entry leaves the list after an hour. |
| `pool.history_unfit` | warning | A job waiting on the pool is known, from its own recent runs, to need more CPU or memory than any host that can run the pool has to give. It is placed as it always was, so it is likely to fail the way it did before, and the jobs beside it are not held back for it. Add a larger host the pool can reach, or split the job. | The job's next run lands on a host with room for it, which its page shows as the host chosen and the class it ran in, and the entry leaves the list. |
| `runners.failed` | warning | Runners are in the failed state with their reasons recorded. | `zoomies runners list --state failed` is empty and the entry leaves the list once the failed runners are gone. |
| `runners.cleanup_failed` | warning | Zoomies could not finish taking a runner away: a container still on its host, or a registration still on GitHub. Different from `runners.failed`, which is a job that did not run; this is something *left behind*. Most often GitHub's own bookkeeping is a few seconds behind the webhook that told Zoomies the job was done, and this clears on the next retry with nothing to do. A container Docker is still removing is waited for rather than raised, and appears here only once the removal has gone ten minutes without finishing; see [Cleanup](https://zoomies.sh/troubleshooting/#what-zoomies-cleans-up-and-what-it-leaves) for the other shapes and what each needs. | `zoomies runners list` no longer shows the runner, the host carries no container for it, GitHub's runner list has no registration, and the entry leaves the list. |
| `runners.not_progressing` | warning | Runners have sat in `provisioning` or `registering` for over half the provision timeout, so the fleet says so while there is still time to look rather than only when it fails them. The entry splits the two shapes, because they are not fixed in the same place: a runner still waiting for a container is a backend or image problem on the host, and one whose container started without registering is the runner process failing to reach GitHub. | The runner reaches `idle` or is failed with its reason: `zoomies runners list --state provisioning` and `--state registering` show nothing older than half the timeout. |
| `capacity_demand.delivery_failed` | warning | An external capacity provisioner did not accept the latest event, after its retries. | |
| `controller.update_available` | info | A newer release of Zoomies has been published than the one this controller was built from. Nothing is wrong and nothing updates itself without being asked: runners, pools and jobs do not depend on the controller's version. It appears only on a controller built from a release tag, and `updates.check_interval: 0` switches both the check and this entry off. Its fix points at Settings → Updates when `updates.mode` is `manual` or `auto` and the update helper is installed, and otherwise says to upgrade the way you installed. | |
| `controller.update_helper_missing` | warning | `updates.mode` is `manual` or `auto` on a controller built from a release, so it may be asked to update itself, but the update helper that would replace its binary is not installed on its host, or its folder or marker is not as the installer leaves it. Nothing else is affected, and neither the mode `off` nor a build from `main`, which cannot be updated this way, raises it. Nor does a controller the helper can never be installed beside: one whose host is not Linux, a native one on a host systemd does not run, a container that runs no runners (it does not mount the shared folder), or a container under a rootless runtime. Settings → Updates says which, as `unsupported`, with the `zoomies upgrade` command that updates it by hand. Installing the helper needs root, so the entry offers no one-click fix: run `sudo zoomies updates helper install` on the controller's host, or set the mode to `off` and upgrade by hand. | Install the helper: `zoomies updates helper status` on the controller's host says it is ready, and the entry leaves at the next pass. |
| `controller.update_failed` | error | The controller's latest update attempt ended as failed or timed out, so it is still running the old release. The entry carries the first line of what the update helper said, cut to 300 bytes and shown as plain text, or the controller's own account when no answer came within 90 minutes. Only the opening of a long reason is shown; the whole of it is on Settings → Updates. It offers no one-click fix. | A failed attempt stays on the list until a new attempt opens, a later one succeeds, or this build reaches the release the attempt asked for, whether through Settings → Updates or by running `zoomies upgrade` on the host; otherwise it stays, because retention keeps the failures on the release the controller last tried or would take now. |
| `controller.development_update_available` | info | The controller runs the moving development channel but its stamped commit is not the current head of `main`. A main CI image publication may still be running, or failed or cancelled before advancing `:dev`; an upgrade can only pull what was successfully published. Let main CI publish, then run `zoomies upgrade` again. | |
| `controller.loop_panicked` | error | A background loop panicked and was restarted, or work that a loop hands to a goroutine of its own (a provider call, creating or removing a runner, a capacity-demand delivery, a scan) panicked and was abandoned; the entry says which. The fleet keeps running, but this is a bug: the stack is in the log, under "controller loop panicked" for a loop and "detached controller work panicked" for the rest, and it is worth reporting. | |
| `controller.problems_partial` | **error** | One of the queries behind this list failed, so the list is incomplete and the entry names which sections are missing from it. It exists because the alternative is worse: this page used to return a 500 for any one failing query, and an operator whose drawer will not load reads that as a fleet with nothing wrong. The controller log carries the query that failed. | |
| `oidc.unavailable` | **error** | Single sign-on is configured but not working: the identity provider could not be reached, or did not answer with an OpenID configuration, or a setting it needs is missing. Password sign-in still works and the sign-in page shows its form while this lasts, but accounts that only ever signed in through single sign-on cannot sign in. When the provider is unreachable the controller tries again by itself, backing off from five seconds to five minutes, and the entry clears once it answers; no restart. Check that this host can reach `oidc.issuer`; a missing setting is named in the entry and needs a restart once corrected. | |
| `backup.failed` | warning | The scheduled backup is failing, and the entry carries the error. Usually the directory `backup.directory` names is missing, not writable by the controller, or out of room for a copy of the database. The controller tries again every fifteen minutes; taking one from the Backups tab shows the same error in the page. | |
| `backup.remote_shadowed` | warning | A destination stored from the Backups page has the same name as one `zoomies.yaml` or the environment describes, and the file has the last word, so nothing is sent to the stored one and its settings are not the ones in use. Rename one of them, or delete the stored destination and keep describing it in the file. | |
| `backup.remote_unreadable` | **error** | A stored destination's sealed secret key or passphrase does not open with this controller's encryption key, so nothing can be sent to it. This is what a database restored onto a host with a different key looks like: put the key this fleet was sealed with back, or open the destination on the Backups page and enter its secret key again. | |
| `backup.remote_failed` | warning | Backups are not reaching one of the destinations under `backup.remotes`, and the entry carries the service's own refusal: a wrong secret, a bucket that is not there, a policy that does not allow writing, or a clock too far from the service's to sign with. The controller tries again every fifteen minutes and the Backups tab tests the remote on demand. The copies on this host are unaffected; they are simply all there is. | |
| `backup.restore_staged` | warning | An administrator has staged a restore from the Backups tab, and it is waiting for the controller to restart. Nothing has changed yet: the database is swapped when the next controller starts, before it opens anything, and the restored fleet comes back fenced. Restart the controller from the tab to apply it, or cancel it there. | |
| `backup.restore_failed` | **error** | The last staged restore did not finish; the entry says why, and which database the controller started on: the one it already had when the restore was refused before anything moved, or the restored one, fenced or not, when it failed after the copy. Put right what the reason names and stage the restore again; the entry clears when it is dismissed from the Backups tab. | |
| `recovery.fenced` | error | This fleet was restored from a backup and is held: the scheduler decides as normal and applies none of it; no runner is created, drained or removed, nothing is reaped from GitHub, and the fallback poller does not sweep. The Overview shows what it *would* do, so you can tell "nothing to do" from "not allowed to". `/readyz` answers 503 while it is on; liveness is unaffected, so a container runtime does not restart it. The fix names the three things a restore does not bring with it, and lifting is `POST /api/v1/recovery/unfence`. | |
| `host.version_behind` | warning | One or more hosts run a different release from the controller. A fleet part-way through an upgrade looks like this and clears itself, which is why it is a warning; one that stays this way has a host somebody has forgotten, running an agent whose odd behaviour has an explanation nobody thinks to look for. The entry names which hosts are behind, which are *ahead* (the direction the policy calls unsupported, where the fix is to upgrade the controller) and which run a build the controller cannot order against its own. Two builds of one tag are the same release and do not appear. | `zoomies hosts list` shows every host on the controller's release and the entry leaves the list. |
| `pool.size_strands_hosts` | info | A pool asks for a fixed CPU and memory on every host, and some of its hosts could hold more runners of it than their slot count allows, so the machine above that slot count goes unused. Not wrong, and often deliberate; it is the state a fixed size drifts into as a fleet acquires unequal machines. Raise those hosts' capacity, or clear the pool's CPU and memory so each runner is given one slot's share of the host it lands on, which fills every slot on every machine whatever size it is. | After the capacity is raised or the size removed, the host's card shows the extra slots in use and the entry leaves the list. |
| `pool.provision_timeout_short` | warning | A pool's own provision timeout lands inside the time its runners may legitimately take to start: the agent's create budget, plus the Docker wait a pool that provides a daemon does before it registers. Runners still coming up are failed and replaced, and the replacement pulls the same image over the link that was slow to begin with. This is `scheduler.provision_timeout_short` asked of one pool, because a pool may override either half for itself. Raise the pool's provision timeout above the two together, or clear it to follow the fleet. | Runners of the pool reach `idle` without being failed for the timeout, and the entry leaves the list after the setting changes. |
| `pool.size_unlimited` | warning | A pool sets no CPU or memory limit (which normally means each of its runners is given one slot's share of the machine it lands on) while `scheduler.default_runner_limits` is off, so that share is charged against the host and never applied. A host's worth of these runners can each take every core at once, which is the shape that stops the Docker daemon answering. Turn `scheduler.default_runner_limits` back on, which is the default, or give each pool a size of its own. | The pool's page shows a size, or `scheduler.default_runner_limits` is on, and the entry leaves the list. |
| `pool.resources_unenforced` | warning | A pool on the `process` backend sets CPU, memory or disk limits, and that backend starts a runner as a plain process with no cgroup, so the limits bind nothing. The scheduler still holds that much room on the host, so the fleet does not oversubscribe; what is missing is the enforcement, and a job that runs away can take the machine with it. Move the pool to `docker` or `podman`, where the same limits become cgroup limits, or clear them. | The entry leaves the list once the pool's limits are removed or it moves to a container backend; its page no longer shows limits it cannot apply. |
| `crypto.key_mismatch` | error | This instance's encryption key does not open the GitHub App credentials in its own database. Every installation fails at once and nothing can authenticate to GitHub, which is what tells this apart from `installation.unhealthy`; that one is fixed on GitHub, this one by putting the right key file back. It is what a restore that brought the database and left the key behind looks like once the instance is running; the startup refusal catches the case where the key file is missing entirely. | |
| `poller.stale` | warning | The fallback poller is enabled and has not finished a sweep for more than two and a half intervals. It is the safety net for a fleet whose webhooks stop arriving, and a net that has stopped sweeping looks exactly like a net with nothing to catch. The controller's log carries the error that ended the sweep. | |
| `poller.paused` | warning | GitHub is rate-limiting one installation, so every background sweep is standing down from it until the moment named. It clears itself. One installation is held at a time -- the quota is per installation -- so the others are still polled, and the entry names which one. | |
| `ai_context.artifact_quota` | error | A repository's AI Context workflow built its context, but GitHub refused to store the file that carries it between the workflow's two jobs, because the account's Actions artifact storage quota is full. Public repositories are not affected; a private one shares its owner's quota with every other workflow that uploads artifacts, so what filled it is usually another workflow's builds. The entry says what the repository's own artifacts hold. GitHub recalculates usage only every six to twelve hours, so Zoomies starts the workflow again every six hours, three times for the commit, and no sooner. | |
| `ai_context.artifact_upload_failed` | warning | The same hand-off between the workflow's jobs failed for a reason other than the quota; GitHub's message is on that step of the run. Zoomies starts the workflow again, half an hour after it ended, twice. | |
| `ai_context.runner_unavailable` | warning | The AI Context run was queued but GitHub never gave its job a hosted runner, so no step ran and the job was cancelled when it timed out. It is GitHub's side and normally passes: Zoomies starts the workflow again half an hour after it ended, up to four times. | |
| `ai_context.startup_failed` | error | GitHub refused to start the AI Context run at all: it found the workflow file invalid, or the account's billing or the organisation's Actions policy does not allow it to run. Zoomies tries again six hours after, twice. | |
| `ai_context.setup_failed` | warning | A step before generation failed: checking out the repository, installing Node, or installing the pinned generator from the npm registry. Almost always an outage at GitHub or npm; Zoomies starts the workflow again half an hour after, twice. | |
| `ai_context.oversized_file` | error | The repository's AI Context workflow was written by an earlier Zoomies release, which stops the whole run when any file is over 1 MiB. Current releases list such a file as omitted and carry on. A new run would fail the same way, so Zoomies does not start one; **Reinstall / repair** moves the workflow to the current generator. | |
| `ai_context.too_much_source` | error | The generator refused to build a snapshot because the repository's eligible source is over a size or file-count limit even with the largest files left out. Zoomies does not start the workflow again; add exclusions in the repository's AI Context settings. | |
| `ai_context.generation_refused` | error | The generator stopped on one of its safety checks: the managed configuration no longer matches what was reviewed, the checkout was not the commit, nothing eligible was left, or the secret scan changed what it would carry. A new run would give the same answer, so Zoomies does not start one; **Reinstall / repair** re-reviews the configuration. | |
| `ai_context.publication_refused` | error | The context was built, but the workflow's last job could not update the `zoomies-ai-context` branch: usually a token that cannot write contents, a ruleset that blocks that branch, files on it that Zoomies does not own, or the trusted branch moving on during the run. Zoomies starts the workflow again once, half an hour after. | |
| `ai_context.delivery_failed` | error | A Zoomies-only upload built its context but did not reach the controller. The controller must be reachable from GitHub's runners over https at its upload address and accept GitHub's identity token for the repository. Zoomies starts the workflow again half an hour after, twice. | |
| `ai_context.timed_out` | warning | An AI Context job started and reached its time limit: fifteen minutes to generate, ten to publish or upload. Zoomies starts the workflow again half an hour after, twice; if it keeps happening, add exclusions for large directories. | |
| `ai_context.run_failed` | warning | The AI Context run failed in a way Zoomies does not recognise; the entry links the run. Zoomies starts the workflow again half an hour after, twice. | |
| `kennel.exposure` | error | A repository this fleet serves has an open, unwaived exposure finding that is an error: code from a stranger's pull request has run on your runners, or a public repository's jobs ran on a pool set up so that such code could reach the host. One entry for the whole fleet, however many repositories it is about: its title is the count and it names none of them. It links to Kennel Club's repositories narrowed to those with an error, which lists every finding with the evidence behind it and is where one is waived with a reason. Dismissing it holds until every such error has cleared. A repository with only warnings raises nothing here: it appears in Kennel Club and nowhere else. Neither does one an administrator has told Kennel Club not to look at: it is not evaluated, so there is no error to raise, and starting again raises it afresh if the error is still there. It is never on the public status page. | Kennel Club's Overview shows no repository with an open exposure error once the pools or the repositories' settings change and the next read lands, and the entry leaves the list. |
| `kennel.unavailable` | warning | Kennel Club has not been able to read an installation's repositories for six hours: GitHub refused the listing, which every App is allowed to make, or the reads keep failing or being held back by the rate limit. What it says about those repositories is as old as that. A permission you chose not to grant is never this: it is shown on the repositories it affects, naming the permission, and is not a problem. Nor is an installation on which every repository has been told not to be tracked: nothing there is read, so the failure is dropped, and it comes back if one is tracked again and the reads still fail. | Kennel Club's Overview shows the installation read within the last refresh interval, and the entry leaves the list after the next successful read. |
| `pool.max_above_room` | warning | The pool's maximum is more runners than its hosts have room for at the size one of its runners is given there; a pool with a minimum counts the one reduced runner a host too small for another standard one still takes, at what it would be given. It is not wrong (the maximum is a backstop rather than a target) but the runners above the room are runners the scheduler will never create, and the jobs that ask for them wait with nothing else on any page saying why. Lower the maximum, ask for less per runner, or give the pool more hosts. A pool whose room is bounded by disk (each runner is charged 10 GB of working room unless the pool sets `resources.disk_gb`) can also set `resources.disk_gb` lower, or free space on its hosts. | The pool's maximum is at or below the room its page shows, and the entry leaves the list. |
| `pool.auto_blocked` | warning | An automatic pool could not be made or kept, and the entry says why and what to change. Either a pool you made already has the name the controller would give the pool for that architecture and size class, or already answers to the class label `zoomies-small`, `zoomies-medium` or `zoomies-large` that pool would carry (the controller never takes over, rewrites or shares a pool of yours, so it makes none and jobs for that class wait for a pool to carry the label) or the pool kept for the installation automatic pools used to belong to still holds the name the new one needs, because pool names are unique across the instance (delete it, which you can because it is no longer kept) or `scheduler.auto_pools` is on and no installation could be chosen, because several are configured and `scheduler.auto_pools_installation` names none. Rename or relabel your pool, delete the old one, or name the installation. | `zoomies auto-pools` lists the pool for that class and the entry leaves the list. |
| `host.auto_pool_skipped` | warning | A host is in none of the pools the controller keeps for each architecture and size class, for a reason you can change: its `size` tag is not small, medium or large, its architecture is neither amd64 nor arm64, or it offers neither the Docker nor the Podman backend an automatic pool uses. A cordoned host and a host that has gone quiet are not reported: both are things a fleet does, and both end by themselves. Fix the tag or the host's runtime, or leave the host to the pools you made. | `zoomies auto-pools` shows the host in a class's pool and the entry leaves the list. |
| `pool.no_eligible_host` | warning | No host can run the pool, and a host's runner profile is why for at least one of them: a host's minimum runner size is above the size the pool states, or a host's standard is below the floor of a pool that takes its size from the host. The entry names each host with the limit that left it out, in the same words the pool's page and the refusal of a host edit use. The pool looks healthy and starts no runner while this holds, so nothing else on any page says why. It is raised only where a profile is involved; a pool no host can run for any other reason keeps the explanations it always had. Change those hosts' minimum or standard runner size, change the pool's size or minimum, or add a host whose profile suits it. | The pool's page lists at least one host that can run it and the entry leaves the list. |
| `pool.profile_default` | info | A pool that takes its size from its host is placed on hosts whose profiles name no standard runner size, so its runners there are the fleet's default size; the same on a small machine and a large one. The entry names the hosts and the default. A pool the controller keeps for a size class is not one of these: its runners are that class's size on a host that names none, which differs with the host because the host's class does. Give each of those hosts a standard runner size on its card, or set `runners.default_cpus` and `runners.default_memory_mb` to the size most of them should run. | The named hosts' profiles show a standard runner size and the entry leaves the list. |
| `pool.host_overcommitted` | warning | A host this pool can land on promises more runner slots than its machine can back at the size this pool typed; a pool sized by its host cannot raise it, because a slot is exactly one of its runners. The slots above what fits read as free capacity everywhere they are counted, and every create for one of them is refused for want of CPU or memory. Adjust the host to the slots its machine can back, or give the pool's runners less. | The host's card shows slots its machine can back at the pool's size, and the entry leaves the list. |
| `pool.elastic_cpu_unsupported` | warning | The pool lends CPU to busy runners, and some of the hosts it can land on run an agent too old to say it can move a live runner's quota. A runner placed there is held at its guaranteed share, exactly as with elastic CPU off, and nothing on the pool says which of its runners that happened to. The dry run names the hosts: upgrade the agent on each (the command is on the host's card) or keep the pool on observe, which asks nothing of the agent, until they are. | After the agents are upgraded, the pool's page lists no host that cannot lend CPU and the entry leaves the list. |
| `pool.elastic_memory_unsupported` | warning or info | The pool has the [memory valve](https://zoomies.sh/elastic-memory/) on, and some of the hosts it can land on run an agent that cannot carry it out: one that predates the valve, or that is not on Linux, where the host's memory cannot be read the way a loan has to be checked. A runner placed there keeps the memory it was created with, exactly as with the valve off, and a pool that is only observing collects nothing from it. A warning for a pool set to lend, which is quietly not doing what it says; a note for one that only observes, which is where a new container pool starts, so that a fleet with one Windows or macOS agent in it is not "degraded" over a thing nobody asked of that agent. The dry run names the hosts: upgrade the agent on a Linux host (the command is on the host's card) or keep the pool off the others with its host selector. | After the agents are upgraded, the pool's page lists no host that cannot carry the valve and the entry leaves the list. |
| `pool.memory_ceiling_reached` | info | A runner of this pool held everything it may be lent, its own share and the pool's [memory ceiling](https://zoomies.sh/elastic-memory/#the-ceiling) together, lowered by the host's where the host names one, and still wanted more. The entry names the hosts. Raise the ceiling (**Memory ceiling** in the pool editor, or `zoomies pools edit <pool> --memory-burst-max`), or give each runner more to start with. It clears fifteen minutes after the last time. | The job's next run completes within the raised ceiling: its page shows a peak below it, and the entry leaves the list. |
| `pool.cache_above_disk` | warning | The pool's cache size limit is above the free disk on the smallest host it can land on. The cache is evicted down to its limit between one runner and the next, so a limit above the free space is not a limit at all: the disk fills first, and a host at or below its disk reserve takes no runner of any pool. | The pool's cache limit is at or below the free disk its page shows for the smallest host, and the entry leaves the list. |
| `pool.dangerous` | warning | The pool was configured to weaken the isolation between a workflow job and the host it runs on; the same sentence the pool page shows for the setting itself, so an operator sees one fact in both places rather than learning it twice. Edit the pool if this was not deliberate. | The pool's page no longer shows the weakening setting and the entry leaves the list; it stays while the setting is deliberate. |
| `pool.docker_client_missing` | error | The pool's `docker_mode` gives its jobs a Docker daemon, and the image its runners boot carries no client to reach it with, so the daemon comes up unused and every job fails at its first Docker step with `Unable to locate executable file: docker`, which names the missing binary and not the reason. A pool that asks for a daemon is moved onto `ghcr.io/eyupio/zoomies-runner-docker` wherever that image is known to exist; a digest names one exact image, and a tag from another build may name a run whose variant was never published, so those are left as you set them. Pin the variant at the same tag or digest, or clear the pool's image so it follows the fleet's default and is moved for you. | A job's first Docker step on the pool succeeds and the entry leaves the list. |
| `pool.cache_shared` | warning | Under an organisation installation, GitHub can hand this pool's runners any repository's job whose `runs-on` matches its labels, Zoomies has no say in which repository that is. A repository-scoped cache is therefore only as private as the pool's labels: give the pool a branded label that only the intended repository's workflows use. A repository-targeted installation registers runners only that repository's jobs can reach, so this never fires there. | The pool's cache scope is `installation`, or the pool is under a repository installation, and the entry leaves the list. |
| `pool.cache_memory_unbounded` | warning | The pool's cache source is under `/dev/shm`, which is the host's memory, and the cache has no size limit. The cache is evicted only down to its limit between one runner and the next, so with none it grows until the host has no memory left, which takes every runner and the agent with it. Set the cache's size limit, keeping it well inside what the smallest host can spare, or move the cache source to a directory on disk. | The pool's cache shows a size limit on its page and the entry leaves the list. |
| `pool.tmpfs_memory_tight` | warning or info | The pool keeps its work folder or `/tmp` in memory, and the room those folders may fill is a large part of the runner's memory limit. A tmpfs is charged to that limit, so the job has what the folders leave. A warning where sizes you typed take more than half the limit; a job that fills them is killed for want of memory; information where folders left to size themselves were fitted down because the limit is small. Either way the fix names the limit that leaves the job the room it has now with the folders on top. It is a proposal and never an edit, because the limit is also what the scheduler charges the host for. A pool whose runners are sized by their host has no limit of its own to judge, so it is judged host by host instead: the limit is one runner's charge there, divided with its Docker daemon by the pool's share, and the hosts where the folders came out smaller than asked for are named. A warning where a folder came out under half of what was asked, because a job that fills it fails with "no space left on device"; the fix is a runner size at which the folders fit, a different daemon share, or turning off the folder that does not. | The pool's page shows its folders' room as a small part of the runner's memory limit, and the entry leaves the list. |
| `pool.tmpfs_unsupported` | warning | The pool keeps folders in memory, and some of the hosts it can land on run an agent too old to say it can mount them. A runner placed there starts with its folders on disk, exactly as with the setting off, and nothing on the pool says which of its runners that happened to. The dry run names the hosts: upgrade the agent on each, or point the pool at other hosts with its host selector until they are. | After the agents are upgraded, the pool's page lists no host that cannot mount the folders and the entry leaves the list. |
| `pool.daemon_share_unsupported` | warning | A Docker-in-Docker pool asks for something other than an even split between the runner and its daemon, and some of the hosts it can land on run an agent too old to say it follows the pool's shares. The host is charged by the share while the agent gives each container half the slot, so the daemon is given less (or more) than the pool says, and nothing on the pool shows which runners that happened to. Upgrade the agent on each host named, or point the pool at other hosts with its host selector until they are; the sidecar-share advice proposes nothing meanwhile. | After the agents are upgraded, the pool's page lists no host that cannot follow the shares and the entry leaves the list. |
| `pool.tmpfs_host_off` | info | The pool keeps folders in memory, and an operator turned in-memory folders off in the runner profile of some of the hosts it can land on. A runner placed there has its folders on disk, as it would with the setting off. It is information and not a warning, because a machine's owner knows how much memory it has and the pool's does not, so the host having the last word is the design; it is said where the pool is saved because nothing on the pool says which of its runners that happened to. The fallback is a tactical fix and is kept visible for that reason. Clear *Fall back to disk on this machine (temporary)* in the host's runner sizes (*Set runner sizes* in its menu, or **Edit** on the **Runner sizes** block under the card's **Details**), or `zoomies hosts edit --tmpfs-off=false`, to use memory there, set the host's folder sizes instead, or point the pool at other hosts with its host selector. | The entry leaves the list once the hosts' profiles allow in-memory folders or the pool stops asking for them. |
| `pool.daemon_share_suggested` | info | A Docker-in-Docker pool divides its slot against its work: over the last six hours, one of its two containers used at least 85% of its own limit for one sample in twenty while the other used at most half of its. Raised only from runners sized by their host (a typed limit goes to both containers in full, so there is nothing to divide), only from at least 60 samples across three runners that span at least half an hour and of which no one runner is more than half, so a pool just restarted or one long job cannot speak for it, and only from samples taken under the division in force now, so an edit clears it. The notice says the time the samples really cover. CPU and memory are judged on their own, so the notice can say more CPU for a sidecar that is building and less memory for one that is not. The proposal, for each resource, is each container's own 95th-percentile use plus a quarter, as a share of the slot, rounded to five and kept between 10 and 90. A proposal is priced on the pool's hosts before it is made, because a thinner half is held to the pool's smallest runner and the slot grows to carry it: a share that would leave the hosts room for fewer runners than they hold now is stepped back toward the current one until it does not, and when no share worth a move of ten points does, the notice says what the figure would have cost and proposes none. It stays for as long as the window shows it and clears itself when the share is changed or the work is. An operator sees a button on the notice that makes the change in one click, leaving everything else about the pool as it was; the same figures are in the problem's `daemon_share` field. Or change the pool's Docker sidecar's CPU and memory shares as the notice says (`--daemon-cpu-share`, `--daemon-memory-share`, or `--daemon-share` when both want the same figure); it applies to runners created afterwards. See [Two containers, two limits](https://zoomies.sh/hosts-and-pools/#keeping-the-work-folder-in-memory). It is not given for a pool the controller keeps, which divides its slot evenly and has no setting for it. Memory is sized from the most each half used, not its 95th percentile, because it is a limit a job is killed at: a runner that used 1 GiB in nearly every sample and 3.5 GiB in a few is not given less than 3.5. A share is only proposed if, turned into the limits each container would really be given on the pool's hosts, it gives the squeezed container more somewhere and leaves the other enough everywhere: where the pool has a smallest runner and the thin half sits at it, every share gives that half the same, so the notice says what sets it (raise the pool's smallest runner) and offers no share to apply. | After the share changes, both containers' samples on the pool's page sit under their limits and the entry leaves the list within six hours. |
| `pool.tmpfs_auto_on_disk` | info | A pool that lets each runner decide where its in-memory folders live has an automatic folder on disk on some hosts, because the runner there has no room for it to be useful: below a floor of 2 GB for the work folder, 2 GB for `/tmp` and 4 GB for the Docker image store (or the size you typed, if that is smaller) a folder fills and fails jobs with "no space left on device" while saving little disk traffic. The setting working as designed, not a fault. It names the hosts and what each runner has, and says what would change it, worked out from the pool's own numbers: per host, the smallest runner size at which the work folder would be in memory (the host's standard runner memory for a pool that takes its size from its hosts, its capacity for one sized by a slot's share) and the slots that leaves, and a daemon share that would put more folders in memory *without losing a runner*, when one does. A share that costs runners is never offered, because raising a thinner half to its minimum grows the slot. The same per-host answer is under the folders in the pool editor's Speed section while the pool is being made. | After the runner is given more room, the pool's page shows the folders in memory on those hosts and the entry leaves the list. |
| `pool.tmpfs_suggested` | info | The pool's jobs have been waiting on disk, and its machines have memory to spare. Raised only when, on one host, all three hold: the pool ran jobs there in the last six hours, the host's I/O wait has stayed above 20% for at least ten minutes (a reading CPU occupancy hides, because it counts that wait as busy), and the host has free memory beyond its own reserve for a work folder's default. It stays for as long as that is true and clears itself when the pool turns the setting on, the disk calms or the memory is spent. Turn on the in-memory work folder for the pool, and raise its memory limit by the proposed amount. See [Keeping the work folder in memory](https://zoomies.sh/hosts-and-pools/#keeping-the-work-folder-in-memory). | After the folders move to memory, the host's I/O wait on its page falls below 10% and the entry leaves the list. |

## Runtime: infrastructure providers

Raised only when a [provider](https://zoomies.sh/providers/) is configured, and silent while
`provider.enabled` is false, except for `provider.renting_off`, which is raised
because it is false, and `provider.machines_behind`, which is about machines
already rented: they are not destroyed while renting is off, so their release is
still worth knowing. The three that name a
machine link to it, because the machine's own page carries the failure in the
words of whatever refused it, Proxmox's task error for a clone, the guest's own
standard error for a bootstrap.

| Code | Severity | What it means | What to do |
| --- | --- | --- | --- |
| `provider.unreachable` | warning, or **error** with a machine mid-operation | The hypervisor could not be reached. This is never evidence about a resource: nothing is created, failed or deleted on the strength of it. | Check the endpoint, the network and the certificate. A provider reached through a private connection has no route to test; its message names the gateway machine, where the checks actually are, and says that both ends keep dialling. Machines already running are unaffected. |
| `provider.credentials_refused` | error | The API token was refused, or is not allowed to do something. Where the provider named a privilege, so does this. | Run the provider's check, which lists every missing privilege and the path it is needed on. |
| `provider.quota_exhausted` | warning, or **error** with jobs queued | The hypervisor refused for want of capacity. New machines stand down for a while; drains and deletes continue. | Free space or capacity, or lower the provider's limit so the fleet stops asking. |
| `provider.machine_failed` | warning | A machine never reached ready, and carries the provider's own words for why. | Read the machine's page, then delete the machine. One that never got a VM costs nothing to delete. For one that did, its VM may still be running and billed: check the provider's console first if you would rather look, and press Delete to destroy it; the delete asks the provider again before it destroys anything. Several can be ticked and deleted together on the Providers page. Three machines failing in a row stand the provider down, so a bad template costs a few machines rather than fifty. |
| `provider.bootstrap_failed` | error | The machine came up and its agent never did. The guest's own standard error is on the machine's page. | Usually the template: a missing guest agent, a missing Zoomies binary, or an `agent.json` left in the image. See [Proxmox VE](https://zoomies.sh/proxmox/#preparing-the-template). Then delete the failed machine, which destroys its VM so it stops being billed. |
| `provider.ownership_unverified` | error | A machine is quarantined: its row and the resource disagree about who owns what. **Nothing will touch it again until a person does.** | Look at both sides. Then either release the row, which forgets the machine without touching the resource, or remove the resource by hand. |
| `provider.orphan_found` | error | A resource wearing this controller's marks has no row behind it. It is never deleted automatically; it may belong to another live fleet. | Review it on the provider's Orphans tab. A restored database and a row pruned early are the two ways this happens. |
| `provider.delete_pending` | warning | A delete was issued and the resource has not been confirmed gone. Something is still costing money. | Check the provider. A delete is only complete when an inspection cannot find the resource; a 200 from the delete call is not that. |
| `provider.unservable` | warning | No pool could ever place a runner on this provider's machines, so anything it rents is money for nothing. | Match the provider's machine shape, labels and platform to a pool, or disable it. |
| `provider.template_unverified` | warning | The provider's settings changed and its prerequisites have not been checked since. | Run the check. It creates nothing. |
| `provider.machines_behind` | warning | The machines this provider rents are not on the controller's release. A machine is cloned from the template this provider builds from, and Zoomies installs nothing of its own inside a rented machine, so the release a machine runs is the release that image carried when it was cloned. A whole provider's machines are behind together for that reason, which is why this is one entry for the provider rather than one per machine. A fleet whose controller is built from main lands every one of them here too, because two builds from main cannot be ordered against each other. Nothing is wrong meanwhile: they place work as normal. | Prepare the template this provider clones from again, from the controller's release, on the Proxmox node: **Providers → your provider → Prepare the template again** gives the command, and running it puts an image built from that release where the old one was, at the same template. The machines running now keep their release and are replaced as they go idle, so the machines built afterwards run the newer one. A template you prepared by hand is left alone by that command, so rebuild its image as [Keeping the runner's agent current](https://zoomies.sh/proxmox/#keeping-the-runners-agent-current) describes. |
| `provider.machine_ca_same_as_cluster` | warning | The certificate for machines is the same one the controller uses to reach the hypervisor. A machine trusts the file it is given for the controller and nothing else, so unless the cluster's authority also signed the controller, its agent refuses the controller, never joins, and logs `certificate signed by unknown authority`. Found by the provider's check. | Clear the certificate for machines if the controller is on a public certificate, or paste the authority that signed the controller's certificate. |
| `provider.machine_disk_small` | warning | The machines this provider builds have less disk than their slots are charged for. Each runner is given 10 GB of working room above the host's 2 GB reserve, so a job on a smaller machine fills the disk part-way through and fails with `no space left on device`. Found by the provider's check. | Raise the machine's disk to the figure the finding names, give the machine fewer slots, or enlarge the template it clones. |
| `provider.renting_off` | info, or **warning** with a pool waiting on it or machines still rented | A provider is configured and enabled, but `provider.enabled` is false, so this controller is not allowed to rent machines. A pool that runs out of room waits for the hosts it has, and machines already rented are not drained, deleted or checked. A provider that is disabled on its own page, or the demo fixture, does not raise it. | Turn on Rent machines in Settings, and set Machines the fleet may rent to the most you are willing to pay for. |
| `provider.provisioning_paused` | info | The kill switch is on. Draining, deleting, recovery and ownership checks all continue; only creation is held. | Nothing, unless you did not mean it. |
| `provider.contract_unsupported` | error | A provider declares a contract version this build does not speak. Machines it already owns stay visible, drainable and deletable; a version mismatch must never strand a running machine. | Upgrade whichever side is behind; the message names both numbers. |

## A provider's preflight

Raised by a provider's own check, which creates nothing. They appear on the
provider's page, in the configuration wizard and in the problems drawer, in the
same shape as every other finding, because a refused credential and a template
that is not a template are *answers*, not failures.

`provider.*` codes come from any provider; `proxmox.*` from the Proxmox one.

| Code | Severity | What it means | What to do |
| --- | --- | --- | --- |
| `provider.preflight_failed` | error | The provider refused its connection check. | Read the message; it carries the provider's own words. |
| `provider.zone_missing` | error | The provider has no zone (node, region) configured, so there is nowhere to put a machine. | Set one. |
| `proxmox.unreachable` | error | The cluster could not be reached at all. | Check the endpoint, the port (8006), the network and the certificate. |
| `proxmox.credentials_refused` | error | The API token was refused outright. | Check the token's user, realm, token id and secret. The form wants them exactly as Proxmox printed them: `user@realm!tokenid=secret`. |
| `proxmox.privilege_missing` | error | The token is valid and not allowed to do something. The finding names the privilege **and** the path it is needed on. | Grant that privilege. [Proxmox VE](https://zoomies.sh/proxmox/#the-api-token) lists every one and why it is needed. |
| `proxmox.insecure_tls` | warning | Certificate verification is off, so the token crosses to whoever answered. | Paste the cluster's CA, `/etc/pve/pve-root-ca.pem`, into the provider's CA field instead. See [Security](https://zoomies.sh/security/). |
| `proxmox.version_unqualified` | warning | The cluster is older than the release this integration was qualified against. It is not refused, but nothing about it has been tested. | Upgrade, or proceed knowing it is unqualified. |
| `proxmox.node_missing` | error | No node is configured, or the configured one is not in the cluster. | Choose one the credential can see; the wizard lists them. |
| `proxmox.node_offline` | warning | The node is configured and not currently online. | Machines cannot be created there until it returns. |
| `proxmox.storage_missing` | error | No storage is configured, or the configured one does not exist on that node. | Choose one the wizard lists. |
| `proxmox.storage_no_images` | error | The storage exists and does not accept disk images, so a clone has nowhere to land. | Choose a storage whose content types include `images`. |
| `proxmox.storage_inactive` | error | The storage exists and is not active. | Bring it up, or choose another. |
| `proxmox.bridge_missing` | error | No network bridge is configured, or the configured one is not on that node. A machine with no network cannot reach this controller to enrol. | Choose a bridge that can reach the controller. |
| `proxmox.template_missing` | error | No template VMID is configured, or nothing exists at it. | Prepare a template as the [runbook](https://zoomies.sh/proxmox/#preparing-the-template) describes and give its VMID. |
| `proxmox.template_not_a_template` | error | A VM exists at that VMID and is not a template. Cloning a running VM is not what this does. | Convert it to a template, or point at the right VMID. |
| `proxmox.template_no_disk` | error | The template has no operating-system disk on `scsi0`, `virtio0`, `sata0` or `ide0`. A machine cloned from it starts, tries to boot from the network and never joins, while being billed. | Attach the cloud image as the first disk from a storage that holds disk images, then convert it to a template again. |
| `proxmox.template_no_agent` | warning | The template does not have the QEMU guest agent enabled. Enrolment reaches the guest through it, so a machine made from this template will boot, cost money and never join. | Install `qemu-guest-agent` in the image and set `agent: enabled=1`. |
| `proxmox.vmid_range` | error | No VMID range is configured, or its bounds are the wrong way round. The range is both a budget and a blast radius: a VM outside it is by construction not ours. | Give a block nothing else allocates from. |
| `proxmox.vmid_range_reserved` | warning | Guests already exist inside the configured range. They are not touched, but the range is meant to be Zoomies' alone. | Move the range, or move those guests. |

| `scheduler.registration_concurrency` | error | Credential request concurrency is outside 1–16. | Use 1 by default; increase only with measured need. |
| `agent.prewarm_timeout` | error | Background preparation budget is outside 1s–15m. | Use 5m by default and restart agents. |
| `agent.prewarm_jitter` | error | Background preparation stagger is outside 0s–5m. | Use 30s by default, or 0s to disable, and restart agents. |

## What the status page says

When `status.mode` is `authenticated` or `public`, the fleet status at
`/status` shows each of the fleet's current problems to people who may have
no account and cannot act on the fleet. They are shown the code and the
sentence below, never the operator's title, detail or fix, because those
name the pool, host or repository a problem is about. Only the fleet's codes
have a sentence: a platform code never reaches the status page at all.

Four groups of the fleet's codes are exceptions: the fleet is shown them, and
they never reach the status page or move its state.

* The `ai_context.*` codes are about whether a repository's AI Context workflow
  can publish, not about whether a job will run. A full artifact quota is an error
  for whoever set AI Context up and says nothing to a developer waiting on a
  runner.
* The `kennel.*` codes are about whether a repository the fleet serves is open to
  a stranger's code. A public page that named which repositories the fleet runs
  code for, and that one of them is exposed, would be a map for the stranger the
  finding is about. (`kennel.unavailable` belongs to the platform, so the rule
  above already keeps it off the page.)
* The three host OS health codes (`host.os_health`, `host.health_stale`,
  `host.reboot_pending`) say whether a machine's settings match a
  recommendation, not whether a job will run, and a stock fleet carries some of
  them for ever.
* The three host update codes (`host.update_failed`, `host.update_helper_stuck`,
  `host.update_unavailable`) say whether a host can be updated from here, not whether a job will run: a host
  whose update failed still runs the release it had and places work on it, and a
  failed update must not turn the page to blocked.

The sentences are held in `publicSentences` in
`internal/controller/status.go`, and `internal/docs` checks this table
against it in both directions, word for word.

| Code | What the status page says |
| --- | --- |
| `controller.problems_partial` | The controller could not check everything, so this status may be incomplete. |
| `host.cordoned_with_work` | A machine is being taken out of service once its current jobs finish. |
| `host.duplicate_agent` | Two machines are claiming to be the same one, so jobs may not be placed on it. |
| `host.image_pull_failed` | A machine cannot download the runner image, so jobs that need it may wait. |
| `host.limits_unenforceable` | A machine cannot hold runners to their size, so jobs may run slower than usual. |
| `host.limits_unverified` | A machine has not yet confirmed it can hold runners to their size. |
| `host.overprovisioned` | A machine is promised to more runners than it can hold at once. |
| `host.resources_unknown` | A machine has not reported how much room it has, so it is used cautiously. |
| `host.runtime_recovering` | A machine's container runtime is recovering, so it is taking no new jobs for now. |
| `host.scratch_unmeasured` | A machine cannot tell how full its in-memory folders get, so a job that fills one fails without saying why. |
| `host.shared_folder_unmounted` | A machine keeps no tool cache, so jobs there download their tools again and start slower. |
| `host.throttled` | A machine is busy enough that it is taking new jobs more slowly. |
| `host.memory_pool_exhausted` | A machine ran out of spare memory to lend to its busiest jobs. |
| `host.unhealthy` | A machine has stopped checking in, so fewer jobs can run at once. |
| `host.version_behind` | A machine is running an older agent than the controller. |
| `installation.unhealthy` | The fleet has lost a permission it needs on GitHub, so jobs from some repositories cannot start. |
| `jobs.label_advice` | Some workflows ask for a machine size their jobs do not need, or need more than they ask for. |
| `jobs.oom_killed` | A job ran out of memory on its machine, so it failed through no fault of its own. |
| `jobs.scratch_full` | A job filled a folder its machine keeps in memory, so it failed through no fault of its own. |
| `jobs.runner_lost` | A runner stopped while running a job, so that job may fail or need a re-run. |
| `jobs.unmatched` | Some jobs ask for runner labels this fleet does not offer, so they will wait until the workflow or the fleet changes. |
| `poller.paused` | The controller has stopped asking GitHub for queued jobs, so a missed notification is not caught. |
| `poller.stale` | The controller has not heard from GitHub recently, so new jobs may be noticed late. |
| `pool.cache_above_disk` | A runner cache is set larger than the disk it lives on. |
| `pool.cache_memory_unbounded` | A runner cache kept in memory has no size limit, so it could use all of a machine's memory. |
| `pool.cache_shared` | A runner cache is shared more widely than usual. |
| `pool.history_unfit` | A job needs more memory or CPU than any machine that can run it has, so it may fail again. |
| `pool.dangerous` | Some runners are set up with more access to their machine than the safe default. |
| `pool.docker_client_missing` | Jobs that use Docker may fail because their runner image lacks the Docker client. |
| `pool.elastic_cpu_unsupported` | Runners cannot borrow spare CPU on some machines, so busy jobs run at their normal size. |
| `pool.elastic_memory_unsupported` | Runners cannot borrow spare memory on some machines, so a job that needs more is stopped at its normal limit. |
| `pool.memory_ceiling_reached` | Some jobs needed more memory than a runner may be lent. |
| `pool.github_rate_limited` | GitHub is limiting how fast the fleet can register runners, so jobs may wait longer. |
| `pool.host_overcommitted` | Runners are sized to more than their machines have, so jobs may run slower. |
| `pool.max_above_room` | The fleet is allowed more runners than its machines have room for. |
| `pool.no_capacity` | No machine has room for some jobs right now, so they wait until one frees up or a machine is added. |
| `pool.auto_blocked` | A pool the fleet should keep for a size of machine could not be made, so jobs for that size wait. |
| `host.auto_pool_skipped` | A machine is in none of the pools the fleet keeps for its size, so it takes no work from them. |
| `pool.no_eligible_host` | A pool has no host that can run it, because the hosts' runner profiles keep it off them. |
| `pool.profile_default` | A pool sized by its host is running at the fleet's default size on a host that names none. |
| `pool.provision_timeout_short` | Runners are given less time to start than they usually need, so some may be retried. |
| `pool.repository_scale_up_deferred` | Starting runners for some repositories is held back until GitHub allows it. |
| `pool.resources_unenforced` | Runner sizes are not enforced on some machines. |
| `pool.runner_group_public_repositories_blocked` | Jobs from public repositories are not allowed on some runners, so they will wait. |
| `pool.runner_group_unresolved` | Some runners cannot be registered in the group they belong to, so jobs for them wait. |
| `pool.runners_failing` | Runners are failing to start, so jobs are waiting longer than usual. |
| `pool.size_strands_hosts` | Runners are sized larger than some machines can hold, so those machines stay idle. |
| `pool.size_unlimited` | Some runners have no size limit, so one job can slow the others on its machine. |
| `pool.tmpfs_memory_tight` | Some runners could run out of memory because the working space they keep in memory takes much of their limit. |
| `pool.daemon_share_suggested` | A pool's two build containers are divided unevenly against the work, so one is short of room while the other idles. |
| `host.work_concentrated` | One machine is being kept busy to the point of slowing down while others sit idle, because new jobs keep going to the biggest one first. |
| `pool.minimum_overcharges` | A pool's smallest runner, multiplied by its thin Docker sidecar, charges every runner far more memory than its jobs have ever used, so machines hold fewer runners while jobs wait. |
| `pool.minimum_held_by_job` | One kind of job in a pool is why every runner there is sized so large, and the rest would fit more runners on each machine in a smaller one. |
| `pool.queue_wait_high` | A pool keeps no runner warm, and the slowest jobs wait minutes for one to be made, even though most start at once. |
| `jobs.workflow_failing` | A workflow job has failed at the same step several times in a row and not passed once, which is the workflow's to fix rather than the fleet's. |
| `jobs.duration_regressed` | A job now typically takes much longer than it did the week before. |
| `host.slots_below_capacity` | A machine could hold more runners than it does, because each runner is sized larger than the machine divides into, while jobs have been waiting for room. |
| `pool.tmpfs_auto_on_disk` | Some runners keep their working files on disk because they are too small for memory to be worth using. |
| `pool.tmpfs_suggested` | Jobs on some runners are slow and their machines have spare memory, so keeping working files in memory could speed them up. |
| `pool.tmpfs_unsupported` | Some machines cannot keep runners' working files in memory, so those runners use disk as before. |
| `pool.daemon_share_unsupported` | Some machines give a Docker-in-Docker runner's daemon half of its slot whatever the pool asks for, until their agent is upgraded. |
| `pool.tmpfs_host_off` | Some machines keep runners' working files on disk by choice, so runners placed there are not sped up. |
| `provider.bootstrap_failed` | A newly rented machine failed to join the fleet. |
| `provider.contract_unsupported` | The fleet cannot rent machines from one of its providers. |
| `provider.credentials_refused` | A machine provider refused the fleet's credentials, so no new machines can be added from it. |
| `provider.delete_pending` | A machine the fleet no longer needs is still being removed. |
| `provider.machine_failed` | A machine the fleet asked for did not arrive, so jobs may wait for room. |
| `provider.orphan_found` | A machine provider has a machine the fleet cannot account for. |
| `provider.ownership_unverified` | The fleet cannot confirm it owns a machine, so it is leaving it alone. |
| `provider.preflight_failed` | A machine provider failed its checks, so no new machines can be added from it. |
| `provider.provisioning_paused` | Adding machines is paused, so jobs wait for the machines the fleet already has. |
| `provider.quota_exhausted` | A machine provider has no more room for the fleet, so jobs wait for the machines it has. |
| `provider.renting_off` | Machines are not being rented, so jobs wait for the machines the fleet already has. |
| `provider.template_unverified` | A machine template has not been checked yet. |
| `provider.machine_ca_same_as_cluster` | Machines may refuse this controller's certificate and never join. |
| `provider.machine_disk_small` | Machines may run out of disk part-way through a job. |
| `provider.machines_behind` | Some machines the fleet rents are running a release other than its controller's. |
| `provider.unreachable` | A machine provider cannot be reached, so no new machines can be added from it. |
| `provider.unservable` | Some jobs need machines no provider can supply. |
| `provider.zone_missing` | A machine provider is missing a location the fleet is set to use. |
| `proxmox.bridge_missing` | A machine provider is missing a network the fleet is set to use. |
| `proxmox.credentials_refused` | A machine provider refused the fleet's credentials, so no new machines can be added from it. |
| `proxmox.insecure_tls` | The fleet talks to a machine provider without checking its certificate. |
| `proxmox.node_missing` | A machine provider is missing a server the fleet is set to use. |
| `proxmox.node_offline` | A server at a machine provider is offline, so fewer machines can be added. |
| `proxmox.privilege_missing` | The fleet lacks a permission at a machine provider, so some machines cannot be added. |
| `proxmox.storage_inactive` | Storage at a machine provider is inactive, so no new machines can be added there. |
| `proxmox.storage_missing` | A machine provider is missing storage the fleet is set to use. |
| `proxmox.storage_no_images` | Storage at a machine provider cannot hold machine images. |
| `proxmox.template_missing` | A machine provider is missing the template new machines are made from. |
| `proxmox.template_no_agent` | The template new machines are made from cannot report their address. |
| `proxmox.template_no_disk` | The template new machines are made from has no disk to start from. |
| `proxmox.template_not_a_template` | The template new machines are made from is set up the wrong way. |
| `proxmox.unreachable` | A machine provider cannot be reached, so no new machines can be added from it. |
| `proxmox.version_unqualified` | A machine provider runs a version this fleet has not been tested with. |
| `proxmox.vmid_range` | A machine provider's numbering range is too small for the machines the fleet may need. |
| `proxmox.vmid_range_reserved` | A machine provider's numbering range overlaps machines the fleet does not own. |
| `recovery.fenced` | The fleet is recovering from a restore and is not starting new runners yet. |
| `runners.cleanup_failed` | Some finished runners could not be cleaned up, which uses room new jobs need. |
| `runners.failed` | Some runners failed, so jobs may wait while they are replaced. |
| `runners.not_progressing` | Some runners are taking far longer to start than they should, so jobs are waiting. |
| `webhook.never_received` | The controller has never heard from GitHub directly, so new jobs are noticed late. |
| `webhook.rejected` | GitHub's notifications are being refused because their signature does not verify, so new jobs are noticed late. |

## Keeping this list honest

Every code above is checked against the source by a test
(`TestEveryProblemCodeIsDocumented`), so a code added to the validator or the
controller without a row here fails the build. The severities and the sentences
are prose and are not checked; if one reads wrong, it is wrong.