# Prometheus metrics for your runner fleet

Canonical page: https://zoomies.sh/metrics/

The controller exposes Prometheus metrics at `/metrics`. They are the same
numbers the Overview draws, without the browser, and they are how you keep a
history of everything that moves minute to minute: Zoomies stores enough of
those to answer "what is happening now" and leaves their past to whatever you
already scrape with. What it does keep is the usage ledger, every finished
runner's session for a year (`retention.runner_sessions`) and a daily roll-up
of runner-hours and cost per pool, host and installation that is never pruned,
so the [usage report](https://zoomies.sh/api-surface/) answers "how many runner-hours did we use
last month, and what did they cost" however short `retention.runners` is. Job
counts, execution time and queue waits still come from job rows and go with
`retention.jobs`.

**The agent exposes nothing.** Everything below is the controller's. Agent-side
work still reaches Prometheus, because the agent reports what it did and the
controller observes the result; the runner-startup histograms are all measured
this way.

## Reaching the endpoint

| Setting | Default | What it does |
| --- | --- | --- |
| `metrics.enabled` | `true` | When false the route is never registered, so the path 404s rather than 403s. |
| `metrics.path` | `/metrics` | Mounted at the root, not under `/api/v1`. A leading slash is added if you leave it off, and a path that collides with another mount is refused at startup. |
| `metrics.public` | `false` | Serves without authentication. Raises the `metrics.public` warning, because anyone who can reach it learns the shape of the fleet. |

Authenticated is the default: a scraper needs a token with the `metrics:read`
scope, which the viewer role already carries.

```sh
zoomies tokens create --name prometheus --role viewer --scope metrics:read
```

A session cookie works too, which is what makes the endpoint readable in a
browser while you are signed in. Sending `Accept: application/openmetrics-text`
gets OpenMetrics rather than the classic text format.

The Go runtime's `go_*` and `process_*` families are exposed alongside these.

## The fleet, right now

These five are read from the database on every scrape rather than kept in
memory, so they are always current and never drift.

| Metric | Type | Labels | What it is |
| --- | --- | --- | --- |
| `zoomies_runners` | gauge | `pool`, `state` | Runners by pool and state. The main shape-of-the-fleet series. |
| `zoomies_jobs_queued` | gauge | `pool` | Jobs waiting for a runner, under the pool that claimed them. This is the backlog worth alerting on. Work an operator removed from the queue is not waiting for anything and is not counted, nor is a job whose workflow run has been cancelled, from the moment GitHub accepts the cancellation rather than when it reports the job over. Paused work is still waiting and is still counted. |
| `zoomies_hosts` | gauge | `state` | Agent hosts, by `healthy`, `unhealthy` or `cordoned`. |
| `zoomies_host_capacity` | gauge | - | Runner slots across healthy, uncordoned hosts: what their operators set, or (on a host given a standard runner size) what its machine holds of that size, up to the capacity set. |
| `zoomies_host_effective_capacity` | gauge | - | The same slots as the hosts' throttles leave them. Equal to the previous while no host is throttled; the gap between the two is the throttle. |
| `zoomies_host_capacity_used` | gauge | - | Slots occupied on healthy, uncordoned hosts; the same hosts the two capacity figures count, so dividing by the effective figure gives utilisation. Runners still finishing on a cordoned or silent host are not in it; `zoomies_runners` counts every live runner. |
| `zoomies_host_cpu_usage_percent` | gauge | `host` | Recent whole-host CPU occupied, including I/O wait. Missing when stale or unmeasured. |
| `zoomies_host_memory_available_bytes` | gauge | `host` | Recent available memory including reclaimable cache. Missing when stale or unmeasured. |
| `zoomies_host_memory_pool_bytes` | gauge | `host` | Memory the host can still [lend to running jobs](https://zoomies.sh/elastic-memory/): what no runner's guarantee needs and the host measured as free, less the floor it keeps for itself. Missing for a host with no runner on a pool that has the memory valve on. |
| `zoomies_host_memory_lent_bytes` | gauge | `host` | Memory the host's runners hold beyond what they were created with. A loan is not taken back, so it falls only as the runners that hold it go. Missing for a host with no runner on a pool that has the memory valve on. |
| `zoomies_host_admission_held` | gauge | `host` | 1 while measured CPU or memory pressure holds new starts; running jobs continue. |
| `zoomies_host_usage_fresh` | gauge | `host` | 1 when usage is less than 90 seconds old; 0 when placement falls back to reservations. |
| `zoomies_host_load_average_1m` | gauge | `host` | Recent whole-host one-minute load average. Missing when stale or unmeasured. Past twice the host's CPUs it is what throttles the host; under one per CPU is calm. |
| `zoomies_host_runtime_recovering` | gauge | `host` | 1 while the host's agent reports its container runtime in a recovery cooldown (new starts held until one recovery attempt) and 0 otherwise. Reported for every host. The drawer's `host.runtime_recovering` says which failure it is and when the attempt is due. |
| `zoomies_host_throttle_level` | gauge | `host` | The rung of the throttle ladder the host is on after sustained pressure, 0 to 3. Reported for every host, throttled or not, so a threshold rule keeps matching when nothing is wrong. |
| `zoomies_host_os_checks` | gauge | `host`, `state` | The checks on a host's latest [OS report](https://zoomies.sh/host-health/) that need attention, by `warning`, `error`, `skipped`, `suggestion` or `accepted`. The first three count only what `zoomies doctor` counts by default (the safe tier, without optional checks) and are the numbers the host page and the `host.os_health` problem use; a pending reboot is not one of the warnings, because `zoomies_host_reboot_pending` says it. `suggestion` is a warning that is only a choice, from another tier or optional. `accepted` is a counted warning an operator has accepted as deliberate: it is no longer in `warning`, so a rule on `warning` keeps meaning unaccepted checks, and the host page and `zoomies doctor --host --verbose` show who accepted it and why, and `zoomies hosts list` shows how many. All five are reported for every host with a report, zeroes included, so a threshold rule keeps matching when nothing is wrong; there is no `ok` state, so the five do not add up to the number of checks. Missing for a host that is offline, has sent no report or sent only a container's partial one. Counts only, never the name of a check. |
| `zoomies_host_reboot_pending` | gauge | `host` | 1 while the host's latest OS report says an update is waiting for a reboot, 0 otherwise. Missing in the same cases as the series above. The drawer's `host.reboot_pending` is an info note, so this is for finding hosts that are overdue for one rather than for paging. |
| `zoomies_host_health_report_age_seconds` | gauge | `host` | How old the host's latest OS report is, by the host's own clock, and never below zero. It is the freshness signal: on a native agent, a host whose collector has stopped is still reported and its age climbs, so a value above 600 is the same ten minutes as `host.health_stale`. On a Docker or Compose installation a stopped `zoomies-host-health.service` makes the series disappear instead, because the container's own partial report takes over; alert on its absence too (see [what it cannot see](https://zoomies.sh/host-health/#what-it-cannot-see)). Missing in the same cases as the series above. |
| `zoomies_host_allocatable_cpus` | gauge | - | CPUs across healthy, uncordoned hosts, less each host's reserve. |
| `zoomies_host_allocatable_memory_bytes` | gauge | - | The same for memory. |
| `zoomies_host_reserved_cpus` | gauge | - | What the live runners have promised away, as of the last scheduling pass. |
| `zoomies_host_reserved_memory_bytes` | gauge | - | The same for memory. Divide by the allocatable pair for "how full are the machines", which is a different question from how full the slots are. |
| `zoomies_job_queue_age_seconds` | gauge | `pool` | How long the oldest job still waiting has been waiting. Zero when the pool has nothing queued, and counted over the same jobs as the depth; an item removed from the queue would otherwise climb here for ever. |
| `zoomies_github_paused` | gauge | `installation` | 1 while that installation is inside its GitHub rate-limit backoff and every background sweep is standing down from it. |
| `zoomies_provider_machines` | gauge | `provider`, `state` | Machines a provider is renting, by state. Every state is reported including the zeroes, so a provider that has stopped buying is visible rather than absent. |
| `zoomies_provider_machines_quarantined` | gauge | - | Machines whose ownership could not be proved. Nothing will move one until a person does, so this is a queue of work rather than a shape, alert on any sustained non-zero value. |
| `zoomies_update_available` | gauge | - | 1 while the release `updates.mode` would take is newer than the build this controller is running, and 0 otherwise. A release still waiting out `updates.soak` counts: this says a newer one exists, and `GET /updates` says when `auto` will take it. It is 0 while `updates.mode` is `off`, until the release list has been read, and for a build that is not from a release, and it is reported at 0 as well as 1 so a rule on how long a release has been waiting has a series to match before there is one. The `controller.update_available` notice is separate: it still follows GitHub's latest release while the mode is `off`. |

**Configured and effective slots are kept apart rather than one replacing the
other.** The effective figure is the utilisation denominator while a throttle
stands (it is what `free` is measured against, on the Hosts page and in the
scheduler) and the configured one is what the operator set, so an alert on
"the fleet shrank" can tell a host somebody resized from a host the controller
stepped down. `zoomies_host_throttle_level` names which.

**Slots and resources answer different questions too.** `zoomies_host_capacity`
counts what the fleet will *take*; the allocatable and reserved pairs say
whether the machines can carry it. A fleet with slots free and no memory left is
the case the slot gauges cannot describe, and it is the one somebody is looking
for when the queue will not drain. Disk is deliberately not in the reserved
pair: free disk already contains what the runners there have written.

**Queue depth and queue age answer different questions.** Ten jobs queued for
four seconds is a fleet working; one job queued for forty minutes is a fleet
that has stopped, and `zoomies_jobs_queued` reads lower for the second. Alert on
the age, and use the depth for capacity planning.

The pause gauge is **per installation**, because the backoff is: a fleet with
two installations, one of them rate-limited, is a fleet half working, and only
the label says which half. It reports 0 for every installation that is not held,
so a threshold rule keeps matching when nothing is wrong.

`zoomies_runners{state}` covers the live states only, `provisioning`,
`registering`, `idle`, `busy`, `draining` and `failed`. There is no `removed`
series, because a removed runner is not a runner.

**These five vanish rather than going to zero if the database cannot be read.**
The scrape logs a warning and returns what it has, so an alert on these should
use `absent()` as well as a threshold; a rule that only checks for a high value
will not fire when the numbers stop arriving altogether.

## What the fleet has done

| Metric | Type | Labels | What it is |
| --- | --- | --- | --- |
| `zoomies_jobs_total` | counter | `pool`, `conclusion` | Jobs seen to completion. The denominator for every other job rate. `conclusion` is GitHub's, or `unknown` when it sent none. |
| `zoomies_jobs_runner_lost_total` | counter | `pool` | Jobs whose runner died before GitHub reported the job over. These are the fleet's failures rather than the workflow's, and any sustained rate is worth waking up for. |
| `zoomies_job_reruns_total` | counter | `pool`, `trigger` | Re-runs Zoomies has asked GitHub for. `trigger` is `operator` for the button on a job and `fleet_fault` for [`scheduler.auto_rerun`](https://zoomies.sh/configuration/#schedulerauto_rerun-and-schedulerauto_rerun_limit). A rising `fleet_fault` rate is the fleet breaking jobs and paying to run them again: read it beside `zoomies_jobs_runner_lost_total` rather than on its own. |
| `zoomies_jobs_sized_total` | counter | `basis`, `class`, `mode` | Jobs put in a size class when they were queued, by how the class was decided (`explicit` for a size label in `runs-on`, `pin`, `history` or `default`) the class, and the [`scheduler.size_routing`](https://zoomies.sh/configuration/) mode. Counted once for each job, by the delivery that classed it, and only for a job a pool of this fleet answers: a job for GitHub's own runners is not classed. Absent while size routing is `off`. |
| `zoomies_jobs_ran_by_class_total` | counter | `class`, `ran_class`, `mode` | Classed jobs by the class they were put in and the class of the host that took them. Where the two differ the job ran somewhere other than where it was classed; for a job that only asked for the base label that is GitHub choosing the runner rather than a fault. Read the share of `class` ≠ `ran_class` under `shadow` before turning routing `on`. |
| `zoomies_jobs_rerouted_total` | counter | `from`, `to` | Queued jobs sent to another size class because their own had no pool, or no room for as long as `scheduler.size_fallback_wait`. |
| `zoomies_job_failures_total` | counter | `pool`, `domain`, `fault` | Job failures split by whose they are. `domain` is `fleet` or `workflow`; `fault` is the category (`out_of_memory`, `host_lost`, `image`, `registration`, `backend`, `backend_busy`, `container_conflict`, `config`, `out_of_disk`, `removed`, `runner_exited`) and is empty for a workflow's own failure. The fleet domain is the same set of jobs as `zoomies_jobs_runner_lost_total`, which is kept so an alert written against it does not disappear on upgrade. |
| `zoomies_runner_start_failures_total` | counter | `pool`, `fault` | Runners that failed before they could ever take a job, by category. These reach no job at all (the job they were meant for stays queued and waits for the next one) so a pool climbing here while its queue never moves is a fleet failing with nothing in the failed-jobs count to show for it. |
| `zoomies_registrations_deferred_total` | counter | `installation` | Runner creations held back because the installation was already at `scheduler.registration_concurrency` credential requests in flight. The scheduler had already chosen a host for each of them, so this counts work deferred rather than work refused: the demand is kept and a later pass takes it. A rising rate alongside a growing queue means runners are appearing slowly for a reason no number of extra hosts will change. |
| `zoomies_scaling_events_total` | counter | `pool`, `direction` | Scheduler decisions that changed a pool's size, `up` or `down`. Flapping shows up here first. |
| `zoomies_webhook_deliveries_total` | counter | `status` | Inbound deliveries by `accepted`, `rejected` or `error`. A rising `rejected` count is a signing-secret mismatch or somebody probing. |
| `zoomies_runner_cleanups_total` | counter | `outcome` | Attempts to take a runner off its host: `succeeded`, or `failed` and recorded on the runner's row. This is the half of a runner's life that goes wrong on the host rather than in the fleet, so it appears in no other series here. |
| `zoomies_reconcile_errors_total` | counter | - | Reconcile passes that failed. A pass that fails observes no duration, so without this a controller deciding nothing looks exactly like one with nothing to decide. |
| `zoomies_agent_polls_shed_total` | counter | - | Task polls answered with a backoff because the controller was holding too many at once. A fleet that keeps working while every host hears about its tasks a little later moves nothing else here, so this is the only place that pressure shows. |
| `zoomies_agent_requests_limited_total` | counter | `limit` | Agent requests refused or cut short by a per-host limit: `rate` (a heartbeat, result or report over the host's budget, answered 429), `poll` (a second task poll while the host already has one held, answered at once and empty) and `runners` (a heartbeat or report carrying more runners than any host runs, answered 413). A well-behaved agent never reaches any of them, so a non-zero rate is one misbehaving host; the controller's log names it. |
| `zoomies_log_relay_dropped_bytes_total` | counter | - | Relayed runner output dropped because one stream went over its byte budget of a megabyte a second. The relay drops rather than waits, so a flood shows here and as gaps in the viewer, never as a stalled agent. |
| `zoomies_github_api_requests_total` | counter | `installation`, `result` | GitHub API calls by outcome: `ok`, `rate_limited`, `forbidden`, `not_found`, `error`. Where rate-limiting and a broken installation become visible. |
| `zoomies_kennel_github_requests_total` | counter | `result` | The part of the line above that was made for Kennel Club, by the same outcomes. Read it against the installation's hourly limit before raising `kennel.api_budget_percent`: the budget is a share of that limit, and this is what it was spent on. |
| `zoomies_kennel_findings_opened_total` | counter | `code` | Findings Kennel Club has raised, by check. One that comes back after a waiver ended is not counted again. |
| `zoomies_kennel_findings_closed_total` | counter | `code` | Findings that stopped being reported, fixed, or gone with the repository's jobs. A check whose findings are mostly closed is earning its place. Waived ones are counted apart. |
| `zoomies_kennel_findings_waived_total` | counter | `code` | Open findings a waiver now covers. When most of one check's findings end up here, the check is wrong for this fleet and the severity, the wording or the check itself should change. |
| `zoomies_kennel_rate_limit_holds_total` | counter | - | Holds that began because a Kennel Club read was refused by GitHub. The target is zero; any is a defect in the budget, and the problems list says which installation is held. |
| `zoomies_provider_operations_total` | counter | `kind`, `outcome` | Provider operations by what was attempted (`create`, `start`, `stop`, `bootstrap`, `delete`) and how it went: `ok`, `ambiguous`, `quota`, `unreachable` or `refused`. `ambiguous` is separated from the failures because it means something different: a create that failed cost nothing, and a create whose answer was lost may already be a machine somebody is paying for. Any sustained rate of it is worth looking at. |
| `zoomies_host_runtime_failures_total` | counter | `kind` | Container-runtime failures agents have reported, each one opening or extending a cooldown on new starts: `unavailable` (the daemon could not be reached) or `timeout` (it did not answer in time). Counted when the controller hears of them, so failures during a controller outage arrive as one step. |
| `zoomies_image_pull_failures_total` | counter | `pool`, `kind` | Runner starts (`kind="start"`) and prewarms (`kind="prewarm"`) that failed because the pool's image could not be made ready on the host. The registry is on the host card and in `host.image_pull_failed` rather than a label, to keep the series bounded. |
| `zoomies_image_prewarms_total` | counter | `pool`, `backend`, `outcome` | Background image preparations by outcome: `prepared`, `cache_hit`, `failed`, or `unknown` for a successful older agent. The hit share says whether shared-image coalescing is saving runtime work. |
| `zoomies_elastic_cpu_decisions_total` | counter | `pool`, `mode`, `outcome` | Elastic CPU plans by pool. `outcome` is `burst`, `base`, `host_busy` (the host was held, throttled, or too busy on CPU, load or memory to lend anything) or `unsupported_agent`; observe mode records the same decisions without changing quotas. |
| `zoomies_elastic_memory_decisions_total` | counter | `pool`, `mode`, `outcome` | What the [memory valve](https://zoomies.sh/elastic-memory/) reported of each runner it watches, once per heartbeat. `outcome` is `healthy`, `raised`, `spilled`, `at_ceiling` (a runner held all it may, and wanted more), `pool_empty` or `host_floor` (the host had nothing it could lend), `unmeasured` (the host's memory could not be read, so nothing is lent), `unsupported` (the container runtime cannot raise a live limit), `failed`, or `unreported` for a runner that was not heard from. Observe mode counts what the valve would have done. |
| `zoomies_elastic_memory_near_limit_total` | counter | `pool`, `mode` | Runners that came within a tenth of a memory limit, counted once per runner. It is the evidence of whether a pool's jobs would use the valve at all: a pool that never moves it has no use for the valve. |
| `zoomies_update_attempts_total` | counter | `kind`, `result` | Update attempts that have ended, by what was updated (`controller`, or `host`) and how: `succeeded`, `failed` (the request could not be written to the update folder, the helper refused it, or the upgrade failed; `GET /updates` carries the sentence) or `timed_out` (no answer and no new version in 90 minutes). Counted as an attempt ends, so one in flight is not here; `GET /updates` names it. An attempt cancelled because its host was deleted is closed with the host and not counted. |

Every `pool` label is the pool's **name**, so a query can join these against
the gauges above on `pool`. Work no pool claims is counted under the literal
`unmatched`; a real pool of that name would merge with it, which is a reason
not to name one that.

Every `host` label is the host's **ID**, as `zoomies hosts list` prints it in
its `id` column, not its name, so a query can join the host series against one
another on `host`. The OS health series are counts and nothing else: no check's
name is ever a label or a value, because a check such as Docker log rotation
says which setting a host has not changed. With `metrics.public` on, these
counts are readable by anyone who can reach the endpoint.

`zoomies_image_prewarm_duration_seconds` is the matching histogram, with the
same `pool`, `backend` and `outcome` labels. It measures the complete agent-side
operation, including fast metadata cache hits; compare `prepared` p95 between
backends and watch `failed` rise before reducing refresh intervals.

## How long things take

| Metric | Type | What it is |
| --- | --- | --- |
| `zoomies_job_queue_wait_seconds` | histogram | Queued to picked up. The number that answers "is the fleet big enough?". |
| `zoomies_job_duration_seconds` | histogram | How long jobs ran once started. Capacity planning. |
| `zoomies_store_write_wait_seconds` | histogram | How long a database write waited for the single writer before it could start. Zoomies has one writer by design, and every heartbeat, webhook, scheduling pass and API write queues for it, so this is where a busy instance slows down first. A p99 climbing into whole seconds means too much is writing at once; past ten, SQLite gives up on the write and GitHub has given up on the webhook it came from. |
| `zoomies_store_write_held_seconds` | histogram | How long each write then held the writer, with every other write waiting behind it. Read beside the wait: a long wait with short holds is a queue, and a long hold is one slow write. A backup holds the writer for its whole copy, so its interval shows here as a spike. |
| `zoomies_reconcile_duration_seconds` | histogram | One reconcile pass, including its GitHub calls. A rising p99 means the control loop is being held up by GitHub rather than by itself. |
| `zoomies_provider_operation_seconds` | histogram | One request to a provider, labelled `kind`. It measures the request, not the clone the request starts: a create that takes four minutes at the hypervisor appears here as the second it took to accept the job. |
| `zoomies_elastic_cpu_target_factor` | histogram | Planned CPU target divided by the runner's guaranteed CPU, labelled by `pool` and policy `mode`. A value of 2 means the planner found room to double the quota. |

## Where a slow start actually goes

Five histograms, all labelled `pool` and `backend`, cutting the path from a
queued job to a running one into stages. The last is the whole path; the four
before it say which stage owns a regression.

| Metric | The stage |
| --- | --- |
| `zoomies_runner_queued_to_create_seconds` | Job queued → runner row created. A proxy for scheduler latency: it starts at GitHub's own queued time, so it includes webhook delivery and any configured scale-up delay, and the runner is attributed to the oldest queued job in the pool rather than the job it will run. |
| `zoomies_runner_create_to_container_started_seconds` | Runner created → container started. Where image pulls show up. |
| `zoomies_runner_startup_queue_seconds` | Host startup admission wait for successful creates. |
| `zoomies_runner_dind_ready_seconds` | Sidecar creation and health readiness after its image is available, for successful creates. |
| `zoomies_runner_container_started_to_registered_seconds` | Container started → registered with GitHub. |
| `zoomies_runner_registered_to_ready_seconds` | Registered → idle or busy. |
| `zoomies_runner_queued_to_job_started_seconds` | The whole path. Put this one on the dashboard. |

Two further histograms use the confirmed lifecycle timestamps, with the same
`pool` and `backend` labels:

| Metric | The interval |
| --- | --- |
| `zoomies_runner_eligible_to_create_task_seconds` | Job eligibility → first create task delivered to the job's actual runner host. Observed when the job is linked to its runner, once per job; retries and later stop/remove tasks do not move its end. Includes capacity waits and configured scaling delays after eligibility. GitHub deployment-review holds are excluded. |
| `zoomies_runner_cleanup_duration_seconds` | Runner finished → both host removal and GitHub registration absence confirmed. Includes the configured retention period; a process exit or successful stop alone does not complete it. Repeated confirmations do not add samples. |

Prewarmed runners, jobs whose eligibility was not observed, older runners with
no first-delivery timestamp, and cleanup with a missing confirmation do not
produce a sample. Missing is not zero. Demo installations are excluded. These
histograms describe observations since the controller started; they are not an
exact historical percentile or a count of every attempted job.

For exact historical intervals, the runner API exposes `create_task_issued_at`,
`host_removed_at`, `registration_deleted_at` and `cleaned_up_at`; join the job's
`runner_id` and `eligible_at`. The runner details show the separate timestamps.
Older cleanup timestamps are retained as `cleanup_estimated_at`, labelled as
estimates, and never used as confirmed completion. Compare scheduling figures
with the scheduler interval and available capacity; compare cleanup duration
with retention before interpreting either as a delay.

## Per-installation report

The histograms above are what the controller has seen since it started. The
per-installation report is the other kind of answer: an operator's own record,
over a month or any window up to a year, of how well each GitHub App
installation has been served, how long its jobs waited on the fleet, how many
the fleet broke, and whether the runners it started were cleaned up. It is
computed exactly from stored timestamps, not from histogram buckets, and it is
the section of the Usage page that opens on one installation, the
`GET /api/v1/installations/{id}/report` route, and, grouped by installation,
the extra columns of `/api/v1/usage.csv` ([API surface](https://zoomies.sh/api-surface/)).

The window is moved back to the UTC midnight it begins in, so every day in it
is whole. A job belongs to the day it was first observed (`Job.QueuedAt`) and
a runner to the day it finished.

### Counts

| Figure | Exactly |
| --- | --- |
| **Observed** | Every job first observed in the window for the installation (`Job.InstallationID`), whatever became of it, including jobs no pool claimed and jobs held for a deployment review. |
| **Eligible** | Observed jobs with `Job.EligibleAt` set: a pool claimed the labels and GitHub was not holding the job for a review, so the fleet could act on it. |
| **Created for** | Eligible jobs that ran on a runner this fleet created, whose first create task (`Runner.CreateTaskIssuedAt`) was issued at or after the job became eligible. That is a runner started while the job was waiting, as opposed to one already there. |
| **Ran here** | Jobs that ran on any runner this fleet created (`Job.RunnerID` set), including one prewarmed or left idle by an earlier job. *Ran here* minus *created for* is the jobs an already-waiting runner took. |
| **Ran elsewhere** | Jobs GitHub gave to a runner this fleet did not create: GitHub reported a runner name that matches no runner here. A GitHub-hosted runner, or another fleet's. |
| **Fleet fault** | Jobs carrying a `fault_kind`; the `fleet` domain of `zoomies_job_failures_total`: the runner died under the job or never worked. A workflow's own failure is not one. |
| **Cleanup pending** | Runners of the installation's pools that finished in the window and whose cleanup has not been confirmed: the host has not confirmed removal, GitHub has not confirmed the registration is gone, or both. A runner pruned in that state stays pending for good. |
| **Cleanup converged** | Runners that finished in the window with `Runner.CleanedUpAt` set, both confirmations seen. |

### Timings

Each is an exact percentile of intervals with both ends recorded. A pair with
either end missing is left out rather than counted as zero, and the response
says how many samples each figure is from. A sample belongs to the window its
interval starts in.

| Figure | From | To |
| --- | --- | --- |
| **Eligible to first create task** | `Job.EligibleAt` | `Runner.CreateTaskIssuedAt` of the runner that ran it, for the jobs counted as *created for*. It includes capacity waits and any configured `scheduler.scale_up_delay` after eligibility. |
| **Create to registered** | `Runner.CreateTaskIssuedAt` | `Runner.RegisteredAt` |
| **Cleanup convergence** | `Runner.FinishedAt` | `Runner.CleanedUpAt`. It includes the configured runner retention before removal. |

The percentile method is **nearest rank**: the p-th percentile of *n* sorted
samples is the sample at 1-based rank ⌈p/100 × n⌉. It always answers with an
interval that was actually observed, never an interpolation between two. Worked
through for the sets the store's tests use:

| Samples (seconds) | p50 | p95 |
| --- | --- | --- |
| 2, 4, 10 (odd, *n* = 3) | rank ⌈1.5⌉ = 2 → **4** | rank ⌈2.85⌉ = 3 → **10** |
| 20, 30, 40, 50 (even, *n* = 4) | rank ⌈2⌉ = 2 → **30**, not the 35 an average would give | rank ⌈3.8⌉ = 4 → **50** |

With fewer than twenty samples the p95 is the largest, which is honest about
how little there is to go on.

### Where the figures come from, and where they stop

Job rows are pruned after `retention.jobs` and runner rows after
`retention.runners`, so a month read from the rows alone would be short. Two
records outlive them:

- **The counts** are rolled up by the prune loop into one row per installation
  per UTC day before the rows go. A day closes once every job observed on it
  has completed and every runner created on it has a session; a job that never
  completes stops holding its day when the jobs prune would delete it, and is
  counted as it stood. Days the roll-up has absorbed are read from it, later
  moments from the rows. Neither jobs nor runner sessions are pruned before the
  day they belong to has been counted.
- **The timings** are read from runner sessions (kept for
  `retention.runner_sessions`, a year by default) and from the rows still
  here. A percentile cannot be summed across days, so the roll-up keeps no
  timings at all rather than an approximation of them.

The response carries `counts_from` and `timings_from`. When either is later
than the window's start, that half of the report does not cover the part of the
window before it (the records were pruned before anything kept them, or
predate the build that started keeping them) and `unavailable` says so in
words. The figures are then for the rest of the window, never an estimate of
the whole. Sessions recorded before the timings columns existed have no create
task or eligibility on them and give no scheduling or registration sample.

## Machines a fleet is renting

Present only when a [provider](https://zoomies.sh/providers/) is configured.

| Metric | Type | Labels | What it is |
| --- | --- | --- | --- |
| `zoomies_provider_machines` | gauge | `provider`, `state` | Machines by provider and lifecycle state, read from the rows at scrape time so it cannot drift across a restart. |
| `zoomies_provider_machines_quarantined` | gauge | - | Machines whose ownership could not be proved. Nothing will act on one until a person does, so this is a queue of work rather than a shape: **alert on it above zero**. |
| `zoomies_provider_operations_total` | counter | `kind`, `outcome` | Provider operations, by what was attempted and how it ended: `ok`, `ambiguous`, `quota`, `unreachable` or `refused`. `ambiguous` is its own value on purpose; a create that failed costs nothing, and a create whose answer was lost may already be a machine somebody is paying for. |
| `zoomies_provider_operation_seconds` | histogram | `kind` | How long one provider request took. It measures the request, not the clone the request starts. |

## Build information

`zoomies_build_info` is a gauge that is always 1, labelled `version` and
`commit`. It is there to be joined against: it tells you which build a series
came from, which matters while a fleet is mid-upgrade.

## A starting point for alerts

Four rules cover most of what goes wrong, and none of them is about a machine
being down:

* `zoomies_jobs_queued` above zero for longer than a job normally waits, which
  is the fleet failing at its one job.
* Any sustained `rate(zoomies_jobs_runner_lost_total[15m])`, because that is
  work being lost rather than failing. The same question with the reason
  attached is
  `sum by (fault) (rate(zoomies_job_failures_total{domain="fleet"}[15m]))`,
  which is the one to put on a dashboard: it says which of your problems this
  is, and the fix differs for every category.
* Any sustained `rate(zoomies_runner_start_failures_total[15m])`, which is the
  failure that shows up nowhere else. A pool that cannot start a container
  fails silently: the jobs stay queued, nothing is marked failed, and the fleet
  reads as busy. Alert on it separately from the line above.
* `zoomies_job_queue_age_seconds` above the longest wait you would accept,
  which is the same question as the first rule asked in the units somebody
  actually complains in.
* Any sustained `rate(zoomies_reconcile_errors_total[15m])`, or
  `zoomies_github_paused` stuck at 1: both are the control loop not running,
  which no gauge about the fleet's shape will show you.
* `zoomies_hosts{state="unhealthy"}` above zero.
* `zoomies_host_throttle_level` above zero for longer than a job takes, which
  is a host with too many slots for its machine rather than a host having a
  bad afternoon.
* `zoomies_host_os_checks{state="error"} > 0` and
  `zoomies_host_health_report_age_seconds > 600` are the two worth a page: an
  error is a check that could not run, so nobody knows what that host is
  doing, and ten minutes without a report is a collector that has stopped:
  the same ten minutes after which the drawer raises `host.health_stale`. On a
  Docker or Compose installation that collector's stopping makes the series
  vanish rather than climb, so a rule there wants
  `absent(zoomies_host_health_report_age_seconds{host="host_example"})` as well.
* `sum(zoomies_host_os_checks{state="warning"})` is a ticket and never a
  page, because a stock host carries some warnings for ever. The useful
  question is whether a host got worse, which is
  `delta(zoomies_host_os_checks{state="warning"}[1d]) > 0`.
* Nothing on `state="suggestion"`. Those are choices the operator has not
  made, and the drawer raises nothing for them either.
* `absent(zoomies_runners)`, which catches the case above where the gauges stop
  being reported at all.

The [problems drawer](https://zoomies.sh/problem-codes/) answers a different question; it says
what is wrong *now*, in sentences, with a fix. Metrics say what has been
happening. An operator wants both.