# Upgrading a Zoomies fleet

Canonical page: https://zoomies.sh/upgrading/

Update your controller and agent hosts from the controller's web UI, or use
the supported local CLI method. Start with **Manual** if you want to review
each update before it runs; **Auto** can update the controller and roll out
its release to hosts after a waiting period.

| You want to update | Where to start | Permission |
| --- | --- | --- |
| The controller | **Settings → Updates** | `platform` role |
| One agent host | **Hosts → Update** on its card | Administrator |
| All eligible hosts behind the controller | **Hosts → Update N hosts** | Administrator |
| A machine locally, including older installations | [CLI or installer](#upgrading-from-the-cli) on that machine | Its service owner, with elevation where needed |

Managed updates need the [update helper](#adding-the-update-helper) on each
machine being updated. A card or page that cannot offer an update explains
why and provides the supported local instructions. Windows hosts use the
[agent installer or a binary replacement](#upgrading-an-agent-host).

Before updating a controller, keep a current [backup](https://zoomies.sh/backup-and-restore/).
Controller first, then its hosts: managed host updates target the release the
controller is running. Running jobs continue in their existing runners; the
services restart, so the UI briefly reconnects and reporting can be delayed.
[Work in flight](#what-happens-to-work-in-flight) covers the backend details.

## Updating from the web UI

### Set up managed updates

1. Open **Settings → Updates** as a user with the `platform` role.
2. Under **Update mode**, choose **Manual**. This enables requested updates
   without enabling automatic restarts. The soak and check interval are on the
   same page; Configuration also exposes these settings. If the environment
   pins a value, the page names the override that must be removed to edit it.
3. If the page says the helper is missing, copy its install command and run it
   on the controller's host. On each supported Linux/systemd agent host you
   want to update from the UI, its owner also installs the helper once:

   ```sh
   sudo zoomies updates helper install
   ```

   Copy the command supplied by the page or host card for a custom
   configuration directory. [Helper setup](#adding-the-update-helper) explains
   supported deployments, its permissions and older installations.

### Update the controller from Settings

1. In **Settings → Updates**, check the installed build and choose **Check for
   updates** to ask for the latest release information. Review the offered
   standard release and its release notes.
2. Choose **Review update to vX.Y.Z…**, where the button names the available release,
   and confirm that version.
3. Leave the page open while the controller restarts. It follows the attempt
   and confirms success once the controller returns on the target release.
   A controller's embedded agent updates with it.

The installed build is labelled Standard, Development or Custom. A development
build shows its installed commit and the latest checked commit on `main`, with
the check time. Matching `main` does not prove that a dev image has finished
publishing. **Review dev upgrade…** opens the confirmation for the latest
published dev build; when the commits match, the button says **Reinstall latest
dev…**. Standard automatic updates continue to follow the configured update mode.

![Settings → Updates: the installed standard build, Check for updates, the available standard release and its notes, and the controller update choices](https://zoomies.sh/screenshots/settings-updates-dark.webp#only-dark){ .zoomies-shot }

### Update hosts from the controller

1. Open **Hosts** as an administrator. A host behind the controller shows its
   current version, the target release and whether its helper can update it.
2. Choose **Update** on one host's card, or **Update N hosts** in the page
   header to start a rollout of eligible hosts behind the controller.
3. Confirm the update. A rollout handles one host at a time and confirms each
   host's target version from its heartbeat before moving on.
4. Follow the rollout on **Hosts**. If a host fails or times out, inspect its
   reason, then use **Resume the rollout** or **Cancel the rollout**. Cancelling
   prevents new requests; an update already handed to a helper finishes.

No routine drain is needed for an update. A host without a usable helper shows
**Update by command** with its local instructions. Where no supported command
can be offered, **Update by hand** links to the manual steps. Windows agents use
the PowerShell installer from their card. Use those instructions rather than
a newer release chosen independently of the controller.
[Host update details](#updating-hosts-and-rollouts)
cover ordering, retries and version confirmation.

A **machine Zoomies rents** is not updated here, and its card says so rather than
offering a command nobody can run. Its agent is the one the provider's template
carried when the machine was cloned, so the way to a newer one is that template:
the provider's page has **Prepare the template again**, and the entry the provider
raises in the problems list names the machines it affects.
[Proxmox VE](https://zoomies.sh/proxmox/#keeping-the-runners-agent-current) describes what that
command does and what happens to the machines already running.

![Two host cards on the Hosts page, both behind the controller's release: one says it can be updated, with its Update button, and the other says to update it by command because its update helper is not installed, with the button drawn but not pressable; under each, folded away, is the command that updates the host by hand](https://zoomies.sh/screenshots/hosts-update-dark.webp#only-dark){ .zoomies-shot }

### Choose manual or automatic updates

| Mode | What it does |
| --- | --- |
| `off` (the default) | Web UI update requests are disabled. Host cards keep their local upgrade instructions; choosing this mode installs no helper. |
| `manual` | Request a controller update on **Settings → Updates**, one host with **Update** on its card, or a fleet rollout with **Update N hosts**. Only hosts whose helper can update them are eligible. Updates start when you request them. |
| `auto` | The controller takes the newest release once it has been public for `updates.soak`, then takes every host behind it to the release it runs, one at a time. The buttons stay, and mean "now, without the wait". |

Only the `platform` role changes the mode, soak and check interval, on
**Settings → Updates** or **Configuration**. Everyone can read the update status;
an administrator updates hosts and manages rollouts. Choosing **Auto** asks for
confirmation because it permits controller restarts without another click.
The default soak is 24 hours. [Auto and the soak](#what-auto-does-and-the-soak)
explain what happens when another release arrives during that wait.

Prefer the terminal for managed updates? [`zoomies updates`](https://zoomies.sh/cli/#zoomies-updates)
requests the same controller and host updates. It is distinct from
`zoomies upgrade`, which upgrades the machine where you run it:

```sh
zoomies updates status
zoomies updates apply          # Request a controller update; platform role.
zoomies updates apply --hosts  # Request a host rollout; administrator.
```

These commands use your [CLI connection](https://zoomies.sh/cli/#talking-to-a-controller) to the
controller and need the same enabled mode and helpers as the UI. `apply`
returns when the request is accepted; follow it with `updates status` to check
the result. At a terminal it asks for confirmation; unattended requests need
`--yes`.

## Upgrading from the CLI

The existing local CLI method remains supported. Run this on the machine
you want to upgrade; it does not update the rest of the fleet:

```sh
sudo zoomies upgrade
```

Use `sudo zoomies upgrade --check` to inspect the deployment without changing
it. `sudo zoomies upgrade --dry-run` goes further: it downloads and verifies
the release it would install, prints the deployment review the real run shows
(the shared folder, mounts, files and settings it would offer), then prints
what the binary half would do (replace the binary and keep the running one as
`zoomies.previous`). It changes nothing, and it stages its download away from
the installed binary, so it needs no write access to `/usr/local/bin`.
Use `--check` when you only want to know whether this deployment is sound;
use `--dry-run` when you want to see the upgrade itself before agreeing to it.
`--version vX.Y.Z` targets a specific release, and `--no-download` applies the
executable already installed. Before it changes anything, an upgrade takes and
verifies a copy of the controller's database through the same backup package
the schedule uses, and prints the copy's name and the command that restores it;
`--no-backup` goes without it for an operator who keeps their own copies or
whose database is elsewhere. A remote agent's card provides the command
pinned to its controller's release.

It checks the existing deployment, downloads and verifies the new binary,
refreshes cached stock runner images, updates the controller or agent service,
and checks the result. Native hosts with both controller and agent services
upgrade both when they share the installed binary. An active native host-health
reporter using that binary is restarted too. Use `--mode controller` or
`--mode agent` to select one service explicitly.

For Compose and single-container deployments, the service image is pulled and
compared with the image the running container uses. If they are identical, the
container is kept running without a restart. An image pulled earlier still
replaces a container using an older build. A stopped container is started even
when its image has not changed.

??? info "Installer and upgrade animations"

    At a colour-capable terminal, a block-letter Zoomies banner lights up from
    blue to cyan, catches a bright sweep, then holds briefly before the stages
    begin. The intro lasts about three seconds and plays once, including when the command
    continues in a downloaded binary. Use `--no-animation` or set
    `ZOOMIES_NO_ANIMATION=1` to skip it. `NO_COLOR`, `TERM=dumb`, `--check`,
    `--non-interactive` and redirected output also skip the animation. Small terminals
    skip it too. True-colour terminals use the brand colours; other terminals use
    256- or 16-colour equivalents.

    The same banner welcomes initial controller and remote-host installs through
    `zoomies init`, including `install.sh --mode agent`, and agent template setup
    with `zoomies agent install`. It runs once when the binary takes over setup,
    with the caption “Ready. Set. Install.”. `install.sh --no-animation` carries
    through to setup, including when setup runs via sudo. Answer-file installs stay
    unanimated.

The terminal shows four short stages: **Binary**, **Deployment**, **Verify**
and **Host health**, followed by the upgrade result. Host health is read-only
and never opens a tuning menu. Run `zoomies doctor` afterwards to review
warnings, or `sudo zoomies tune` to review individual fixes. Upgrade approval,
including `--yes`, does not approve OS tuning.

`zoomies update` and `zoomies deployment update` remain compatibility aliases
for the same complete flow. They now include the binary update. Use
`--no-download` when you deliberately want to apply the installed binary.
A failed download stops the upgrade and gives a retry command; it does not
silently continue with an older executable.

The one-line installer remains available for older installations:

```sh
curl -fsSL https://zoomies.sh/install.sh | sh -s -- --upgrade
```

The first controller start applies schema migrations. Upgrade does not run
setup again or require a new join token. The web UI uses the same upgrade
engine through the helper installed on each machine; a local CLI upgrade
needs no helper. [Managed updates](#updating-from-the-web-ui) cover controller
updates, individual hosts and fleet rollouts.

For a **remote agent**, copy the upgrade command from its card on **Hosts**.
That command targets the controller's published version instead of blindly
installing the newest release. A newer agent tells you to upgrade the
controller first. A local or unpublished controller build has no matching
published command. The `dev` channel moves, so it can contain a build newer
than the controller by the time you run the command.

The upgrade keeps configuration, credentials, host identity, data volumes,
ports and runtime options. It refreshes locally cached stock runner images
under their existing tags, including Docker-enabled and full variants; running runners
keep their current images. It does not retag a pool's pinned or custom image.
A custom **service** image needs an explicit `--image <reference>`.

Container upgrades allow up to twenty minutes for the old service to stop.
This is a ceiling, not an added delay: the agent finishes admitted runner
creation and cleanup, reports the result, then exits. Running CI jobs keep
running in their existing containers; the upgrade does not wait for them to
finish. A cold image pull can therefore make a restart take longer than an
idle upgrade. Remote agents keep sending heartbeats during this wait.

The upgrade command supplies this timeout even for older Compose files;
newly generated Compose files also set `stop_grace_period: 20m` for manual
restarts, and a single-container (`docker`) deployment is created with
`--stop-timeout 1200` so a plain `docker stop` or `docker restart` waits as
long. A single-container deployment created by an older binary keeps Docker's
ten-second default until it is recreated with `zoomies init`; until then, stop it
with `docker stop --time 1200 zoomies`. Existing native systemd units are preserved during upgrades. If
yours has a shorter `TimeoutStopSec`, use `systemctl edit zoomies` (or
`zoomies-agent` on an agent host) and set `[Service]` / `TimeoutStopSec=1200s`
before upgrading. New systemd installations use that budget by default.

When first upgrading from an older binary, its shorter internal shutdown
limit still applies to that one stop. To avoid interrupting a creation on
that transition, cordon the controller's host in **Hosts**, wait for its
provisioning runners to finish starting, then upgrade and uncordon it.
Existing running jobs can continue throughout.

For a custom installation, pass `--config-dir <directory>` and, where needed,
`--installed-binary <path>` (`--prefix <binary-directory>` for `install.sh`). Upgrade uses `deployment.json` for container
installs and the existing systemd or launchd service for native ones. It
refuses an unrecognised deployment instead of guessing at a replacement.

After the restart the upgrade waits for the controller to answer before it
reports success. A controller serves nothing until its migrations finish, and
on a large database that can take minutes, so the upgrade says it is waiting
and repeats the controller's latest log line every thirty seconds. A container
is asked through its own health check, run in it straight away; a native
service through `/healthz` on the listener its settings name. The wait lasts up
to thirty minutes. If the controller stops during it, or is still starting at
the end, the upgrade exits with an error that says which and points at its
logs. A **native** controller that does not answer is then rolled back by
itself, unless the upgrade migrated the database; [rolling
back](#rolling-back-by-hand) has the rule. A container deployment's service is its
image, which the upgrade does not keep the one before, so it is not rolled back
by itself. An agent serves nothing to ask, so an agent's upgrade does not wait.

A failed image pull stops before a restart. If a replacement container cannot
start, the upgrade attempts to restore the previous container or Compose image.
That restores the process, **not a migrated database**; the rollback rules
below still apply. An interrupted upgrade can leave `upgrade.lock` in its
configuration directory: remove it only after checking no upgrade is running.

## What a release adds to the host

A release sometimes needs more of the host than the one you installed: a
folder, a mount, a file the installer wrote and somebody has since removed,
and a Compose file written by an older release, or by hand, can lack a mount
this one would write. Before it pulls or restarts anything, an upgrade works
out what this deployment should have, compares it with what it has, lists what
is missing, and adds it only with your approval.

What the deployment should have depends on what it runs, and that is read from
its **settings, in its database** (the same place the Settings page writes)
layered exactly as the controller layers them at start. The upgrade opens the
database read-only beside the running controller, as a backup does; for a
container deployment it asks the container where its data volume is. So a
controller whose embedded agent you turned off on the Settings page is not
offered a runner host's folder and mounts, and one you turned on is. When the
database cannot be read (Docker Desktop keeps volumes inside its own VM), the
upgrade says so and offers what a host that runs runners needs.

| Missing | Offered to | What the upgrade offers |
| --- | --- | --- |
| The [shared folder](https://zoomies.sh/configuration/#the-shared-folder), or a folder in it | hosts that run runners: an agent, or a controller with its embedded agent on | Create it, owned by the account Zoomies runs as, uid 65532 in the container images, the state directory's owner for a native install. A shared folder owned by someone else is given back to that account. |
| The shared folder's mount | the same, in a container | `/var/lib/zoomies/shared:/var/lib/zoomies/shared` |
| A pool's cache folder | the same, in a container, for each pool whose cache has a size limit and lives in a host folder outside the shared folder | The folder at its own path. The host's daemon mounts it into runners on its own, so the cache works without it, but the agent keeps the limit by measuring the folder, and without the mount it finds nothing to measure and never evicts. A folder the container already sees, through a folder above it bound at its own path, is left alone. |
| The runtime's socket, and the group that owns it | the same, on the Docker or Podman backend | The socket at its own path, and its owning group as `group_add` so the image's account can use it |
| The TLS certificate and key | a controller serving TLS from files | Both, read-only, at their own paths |
| `docker-compose.yml` itself | every Compose deployment | Write it again from `deployment.json` and the deployment's settings. |

Everything missing from a Compose file is added in one edit: the file is
copied to `docker-compose.yml.bak.<time>` first, and edited rather than written
again, so your own services, labels and comments stay; blank lines between them
do not survive the edit. On a `docker` deployment the replacement container is
created with the missing binds and group as well as everything the old one had.

A running host checks the part that matters most for itself: a containerised
agent that runs runners without the shared folder mounted from the host keeps
no tool cache rather than hand runners a folder the host's daemon cannot see,
and says so as
[`host.shared_folder_unmounted`](https://zoomies.sh/problem-codes/). One whose container was
never given the runtime's socket says that, and names the line to add, rather
than telling you to install Docker.

How the approval is given depends on how the upgrade runs:

* **At a terminal**, it asks `Add them now? [Y/n]`. `install.sh --upgrade`
  asks on the terminal even when the script itself arrived through a pipe.
* **With `--yes`** (`zoomies upgrade --yes`, or `install.sh --upgrade --yes`),
  it adds them without asking.
* **Unattended** (`--non-interactive`, or no terminal at all) it adds
  nothing, finishes the upgrade as before, and ends by listing what is missing
  and the command that adds it: `zoomies upgrade --yes`. The shared folder is
  only needed by pools that keep a [tool cache](https://zoomies.sh/configuration/#keeping-a-tool-cache),
  so a fleet without one loses nothing by waiting.

A missing `docker-compose.yml` is the exception: there is nothing to bring
the new image up with, so an unattended upgrade stops before it changes
anything and says so. `--check` lists what the real run will offer and adds
none of it.

`zoomies upgrade --check` checks the existing deployment without modifying it,
and `zoomies upgrade --dry-run` downloads the release it would take and previews
the whole run without changing anything.
`zoomies update` is a compatibility alias and accepts the same flags. Both commands
fetch the newest release (or the rolling `dev` build, for a host running one),
verify its checksum, replace the installed binary and carry on in the new one,
so a host upgraded this way gets the new release's checks and prompts too. They
never downgrade, and a download that cannot be verified is not installed.
`--version v1.4.0` pins a tag and `--no-download` applies the binary already
on disk.

The parts of that worth knowing before you do it are what happens to work in
flight, how far the pieces may drift apart, and the one direction you cannot
go back in.

## How managed updates work

With the mode `off`, controller and host update requests are refused with
`update.mode_off`. A controller fenced for recovery or without its lease
starts no update. Local `zoomies upgrade` remains available independently of
the controller's managed update mode.

A release is offered only once it is complete: a tag of the form `vX.Y.Z`,
neither a prerelease nor a draft, with `checksums.txt` and the binary for the
machine's system attached. GitHub makes a release public before all its files
are uploaded, so the newest release is not always one that can be installed. A
controller built from `main`, or from a checkout of your own, is offered the
newest release as a replacement, by the button and never by `auto`: it is
usually ahead of that release, so the update may take it back, and the
confirmation says so.

### Adding the update helper

On a systemd host that has no update helper yet, an upgrade asks a question of
its own once the deployment's additions above have been reviewed:

```text
Add the update helper? [y/N]
```

The [update helper](https://zoomies.sh/security/#what-the-agent-owns-on-a-host) is what lets
the web UI update the host. The update helper runs `zoomies upgrade` as root for a validated request, so that the web UI can update this host (and on the controller's host, `zoomies updates apply` can too). Only the host's owner installs it, and `sudo zoomies updates helper remove` takes it away.

It is a pair of systemd units, and a request names a release and nothing else.
The account Zoomies runs as can trigger an upgrade by writing a request, which
is why it is a question of its own and never part of the batch above.

* **`--yes` does not answer it.** `zoomies upgrade --yes` still adds the layout
  additions and moves the settings, and still leaves this question at its
  default.
* **At a terminal**, Enter, `n` and the end of the input all mean no. Only `y`
  or `yes` installs it.
* **`--update-helper`** (`zoomies upgrade --update-helper`, or
  `install.sh --upgrade --update-helper`) installs it without asking.
* **Unattended** (`--non-interactive`, or no terminal at all) it adds nothing
  and prints `Add later: sudo zoomies updates helper install`, with
  `--config-dir` appended when the upgrade ran against a configuration
  directory other than the default.
* **Once the helper is installed** the upgrade it runs for the web UI adds
  nothing and prints nothing about the helper.

The question comes after the deployment's additions are reviewed and before
anything is pulled or restarted. It is not asked, and nothing about the helper
is said, on a host without systemd, on macOS (launchd), for a private
provider's own service, for a service that runs as root (which can upgrade
itself and needs no helper), or once the helper is installed. When a service
cannot be resolved for a reason you can put right (an unreadable unit, an
account that no longer exists, a unit that names another `--config` directory),
an upgrade at a terminal says so in one line.

A container that does not yet mount the shared folder gets the mount from the
restart in the same upgrade, and the helper cannot be installed before then.
So in that run the question is not asked: the upgrade says that the helper
needs the shared folder mounted, which the restart adds, and to run
`sudo zoomies updates helper install` afterwards.

If the install is refused (the binary sits under a folder a group can write, a
path is under `/home`, Docker remaps user namespaces) the upgrade does not
fail: it prints the helper's own sentence, says the helper was not added, and
finishes. Put the refusal right and run `sudo zoomies updates helper install`.
`--check` asks nothing and adds nothing.

Where the helper can never be installed, the web UI does not suggest it. A
controller on macOS or Windows, on a Linux host that systemd does not run, in a
container that runs no runners (and so does not mount the shared folder), or in
a container under a rootless runtime shows its helper as **unsupported** on
Settings → Updates, with the reason and `sudo zoomies upgrade` to copy instead
of the install command, and raises no `controller.update_helper_missing`. On
Windows, which has no `sudo` and no service `zoomies upgrade` can update, there
is no command to copy, and the reason says how to replace the binary by hand. A host
behind the controller for one of the same reasons (its agent says which on its
heartbeat; an agent too old to say is known only by an operating system other
than Linux) shows **Update by command** on its card with the reason and no
button, and the upgrade command beneath it is the way. A Windows agent's card
provides the PowerShell installer pinned to the controller's published release.
Where a command cannot be offered, the card gives version guidance and links
to [upgrading an agent host](#upgrading-an-agent-host).

The same question is asked once more, in the same words, at the end of a fresh
`zoomies init` and of `zoomies agent join`, once the service is installed (for a
container deployment, once it is up and has its shared folder). It follows the
same rules:

* **It defaults to no, and `--yes` does not answer it.** `zoomies init --yes`
  and `zoomies agent join --yes` leave it at its default; on `agent join`,
  `--yes` means only "replace the credentials this host already has".
* **`--update-helper`** (on `zoomies init` or `zoomies agent join`) adds the
  helper without asking. In an [answer file](https://zoomies.sh/quickstart/#unattended-installs)
  the key is `update_helper: true`.
* **Unattended** it adds nothing and prints
  `Add later: sudo zoomies updates helper install`.
* **A controller-only container** is not asked, since it has no shared folder
  for the helper to watch.
* **If the install is refused**, the install or join still finishes: it prints
  the helper's own sentence and the command to retry.

`zoomies init` on a host where Zoomies is already installed only upgrades it in
place and does not ask; `zoomies upgrade` is where the question is asked then.

### What the helper does with a request

The service, which runs as an account without root, writes `request.json` into
the *update folder*: `update` under the state directory on a native install, or
under the shared folder in a container. The request names an attempt and a
release tag, and nothing else: no address, path, command or flag. Its arrival
starts `zoomies-update.service` as root, which treats everything in the folder
as hostile. It refuses a request that is not a plain file owned by the service's
account, a tag of any other shape, an attempt it has already seen, a third try
at one release, or a second request inside ten minutes, and it does not start
while `upgrade.lock` says an upgrade is already running. A release already
installed, or an older one, is answered as done and nothing runs. The `dev`
channel, which names whatever `main` built last, has no count of tries and is
held to the ten minutes alone, because two updates through it are two updates
and not two failures; a host that follows a controller built from `main` is
taken through it each time that controller moves.

Otherwise it runs the upgrade you would run, unattended, as
`zoomies upgrade --version <tag> --non-interactive`, and never with `--yes`.
That downloads the release's binary, checks it against the release's
`checksums.txt` and asks the downloaded binary to check this deployment before
anything is replaced, so a release that needs something the deployment lacks
stops with the binary and the service as they were. A required addition stops
it, with the command that adds it; an optional one is skipped and named. Then
the binary is replaced, the one it replaced is kept beside it as
`zoomies.previous`, and the service restarts. The helper writes `result.json`
saying how it went, which the controller reads, and
`sudo zoomies updates helper status` shows it with the end of the log.

### Updating the controller

**Update to** the release, on Settings → Updates, or `zoomies updates apply`,
asks the helper beside the controller to take it to the newest release that can
be installed (with `--version`, to that one). The controller records the
attempt, writes the request, and restarts as the new release. The page follows
the restart and says the update worked only once the controller does: the new
process closes the attempt when it finds the helper's result, or finds itself
running the release. A
controller that runs its agent inside itself updates that agent with it.

If the new controller does not come back, nothing in the web UI can say so. On
the host, `sudo zoomies updates helper status` and the journal of
`zoomies-update.service` can, and [rolling back](#rolling-back-by-hand) says
what to do.

#### When the helper stops reading requests

systemd stops the helper's path unit when its service starts too often, and a
host whose `zoomies-update.path` is `disabled` or `inactive (dead)` reads no
request at all. Before this was handled, a stale `request.json` then made the
agent refuse every later update with "an update request is already waiting for
the helper", and the host stayed on its old build with only a bare "update
failed" on its card.

Now the agent looks first: how old `request.json` is, whether `upgrade.lock`
is there, and what `systemctl` says of the two units. If the helper is running
an update it touches nothing and the attempt says so. If the helper is stopped
it asks root, through the update folder, to repair it. The agent cannot do that
itself, and does not try. The repair clears the units' failed state, rewrites
and starts `zoomies-update.path`, and only then removes a `request.json` or
`upgrade.lock` that is older than 15 minutes with no helper and no upgrade
running. Each host is repaired at most three times a day and ten minutes apart.

When the repair works the update goes ahead and the repair is in the audit as
`update.helper_repair`. When it does not, or cannot run, the attempt fails with
a sentence that names the cause, the host card says **Helper stopped**, and the
problem `host.update_helper_stuck` carries the attempt, how long the request
waited and what systemd says of the units. The fix by hand is
`sudo zoomies updates helper install`, which does the same. A host whose helper
was installed before the repair existed has no repair units and needs that
command once.

### Updating hosts, and rollouts

A host is updated by its own agent, never by the controller reaching into it:
the controller hands the agent an `update_agent` task on the agent's own
long-poll, the agent writes the request into its host's update folder, and the
helper there does the rest. The agent offers this only while the helper's
marker is in an update folder it can write to, so a host without the helper is
never asked, and its card says why and keeps the command. A host is always taken
to the release its **controller** runs, never to the newest release, so an agent
is never ahead of its controller ([version skew](#version-skew)). An update has
worked when the host's heartbeat reports that release, not when the task is
answered. The agent repeats the helper's answer on its heartbeats until the
controller has recorded it, so an answer that arrives while the controller is
fenced, or cannot write its database, is recorded on the first beat after. A
fenced controller times nothing out, and once the fence lifts a host's 90-minute
time-out waits one heartbeat interval (and a few seconds more), so an update that
passed its 90 minutes under the fence still ends with the helper's reason rather
than a time-out.

**Update** on a host's card updates that host. **Update N hosts** on the Hosts
page, or `zoomies updates apply --hosts`, starts a *rollout* of every host behind
the controller (`--host` names some): one host at a time, the one running the
fewest jobs first, then by name. No host is cordoned or drained for it. The first
update in a rollout that fails or times out **halts** it, and a halted rollout
starts nothing until an administrator resumes or cancels it, on the Hosts page or
with `zoomies updates resume` or `zoomies updates cancel`. Resuming moves on to
the next host while the one that failed waits out its retry. Cancelling starts
nothing more; an update a helper has already been handed finishes by itself and
is recorded. While any update is open, neither a rollout nor `auto` starts
another, so automatic updating never updates the controller under a host or a
host under the controller. A person pressing a button is not held back the same
way: only a second update of the same machine is refused.

### What auto does, and the soak

In `auto` the controller updates itself first, once the newest complete release
has been public for `updates.soak` (24 hours by default), counted from when GitHub
published it. A newer release restarts the wait, so a release replaced within the
soak is never installed: `v1.3.2` published four hours after `v1.3.1` means
`v1.3.1` is passed over and `v1.3.2` is taken a day after its own publication.
The cost is that a project publishing faster than the soak would never be taken
by `auto`, and Settings → Updates says so. Then it starts a rollout of the hosts
behind the release it now runs. A controller that is already behind a release
older than the soak when you switch to `auto`, with its helper installed, is
updated on the next pass, and the hosts are rolled after it. The soak is `auto`'s alone: a person pressing a
button has decided.

Auto acts only through helpers that are there. While a release is due and the
helper beside the controller is not installed, it waits, and updates no host
either, because no host may go ahead of its controller. Where that helper can
never be installed (see above), the controller is left to a person and hosts
follow the release it runs. A rollout cancelled by hand is not started again for
the same release, however short `retention.update_attempts` is: a rollout a
person cancelled is kept when older history is pruned, for as long as no later
rollout to that release has ended. Switching to `manual` cancels a rollout `auto` started, and
switching to `off` cancels any open rollout; an update a helper is already
running finishes either way, and is recorded.

### What happens to running jobs

Nothing. An update restarts a service, and [a restart does not touch a running
job](#what-happens-to-work-in-flight): the job runs in its runner, the new agent
adopts every runner it finds, and the controller's restart only delays the
reporting. A rollout therefore does not wait for a host to be idle, and the
confirmation says so. A drill updates an agent while a job runs on its host and
holds the job to finishing.

### When an update fails

An attempt that has not finished in 90 minutes is recorded as timed out (the
engine's own worst case is fifty minutes, so a shorter limit would fail an
update that was still working). A failure or a time-out leaves the machine on the
release it had, raises `controller.update_failed` or `host.update_failed` (see
[Problem codes](https://zoomies.sh/problem-codes/)), with the helper's own sentence for the
`platform` role on Settings → Updates or the host's card, and in a rollout halts it. After a failure Zoomies waits 30
minutes before it tries that machine again by itself, and after two failures of
one release it leaves that release on that machine to a person. Those failures
are kept while that release is still the one the machine would be taken to (for
the controller, also while it is the release it last tried, which covers the mode
`off`), so `retention.update_attempts` never counts them back down. The helper keeps its
own count as well, in `/var/lib/zoomies-update`, where the service cannot reset
it.

### Rolling back by hand

A **native controller** that does not answer after an upgrade is rolled back by
itself: if the upgrade ran no migration, the previous release is put back over
the installed binary, the service restarts on it, and the upgrade reports that
it failed and the previous release is running again. A migration is the one
thing that stops this, because migrations are one-way ([there is no
downgrade](#there-is-no-downgrade)): a release that migrated the database is
left in place, and the upgrade names the pre-migration copy to restore. A
container deployment's service is its image, which an upgrade does not keep the
one before, so it is not rolled back by itself. What an update leaves makes the
rest possible by hand:

* **The binary it replaced**, kept beside the installed one as
  `zoomies.previous` (`/usr/local/bin/zoomies.previous` on a default install).
  Each update replaces it, and `sudo zoomies updates helper remove` leaves it;
  only `zoomies uninstall` removes it.
* **The database as it was**, which the new controller copied to
  `pre-migration/zoomies-<timestamp>/` beside the database before it applied a
  migration, if the release had any.

On a native controller, stop the service, put `zoomies.previous` back in place of
`zoomies`, and start it. If the older binary refuses to start because the
database has migrations it does not have, stop it again and put the pre-migration
copy back with [`zoomies restore`](https://zoomies.sh/backup-and-restore/#upgrades-copy-the-database-first)
first. That has two costs: the restore fences the fleet and ends every session,
and anything written since the update is lost, the update's own attempt rows
among it. A host's agent has nothing to migrate, so the binary is all it needs. Turn
the mode to `off` or `manual` before you start, or `auto` will take the release
again once it is due. A container deployment's service is its image, so there
roll back by running the previous release's `vX.Y.Z` image, under the same rule
about the database.

### The first update is by hand

A machine that runs a release from before updating from the web UI cannot be
updated from it: its agent does not know the task, and its binary has no helper
to install. Upgrade each one by hand once, controller first, with
`sudo zoomies upgrade`. Answer yes when it asks to add the update helper (or pass
`--update-helper`), and from then on the web UI can update it. A service that
runs as root can upgrade itself and needs no helper, but the web UI cannot
update it either, and Settings → Updates may still offer
`sudo zoomies updates helper install`, which then refuses.

## Settings that were in `.env`

A controller's settings live in its database, where the Settings page changes
them. A container deployment installed by an older release carries them in its
`.env` instead (the external URL, bind address, TLS, trusted proxies, the
embedded agent's backend, socket and capacity, log settings) and a `ZOOMIES_*`
variable wins over the database, so each one shows on the Settings page as
locked, and a change made there never takes effect.

An upgrade finds them (in the running container's own environment, which is
what the controller reads, whether it came from `.env`, from a literal in the
Compose file or from a `docker run` long ago) lists them with the setting
each one sets, and with the same approval as above moves them:

1. It pulls the new image and stops the controller, which holds the database's
   lock.
2. A one-off container of the new image, on the deployment's own volume, runs
   `zoomies config import-env` and stores every value in the database in one
   write. The values reach it on standard input, never as arguments. A value
   the controller could not run with stores nothing at all.
3. Each line in `.env` is commented out, with a line above it naming the
   setting it moved to; a credential's value is not kept in the comment. On a
   Compose deployment the variables leave the service's `environment:` too,
   and anywhere else the file used them (an older file mounted the
   certificate as `${ZOOMIES_TLS_CERT_FILE}`) gets the value written in. The
   Compose file is copied to `docker-compose.yml.bak.<time>` first. On a
   `docker` deployment the replacement container is created without them.
4. The controller comes back up on the same values, from the database.

If the new image cannot store them, nothing is edited and the upgrade goes on
with the settings where they were. If the recreate fails, `.env` and the
Compose file are put back as they were before the move, because the image
being rolled back to may predate reading them from the database.

What stays in `.env` is what is needed before the database can be opened (the
encryption key, the database path, the state directory) and what Compose or
`docker run` reads itself: the image, the published address and port, the
docker group. An agent keeps its whole environment: it has no database.

`docker compose pull && docker compose up -d` upgrades the image and moves
nothing. For a Compose file you wrote yourself, such as the repository's
[`docker-compose.yml`](https://zoomies.sh/compose/), stop the controller, run
`docker compose run --rm --no-deps -T zoomies config import-env < .env`, take
the lines it stored out of `.env` and the file's `environment:`, and bring it
up again.

## Host usage during a rolling upgrade

Resource-aware allocation adds migration `0028_host_usage.sql` and an optional
heartbeat field. Upgrade the controller first, then the agents, to make their
new measurements available for placement. Older agents can keep running: a
host with no fresh usage reading retains configured capacity and reservation
checks. Local Linux agents begin reporting memory on their first sample and
CPU after a second sample. Unsupported or remote runtime measurements are
shown as unavailable, never as zero usage.

The [pressure rules](https://zoomies.sh/hosts-and-pools/#current-usage-and-automatic-holds) affect
new runner starts and do not clear operator cordons. No new configuration is
required. This remains a schema migration, so the backup and rollback rules
below apply.

## Default allocations and throttling during a rolling upgrade

Migration `0029_host_throttle_and_runner_allocation.sql` adds the throttle
column to hosts and the allocation columns to runners, and the release that
carries it changes what a runner is given. Both halves are **on by upgrade**:
`scheduler.default_runner_limits` and `scheduler.host_throttling` default to
true, and each is a warning when turned off. What that means for a fleet, in
the order it will be noticed:

* **A pool that sets no `cpus` or `memory_mb` no longer gets unlimited
  containers.** Its runners are created with one slot's share of their host's
  allocatable CPU and memory as a real cgroup limit; the same share the
  scheduler was already charging them. A job that used to have the whole
  machine to itself on a quiet host now has its share of it, and a job that
  needed more memory than its share is killed for exceeding a limit nobody
  typed. The runner's message says the limit was the host's default share and
  names the two ways out: set `memory_mb` on the pool, or lower the host's
  capacity so each runner's share is larger. `host.overprovisioned` says
  before any job does when a host's capacity gives each runner less than a
  core or under 2 GB. [Default
  allocations](https://zoomies.sh/hosts-and-pools/#default-allocations) has the rules.
* **The host's CPU reserve now has a floor** of half a core, or a twentieth of
  the machine on a large one, held back for the daemon. A fleet whose explicit
  pool CPU limits summed to exactly the host's CPUs (four pools of 2 CPU on
  an 8-CPU box, say) takes **one runner fewer per host** than it did, because
  the last one no longer fits; pools with no limits simply get a slightly
  smaller share. A host's `allocatable_cpus` in the API and its committed CPU
  bar on the card show the figure the floor leaves. An operator's own
  `reserve_cpus` replaces the floor where it is larger.
* **Runners created before the upgrade keep whatever limit their pool set**,
  none where it set none, but carry no recorded allocation and an empty
  `allocation_source`: the allocation is written when a runner is made, and
  nothing is applied to a live container retroactively. Two things follow
  while such a runner is on a host. It is counted among the host's
  `unlimited_runners` whatever its pool's limit, so a sustained CPU hold there
  can still throttle the host; and the throttle cannot slow it, because the
  agent lowers only a quota whose allocation was recorded with the container,
  so its job runs at full speed and only the host's smaller effective capacity
  applies. Both end when its pool's next runner replaces it, which for an
  ephemeral runner is after one job.
* **Upgrade the agents before expecting either half.** The controller gives a
  default only where the host's daemon has said it can apply the limit, and an
  agent from before the probe has not said, so its hosts get no defaults and
  `host.limits_unverified` names them until the agent is upgraded. Throttling
  needs the agent's measurements (including the load average, which older
  agents do not send) and needs the agent to understand the throttle
  directive in the heartbeat response; an older agent ignores it, its running
  jobs are never slowed, and only the smaller effective capacity applies.
  Everything else about an older agent holds as before: a host with no fresh
  usage is placed by its configured capacity, and a throttle whose host stops
  measuring is lifted after ten minutes rather than left standing.

To turn either half off: `scheduler.default_runner_limits: false`
(`ZOOMIES_DEFAULT_RUNNER_LIMITS=false`) restores unlimited containers for pools
that set no limits, and `scheduler.host_throttling: false`
(`ZOOMIES_HOST_THROTTLING=false`) stops the controller stepping hosts down and
lifts any throttle already standing. Each is warned about at startup and in the
problems drawer for as long as it stands, because both are the thing that keeps
a host's Docker daemon answering. The CPU floor has no switch; a fleet that
wants every core placed has a smaller reserve than the daemon needs.

## A docker-in-docker slot is one runner again

A `dind` pool runs two containers per runner: the runner, and the daemon its
builds run inside. A pool that **types** its own CPU and memory has both given
those figures (the build would gain nothing from a limit on the container that
is not building) so the host is charged twice, as it has been since the
release that started charging for the pair at all.

What changes here is the pool that leaves its size to the host. Its runner and
its daemon now **split one slot** between them, and the host is charged one. So
a host set to eight slots carries eight runners of such a pool, the same as any
other, where the last release made it four, and an operator who followed the
`pool.host_overcommitted` advice to "adjust the host to the slots its machine
can back" was walked down to fewer slots each time they took it, arriving at
one slot, which held nothing: one slot is the whole machine's share, doubled is
the whole machine again, and a machine always measures a little less free than
it is allocatable.

For such a pool that means **twice as many runners per host** as the last
release, each with half the machine it had: the pair divides the slot rather
than taking two. Nothing changes for a pool that is not `dind`, nothing changes
for one that typed its own figures, and no job is failed by it. A slot too
small to give both halves what a runner needs (under half a core, or under a
gigabyte, after the reserve) is refused with a sentence naming the capacity
that divides it, rather than divided into a runner and a daemon with no limit.
`host.overprovisioned` now counts a slot as a pair only for pools that typed
their limits.

## Size classes and automatic pools arrive off

[Size routing and automatic pools](https://zoomies.sh/auto-pools/) are two switches,
`scheduler.size_routing` and `scheduler.auto_pools`, and both are `off` after an
upgrade. Nothing is classed, no pool is made, no pool is changed, and every column
the migrations add holds the value an existing row already had, so no host, pool or
job is different until somebody sets one. No agent needs upgrading and the protocol
version is unchanged: an agent that sends no CPU throttling counters only means its
jobs are classed on memory alone.

There is one change that is not behind a switch, and it is a repair. A host that
joins again (a rebuilt machine, an agent restarted with its old credentials) now
keeps the labels an operator added since it first joined, and the size class it
held. Before, the join built the row from the token and the agent alone and quietly
dropped every edit, which the comment on that code said it did not. The join
token's labels still win, then what the agent declares now, then what was stored: a
label the agent declares is the agent's, so an operator's edit of it is put back,
and one the agent's configuration dropped stays until `zoomies hosts edit --untag`
removes it.

Two things in the UI are not behind a switch, because they cost nothing while the
feature is off. A host's card (under **Details**) and its dialog call the labels on
it *tags*, and list beside them the ones the controller works out from the
machine, `os` and `arch`;
what a pool's host selector matches is unchanged. And the Overview's feed gains an
**Automatic changes** kind, on by default and empty until a switch is set, which
**Settings → Events** turns off.

## Kennel Club arrives off

[Kennel Club](https://zoomies.sh/kennel-club/) is one switch, `kennel.enabled`, and it is `off`
after an upgrade. Nothing is read from GitHub, nothing is evaluated, and no
problem is raised, so no repository is different until somebody sets it. AI
Context, which now sits under the same **Kennel Club** heading in the UI, carries on
exactly as it was, and its old `/ai-context` address still works.

The migrations add three tables and change nothing that exists: `0080` adds the
evaluation and the waiver, and `0082` adds the record of a repository Kennel Club
has been told not to look at. They apply with the rest, and the usual backup and
no-downgrade rules cover them. Turning it on the first time reads the repositories
this fleet serves, within a GitHub request budget of its own (`kennel.api_budget_percent`,
20 per cent of the limit an installation reports by default), and the first
evaluations arrive over the next few minutes, not at once.

## The usage ledger

Migrations `0047_runner_sessions.sql` and `0048_usage_daily.sql` add two
tables and change none. The first records a session for every runner already
cleaned up when the upgrade runs, reading the runners table without rewriting
it, so the usage report's runner history begins at the oldest runner row the
database still had (about a week back with the default `retention.runners`)
rather than at the upgrade. Runners pruned by an earlier build are gone, and no
migration can bring their hours back; `history_from.runners` on `/usage` says
where the ledger begins.

From then on every runner gets one session when its cleanup is confirmed, or
when the prune takes a row whose cleanup never was, and the prune pass rolls
whole UTC days into `usage_daily` before anything they were computed from can
go. `retention.runner_sessions` keeps sessions for a year by default and never
deletes one the roll-up has not absorbed; the roll-up itself is not pruned. No
configuration is required. The new migrations mean the backup and rollback
rules below apply.

## The per-installation report

Migration `0051_installation_report.sql` adds the daily roll-up of the
[per-installation report](https://zoomies.sh/metrics/#per-installation-report)'s counts and two
columns to `runner_sessions`: the runner's first create task and its job's
eligibility, the two ends of the scheduling interval. Sessions already recorded
get them where the runner and job rows are still there to copy from; the rest
give no scheduling or registration sample, which the report counts as missing
rather than zero. The first roll-up starts from the oldest job or runner row
the database still has, and `counts_from` on the report says so.

From then on the prune pass holds the jobs prune and the sessions prune behind
that roll-up, so a job row is never deleted before the day it belongs to has
been counted. A job still unfinished when `retention.jobs` would take it is
counted as it stood. No configuration is required.

## What happens to work in flight

A restart does not touch a running job. The runner is a container on its host,
the job is executing inside it, and neither is talking to the controller while
that happens, GitHub is. That holds on a host with its own agent, and it now
holds on a single-VM install too, where the agent runs inside the controller:
an agent lists what is already on its host before it starts its loops and
adopts every runner it finds, so a restart finds its own work rather than a set
of containers nobody claims.

It is the controller that says what may be cleaned up. An agent reports the
runners it adopted, and the controller answers with the ones it has no record
of; a runner deleted while the agent was down, say. Only those are removed.
The agent never decides on its own that something is litter, because it cannot
tell "the controller deleted this" from "I have forgotten it", and only one of
those should cost somebody a job.

What a restart interrupts is the *reporting*: webhook deliveries during the gap
are missed, and the fallback poller catches up when the controller returns,
which is one of the reasons to leave it on.

The task queue does not survive a restart either, and that is deliberate: every
task is derived from state the database already holds, so persisting it would
add a second source of truth that could disagree with the runners table. An
agent that finishes work across the gap reports a result for a task the new
controller never issued, and it is applied anyway; the agent did the work, and
the row is the only place that fact can land. Each runner row also records when
its task was last handed to its host, so a restart does not lose how long the
host has actually had it.

One thing a late result cannot do is bring a runner back. If the host was quiet
long enough to be given up on, the fleet has already told an operator, and the
job's timeline, that the runner was gone; a success arriving afterwards does not
make that untrue. Instead the runner keeps its terminal row and the *workload*
is settled: while a job is still running on it the container is left alone to
finish, and once nothing is, it is removed. The job's timeline gains a **runner
returned** entry, so a job that completes normally after its runner was written
off does not read as a contradiction.

Agents keep working while the controller is down. They long-poll, so a
connection that fails is retried with backoff, and a host that cannot reach the
controller does not stop the runner it already started.

Two consequences follow:

* **Upgrade whenever you like.** There is no drain-first ritual. A fleet with
  fifty jobs running is as safe to upgrade as an idle one.
* **A host that stays silent past `store.HeartbeatTimeout`, ninety seconds,
  is counted unhealthy.** A controller that is down for longer than that will
  show every host as unhealthy for a moment when it comes back, until the next
  heartbeat arrives. That is the display catching up, not a fault.

## Version skew

The controller and its agents are separate binaries on separate machines, and
they do not have to match.

The policy, in three rules:

* **The protocol version must match.** It is checked when an agent joins (a
  mismatch is refused there, because an agent that cannot join has nothing
  running to strand) and on **every heartbeat** after that, because an agent
  that joined before a bump would otherwise keep polling and receiving tasks it
  could not understand.
* **An agent may lag the controller by releases**, as long as the protocol
  matches. This is the normal state during a rolling upgrade. The Hosts page
  shows what each host is running.
* **A newer agent against an older controller is unsupported.** It usually
  works, because the controller's API is additive, but it is not a direction
  anyone tests. Upgrade the controller first: it owns the schema and the API,
  and an agent has nothing to migrate.

### What happens when the protocol stops matching

The host is **excluded from placement, exactly as a cordon excludes it**, and
nothing else. Its runners keep working, its agent keeps draining and stopping
them, and the fleet shrinks host by host as it goes.

It is deliberately not a refusal. Answering a heartbeat with an error would
send every agent in the fleet into its re-join path at the same moment, which
is the outage the upgrade was meant to avoid.

The host says `incompatible` on the Hosts page with both protocol versions, a
pool that can no longer place says "running an agent this controller cannot
talk to" with the fix, and the agent logs an error about itself on every
change. Upgrading that agent clears it on the next heartbeat, with no re-join.

An agent old enough not to report a protocol version at all is **not** judged.
It is the one case the controller cannot decide, and guessing would empty a
fleet the moment its controller learnt to ask.

### A task kind an agent does not know

An agent handed a task kind it does not understand reports that task as failed,
with a message saying to upgrade it. The controller treats an unrecognised kind
as **not** lifecycle work, so the runner the task concerned is left alone; a
runner that is running a job is not made to fail by a message neither side can
name. Only `create_runner`, `stop_runner` and `remove_runner` are lifecycle,
and that list is an allowlist so a kind added in a later release is safe on an
older controller by default.

### Two agents as one host

Copying a VM, or a state directory, to a second machine gives two agents one
host identity. Both hold a valid token, both report real work, and each sees
only half of that host's tasks, which looks like a fault almost anywhere else.

Zoomies notices. An agent takes a fresh session each time it starts and never
returns to an old one, so a session a host has already moved on from can only
be a second agent still running, and `host.duplicate_agent` says so. Neither
session is refused: both are executing real jobs, and picking one would end the
other's. Stop the agent on the machine that should not be there and re-join it
with its own join token; the problem clears itself an hour after the sessions
stop swapping.

## The platform role, and what your administrators keep

This release adds a fourth role, `platform`, above `admin`, for whoever runs
the process rather than the fleet. Two things move behind it: lifting the
recovery fence, and taking, downloading and restoring backups. A backup is
the whole database (every account's password hash and every sealed
credential, under the key this host holds) so it belongs to whoever operates
the instance.

**Nobody loses anything on the way through.** Every account that held `admin`
comes up holding `platform`, and so does every API token minted at `admin`
that has not been revoked. That is the same authority as before and not one
action more: `platform` is `admin` plus the two things above. A nightly
`zoomies backup` running on an administrator's token keeps working.

What is carried across is what held `admin` *before* the role existed. If you
track `main` and have already started a build that added the role, an account
or token you have made at `admin` since then stays `admin`; you made it
knowing what `admin` no longer reaches, and an upgrade should not overrule
that. Upgrading from a release, the two steps run seconds apart on the same
start, so this excludes nothing you have.

You do not have to do anything. On an instance one team runs, the change is
invisible: everyone who could take a backup yesterday can take one today, and
the Backups page looks the same.

The role earns its keep on the other shape; an instance one team operates
while another uses the fleet. There you separate the two deliberately, by
giving the fleet's people `admin` and keeping `platform` for whoever runs the
process. An upgrade will not do that to you on its own.

The last enabled `platform` account cannot be demoted, disabled or deleted,
for the same reason the last administrator could not be: an instance nobody
can operate is one only a shell can rescue.

## Schema migrations

Migrations are embedded in the binary, run on first start, and recorded in a
ledger keyed by file name. Two rules the code enforces and a test holds:

* A shipped migration's file name never changes. Renaming one re-applies its
  DDL to every existing database.
* A new migration takes the next unused numeric prefix, alone.

They run in one transaction each, in lexical order, and a failure stops startup
with the name of the file that failed. Nothing is applied twice.

Two migrations change rows rather than shape. `0010_docker_pools_get_a_client`
moves a pool whose `docker_mode` is not `none` from the stock runner image
under a moving tag (`latest`, `main`, or none) to
`ghcr.io/eyupio/zoomies-runner-docker` under the same tag, because the stock
image has no client for the daemon that mode gives it; the API makes the same
change to every pool saved from then on. A pool pinned to a `sha-<commit>` tag is not
touched, and neither is a pool on a digest or on an image of its own. What the
migration left behind, the controller resolves as it makes a runner: every tag
the running build publishes (the channels and the operating-system aliases,
`vX.Y.Z` among them) is swapped there, and a pool on a reference that cannot be
swapped raises `pool.docker_client_missing` rather than failing its jobs one at
a time. Idle runners made from the old
image are drained and replaced on the first scheduler pass. See [Jobs that
build container images](https://zoomies.sh/configuration/#jobs-that-build-container-images).

`0012_job_installation` is the other. It records on every unfinished job which
GitHub App installation covers its repository, matching an installation on the
repository itself before one on the organisation that owns it, and it then
unclaims any waiting or queued job whose pool turns out to belong to a
different installation. Before it, a job carried no installation identity at
all and a pool was chosen for it on labels alone, so on a controller with more
than one installation a job could be, and deterministically was, matched to a
pool in the wrong GitHub target. Those matches are the ones it takes back: the
pool would never have run the job, and the next scheduler pass decides again.
A job in a repository no installation here covers keeps no installation and is
unclaimed for that reason, which the Jobs page and the problems drawer both
say. Jobs that have already finished are not touched, and neither is a job
already in progress: its runner exists, and where it ran is a fact worth more
than a tidy row. **A controller with one installation sees no change**, because
every pool on it belongs to that installation.

## There is no downgrade

**Migrations are one-way.** There are no down migrations, and there is no
command that removes one.

An older binary started against a newer database **refuses to start**, and says
which migrations it does not have. That check arrived after `0.2-beta`, so
rolling back *to* `0.2-beta` itself is the one case where nothing stops you:
that release will come up on a migrated database and look perfectly healthy.
Put the pre-upgrade copy back alongside the binary, which is what the rest of
this section is about. SQLite itself does not object (it has no
opinion about columns nobody reads) which is exactly why the check exists:
without it, the older binary comes up, looks healthy, reads columns whose
meaning it does not know and writes rows the newer one will not accept, and
does all of it silently. Rolling a release back is a thing people do under
pressure, and this is the moment to be told that the database went forward
with it.

So the rollback plan is a copy of the database from before the upgrade, and
**the controller takes one for you**. Whenever it starts and finds migrations
pending on an existing database, it copies the database to
`pre-migration/zoomies-<timestamp>/` beside it before applying anything, and
keeps the last two. It is the same layout `zoomies restore` takes, so putting
one back is one command.

Take your own as well before an upgrade you are unsure about: the automatic one
is beside the database, and a disk that fails takes both.
[Backup and restore](https://zoomies.sh/backup-and-restore/) is the subject.

## Which image tag to run

Four images are published, and the tag says where the build came from rather
than only how new it is.

| Tag | Means | Moves |
| --- | --- | --- |
| `:latest` | the newest full release | when a release is published |
| `v1.2.3` | that release, and only that | never |
| `:dev` | the newest commit on `main` | on every merge |
| `:main` | the same as `:dev` | on every merge |
| `:sha-abc1234` | one commit | never |

```text
ghcr.io/eyupio/zoomies                 the controller
ghcr.io/eyupio/zoomies-agent           an agent, for a host that runs one in a container
ghcr.io/eyupio/zoomies-runner          the runners a pool starts
ghcr.io/eyupio/zoomies-runner-docker   the same, with a Docker client
```

`:latest` on every image means the newest **full release**. It used
to mean the newest commit on `main`, which cost more than a name: both the merge
and the release wrote it, so whichever ran last won, and an operator who pulled
it could get an unreleased build stamped `main-sha-abc1234`. CI now keeps a
rolling `dev` prerelease asset beside the `:dev` images, so the Add Host command
can install the same channel as a controller tracking `main`. Run `:dev` when
you want `main`; it says so.

A prerelease (a tag with a hyphen in it, `v0.1-alpha`, `v1.0-rc1`) is
published under its own tag and does **not** move `:latest`. Name it to run it.

Runner images use the same split. Their default `:dev` follows `main`, while
`:latest` follows the newest full release. Per-platform development tags use
the explicit form `ubuntu-2404-dev`; release builds use
`ubuntu-2404-v1.2.3`. Pin either form on a pool when it must not cross channels.

## What a release carries

Every published binary and the controller image carry a **build-provenance
attestation**: a signed statement that these bytes were built by this
repository's release workflow, from this commit. `install.sh` already checks
the checksum, which says the bytes match what the release names; provenance
says where they came from.

```sh
gh attestation verify zoomies_linux_amd64 --repo eyupio/zoomies
gh attestation verify oci://ghcr.io/eyupio/zoomies:v1.2.3 --repo eyupio/zoomies
```

The attestation is attached twice. `zoomies-provenance.sigstore.json` is the
Sigstore bundle `gh attestation verify` reads; the signed statement together
with the certificate and transparency-log entry that say who signed it.
`zoomies-provenance.intoto.jsonl` is the same signed statement on its own, in
the DSSE envelope that SLSA tooling and the OpenSSF Scorecard recognise as
provenance. Both describe every binary in the release; neither is needed to
install, and `install.sh` checks neither.

### Signing releases

The checksums file travels with the binary, so on its own it says only that a
download was not corrupted on the way, not who made it. A **detached
signature** over `checksums.txt`, checked against a public key compiled into
the running binary, says the release is this project's. `zoomies upgrade`
refuses a release whose `checksums.txt.ed25519` is missing or does not verify,
and `zoomies release verify` checks one by hand:

```sh
zoomies release verify dist/checksums.txt dist/checksums.txt.ed25519
```

The key is minted once, by the owner:

1. Mint a pair. The private half is a credential and is written to a file,
   never printed:

   ```sh
   zoomies release keygen --out ./release-key
   ```

2. Commit the public half, the line in `release.pub`, in five places, so every
   binary and every script trust the same key: `Default` in
   `internal/releasekey/default.go`, and `RELEASE_KEY` in `install.sh`,
   `connect-proxmox.sh`, `deploy/instapods/fetch.sh` and
   `internal/proxmoxsetup/release_signature.sh`. A test fails when any of them
   differs from `Default`.
3. Store the private half, `release.key`, as the repository secret
   `RELEASE_SIGNING_KEY`, and delete it from the machine.

Settings, Updates says which keys the running build trusts, or that it holds
none and takes the checksums on trust, so an operator can tell at a glance
whether a download was held to a signature. `install.sh` checks the signature
too, as far as it can: with a key built in and `openssl` on the host, a
download whose signature does not match the key is refused, and without either
it installs on the checksum alone and says so. It is the one-line installer, so
that is best effort rather than a hard requirement; `zoomies upgrade` holds a
running fleet to the signature in full.

The Proxmox connector (`connect-proxmox.sh`, and the full command the provider
wizard shows for a host that cannot reach zoomies.sh) and the InstaPods build
script hold their downloads to the same key, and run what they download, the
first as root. They add one rule, about which releases must be signed. Only two
kinds of name are exempt: `dev`, which is whatever `main` built last, and a tag
the script can read as a version below 1.3.6 (three numbers, each of one to four
digits), such as `v1.2.0` or `v1.3.5`. A release that old was published before
signing began (v1.3.6 itself was published with no files, so v1.3.6.1 is the
first with a signature), so there is nothing to check on it, and one that is
named outright is taken on its checksum with a warning. Every other name needs a
signature: v1.3.6 and everything above it, and any name the script cannot read as
three numbers of that size, because whoever publishes a release picks its name
and a name nobody can place must not be the way out of the check. That includes
the early tags that do not look like `vN.N.N`, `v0.1-alpha` and `V1.1.0` among
them: they were never signed and the scripts cannot tell that they are old, so
pinning one is refused.

A release that needs a signature is refused when its signature is missing,
cannot be read, or cannot be checked because the host has no `openssl` of 3.0 or
later, because a script that only warned would run whatever was left after
somebody deleted the signature. A signature that is there and does not match
refuses in every case, whatever the release. `install.sh` keeps its best-effort
rule for now.

The InstaPods build script refuses one thing more. With no `ZOOMIES_VERSION` it
takes the release GitHub calls the latest, which is the newest by date, under a
name its publisher chose. If that name is one that needs no signature, the script
stops before it downloads anything and says to set `ZOOMIES_VERSION` to a tag you
trust. An old release that you name yourself keeps the warning and the install,
because naming it was your decision.

`release.yml` signs
`checksums.txt` after it builds, verifies the signature
against the committed key, and attaches `checksums.txt.ed25519` to the release.
A release cut without the secret fails rather than publishing unsigned, and a
tag cut before the public key is committed fails too. A `dev` build, which is
whatever `main` built last, has no release to sign and is not held to a
signature.

To rotate, `Default` carries more than one key, one per line, for one release:
that transition release is signed with the old key (still the secret) and its
binary trusts both, so an instance running an older binary still accepts it and
one running the transition binary accepts the next release, signed with the new
key. `zoomies release verify` says which key a signature names. Ed25519 is the
standard library's; nothing new is depended on. `install.sh`, `connect-proxmox.sh`,
`deploy/instapods/fetch.sh` and `internal/proxmoxsetup/release_signature.sh` (the
copy the provider wizard's full command is built from) carry the same set in their
`RELEASE_KEY`, one key per line, and accept a signature from any key in it; a test
in `internal/docs` keeps each of them equal to `Default`, so a rotation edits all
five together.

To hold one download to a different key, for a mirror or a test that builds its
own, pass the base64 public key to `zoomies upgrade --release-key`, or set
`ZOOMIES_RELEASE_KEY`. The flag wins over the environment
variable, and the environment variable wins over the keys compiled into the
binary. With none of the three, nothing is held to a signature, and the download
says so once.

Every image says what it is without being started, in the standard OCI labels,
`org.opencontainers.image.version`, `.revision` and `.created`. That includes
the runner images, which until recently carried no version at all.

A tag with a hyphen in it (`v0.1-alpha`, `v1.0-rc1`) is published as a
**prerelease**. GitHub keeps prereleases out of `/releases/latest`, so while
every release so far is one, `install.sh` with no `--version` asks the API for
the newest release of any kind instead. Once there is a full release, that is
what "latest" means and prereleases stop being offered. Either way, name the
tag with `--version v1.2.3` when it matters which one you get.

A published full release cannot be rebuilt: the release workflow refuses.
A published prerelease can, because it is still explicitly not finished. A
released tag is a promise about specific bytes, and replacing them behind
people who have already downloaded them is not an upgrade anyone can reason
about.

## Upgrading an agent host

After updating the controller, use **Hosts → Update** for one eligible host
or **Update N hosts** for a rollout. Each host targets the controller's release.
[The UI steps](#update-hosts-from-the-controller) explain how to follow it.

For a local upgrade, copy the command from the host's card. On a supported
native, Compose or Docker installation, the CLI equivalent is:

```sh
# Replace vX.Y.Z with the release shown for the controller.
sudo zoomies upgrade --mode agent --version vX.Y.Z
```

This keeps the existing credentials and host identity. An older installation
can use the [installer upgrade command](#upgrading-from-the-cli); copy the
version-pinned command on its card rather than installing a newer agent
independently of its controller.

A routine update does not require draining the host. If you want a maintenance
window with no new placements, **cordon** it, wait for its current jobs to finish,
upgrade, then **uncordon** it. Use `zoomies hosts list` to watch its runners.
`hosts drain` is a separate retirement action with a stop grace period; it is
not required for the restart to adopt running runners.

On a Windows host, use the installer command from its card in an elevated
PowerShell. The version below is a placeholder for the controller's release:

```powershell
& ([scriptblock]::Create((irm https://zoomies.sh/install.ps1))) -Mode agent -Version 'vX.Y.Z'
```

On a machine that is already enrolled the script stops the service, replaces
`zoomies.exe`, keeps the credentials and starts the service again, with no
token. A Windows host's card gives this command in place of `sudo zoomies
upgrade`, which knows no Windows service. Doing it by hand is the same:
stop the service with `sc.exe stop zoomies-agent`, replace `zoomies.exe`,
then start it with `sc.exe start zoomies-agent`. A Windows agent runs the
`process` backend, so the runner release it downloads is pinned by the pool's
`runner_version` and the digests in the
binary, and an agent behind the controller's release may not know a digest the
controller's default asks for. Keep the Windows agent on the controller's
release so it has the matching runner digests.

An agent that comes back on a release the controller does not recognise is
excluded rather than refused: it keeps heartbeating, its running work finishes,
and no new runner is placed on it. The Hosts page says so on the card. See
[What happens when the protocol stops matching](#what-happens-when-the-protocol-stops-matching).

## Upgrading a container deployment

Use the web UI when this deployment has a supported helper, or
`sudo zoomies upgrade` locally. Both use the recorded deployment, keep its
runtime options and provide the [shutdown budget](#upgrading-from-the-cli).
The earlier Compose method remains available when you manage the service image
and Compose file yourself. Pin the intended service image first, then run:

```sh
docker compose pull
docker compose up -d
```

The database lives in a named volume rather than in the container, so replacing
the container keeps it. `docker compose down` is safe; `down -v` deletes the
volume with the database in it, which is the one command on this page that
cannot be undone.

## After an upgrade

Check **Settings → Updates** for the controller's running release and completed
attempt. On **Hosts**, check that each updated host reports the controller's
version and that the rollout has finished. A failed host halts a rollout;
review its reason before resuming it.

From the CLI, `zoomies updates status` follows managed update attempts and
rollouts; `zoomies status` checks the fleet. `zoomies version` reports the local
executable, so it does not prove that a remote controller or agent has updated.

If a setting was removed or renamed in the release, the
validator says so by name at startup rather than ignoring it: an unknown key in
`zoomies.yaml` is refused, so a setting that silently does nothing is not a
state this can get into.

## RC1 runner timing and restart reporting

An agent restarted over a process runner may not know its eventual exit code,
because the process belonged to the old agent. That outcome is now explicit:
the runner is removed with an unknown-exit message, while GitHub remains the
source of the job result. An exit code the agent did record still distinguishes
a clean exit from a failure.

The RC1 migrations retain older cleanup timestamps as estimates. Confirmed
cleanup needs both host and GitHub observations, so an older row may show
“Awaiting confirmation” alongside its earlier estimate. Upgrade the agents as
well as the controller to receive autonomous host-cleanup confirmations.

## Host health after an upgrade

Upgrades finish with a read-only host health summary and point to `zoomies
doctor` when warnings appear. They never apply OS tuning, including with `--yes`. Container
deployments are offered a native read-only health service through the existing
layout review; approval is required to add it. See [Host health and tuning](https://zoomies.sh/host-health/).