Skip to content
Get started
The Zoomies handbook

The provider contract

An infrastructure provider rents machines. Zoomies asks one for a machine when a pool has queued work and no host can run it, installs an agent inside what comes back, and asks for the machine to be destroyed when it is no longer needed.

This page is for whoever writes one. It is the other side of Proxmox VE, which is for whoever operates the one that exists.

The contract is deliberately small. A provider performs infrastructure operations and nothing else: it does not decide how many machines should exist, when to give up, how long to wait, or what to delete. One reconciler in the controller owns all of that, for every provider, so that a second provider is a new package rather than a new branch in the controller.

What a provider implements

type Provider interface {
    Kind() store.ProviderKind
    Capabilities() Capabilities

    Preflight(ctx context.Context) Report
    Allocate(ctx context.Context, spec MachineSpec) (MachineRef, error)
    Create(ctx context.Context, ref MachineRef, spec MachineSpec) (OperationRef, error)
    Inspect(ctx context.Context, ref MachineRef) (Machine, error)
    List(ctx context.Context, owner Owner) ([]Machine, error)
    Delete(ctx context.Context, ref MachineRef) (OperationRef, error)
    Operation(ctx context.Context, op OperationRef) (OperationStatus, error)
}

Three capabilities are optional, and each is a separate one-method interface found by type assertion rather than a flag: PowerController (start and stop without destroying), Bootstrapper (push the enrolment payload into the guest), and Discoverer (list the nodes, storages and networks the credential can actually see, so the configuration form offers them instead of asking for a string). A provider that cannot do one of these does not implement it. It never implements it and returns "unsupported": a method that is always going to refuse is a method the reconciler will keep calling.

Allocate is separate from Create on purpose

Allocate picks the identity a machine will have (which zone, which native identifier, which name) and creates nothing. Zoomies writes that identity into its own database before it calls Create.

That ordering is the single most important thing in this design, and the whole reason the two are separate calls. When Create times out, or the connection drops after the request went out, the machine may or may not exist. Zoomies then has a name and an identifier it chose in advance, so the answer is to go and look (Inspect, then List) rather than to ask for a second machine.

A timeout is not evidence that creation failed. A provider that makes this untrue (by allocating identity only after creation, with no way to find the result again) cannot be made safe by the reconciler. If your identity is only knowable after the fact, return a ref carrying Name alone from Allocate and make that name findable through List.

Create must also be idempotent for one ref: calling it twice for the same MachineRef must not build two machines.

Failure categories

Every error a provider returns carries a category, because the reconciler's next move is different for each one. Returning a bare error means internal, which retries nothing.

Kind What it means What the reconciler does
config A setting is wrong. Fails the machine. Nothing but an edit fixes it.
auth The credential was rejected. Fails, and stands the provider down for creates.
permission The credential is valid and not allowed to do this. Fails, naming the privilege and the path.
unreachable We could not talk to the provider. Backs off. Never evidence about a resource.
quota Refused for want of resources or a limit. Retries slowly; never by asking for more.
conflict Something else holds the resource; a lock, an identifier taken. Waits and observes.
not_found The resource is not there. For a delete, that is success.
refused A definite refusal, in the provider's own words. Retries with backoff, then fails.
ambiguous We do not know whether it happened. Resolves it by looking. Never retries the operation.
internal Anything unclassified, including a category from a newer contract. Retries nothing, and asks for a person.

permission is kept apart from auth because the advice differs: one is "this token is wrong", the other is "this token is right and lacks VM.Clone on /vms/9000", and an operator sent to the wrong one of those loses an afternoon.

The classification that decides everything

Only the transport knows whether a deadline or a reset happened before or after the request body went out. Before it, nothing can have happened: unreachable. After it, the provider may have done the work and lost the answer: ambiguous.

Get this wrong in the safe direction and a machine waits. Get it wrong in the unsafe direction and you rent two machines and pay for the one nobody is tracking. Classify at the point where the answer is known, inside the HTTP client, and never above it.

An ambiguous failure is never retryable. Its only legal next move is observation.

Operations are durable handles

Create, Delete and the optional power operations return an OperationRef: an opaque handle the provider can be asked about later, through Operation.

It has to survive a controller restart, because that is what it is for. Zoomies stores the handle on the machine row before it considers the call done, and a controller that comes back holding nothing but that handle asks about it rather than acting again. Proxmox's UPID is a good shape: it encodes its own node, so the handle alone is enough to follow the task.

A provider whose operations are synchronous returns a zero OperationRef and sets AsyncOperations: false in its capabilities.

Ownership

Zoomies marks the resources it creates, and verifies those marks before it deletes anything. Owner carries the controller's identity, the provider's, the machine's, a per-machine fingerprint minted with the row, and a creation time.

None of it is authenticated. Anyone with write access to your infrastructure can forge the lot, and the contract says so out loud: the marks are tamper-evidence, not authorisation. The database is the authority. What the marks catch is the realistic accident (a recycled identifier, another controller's resource, one somebody made by hand) rather than an attacker.

A delete therefore needs four facts to agree: the row, a fresh inspection (a cached sweep result is not evidence), this controller holding the fleet's lease, and a state that permits it. Any disagreement quarantines the machine for a person to look at. Nothing is deleted on a guess, and an untracked resource wearing our marks is reported and never removed; it might be another live fleet's.

List must return both the machines carrying this owner's controller mark and any carrying our naming grammar with somebody else's mark. The caller needs both to tell an orphan from a foreign resource, and it must be cheap: one API call where the API allows one.

Deadlines

Every method takes a context and must honour its deadline. A provider declares its own honest budgets in Capabilities().Deadlines, and the reconciler applies the larger of those and the operator's configuration, so a provider may ask for longer, and never for less supervision than the operator asked for.

A provider does not retry, back off, or read a clock for its own decisions. Those belong to the caller, as they do for the GitHub client.

Versioning

ContractVersion is one number. Adding a field, a failure category or a method to an optional interface never raises it; a change that would make a provider written for the previous version behave wrongly does.

A provider declares the range it speaks. The controller refuses to build one whose range does not contain this build's version, names both numbers, and says which side to upgrade, and the machines that provider already owns stay visible, drainable and deletable, because a version mismatch must never strand a running VM.

Two forward-compatibility rules follow from the same instinct, and both fail safe: an unknown failure category reads as internal, which retries nothing; and an unknown machine phase reads as unknown, which never authorises a delete.

Exactly one version is supported, and every provider is in the Zoomies tree. Accepting a range of versions is a decision for the day an out-of-tree provider exists; it was considered and deliberately deferred, because a plugin ABI is a compatibility promise that is much harder to withdraw than to make.

Writing one

RunContractTests(t, name, open) is exported from the provider package and exercises every rule on this page against any implementation. A new provider's test file starts with one line calling it, and the suite is the definition of done:

func TestTheProviderObeysTheContract(t *testing.T) {
    provider.RunContractTests(t, "example", func(t *testing.T) provider.Provider {
        return newExampleProvider(t)
    })
}

provider.NewFake() is an in-memory provider that obeys the same rules, with knobs for the cases that are hard to reach against real infrastructure: an operation that does the work and then reports an unknown outcome, one that reports an unknown outcome having done nothing, a quota that runs out, a resource that vanishes underneath you, and a foreign resource wearing your naming grammar. The whole reconciler is tested against it, with no network.

Two tests keep the contract honest in the other direction: one asserts the package imports nothing beyond the domain types, and one fails on any identifier or comment in it that names a specific provider. A contract that has learned the word vmid has stopped being one.

A provider that opens its own connections has one more rule to keep. Config carries a DialContext, set when the operator reached the provider through a private connection, and every socket the provider opens has to go through it, with no proxy in between: the endpoint's host is then only the name TLS verifies the certificate against, and a provider that dialled it directly would report a hypervisor down that is merely at home. The Proxmox client does this in one place, in its http.Transport, which is where a new provider should do it too.

Last edited 2026-10-08 Improve this page Read as Markdown