Security Model and Multi-Tenancy¶
This page describes what VirtRigaud v0.4.2 enforces when several teams (namespaces) share a cluster, a Provider or a hypervisor, and what you have to configure for it. It covers VM ownership, cross-namespace references, the admission policy, the Provider in-use finalizer, prepared-image identity, private cloud-init seeds, migration storage mounts and the disk-size limit. A table at the end lists what can break an existing setup when you upgrade from v0.3.11.
Out of scope
Transport security between the manager and the providers (mTLS, bearer tokens, network policies), credentials handling, RBAC and container hardening are covered in Security and in the pages under Providers > Security (mTLS, bearer token, network policies). This page does not repeat them.
Trust boundaries stay with you
VirtRigaud's checks are only as strong as the permissions around them. The manager has cluster-wide RBAC, so the checks below are what stop a tenant from driving another tenant's objects. They assume tenants can write objects in their own namespaces and nothing cluster-scoped, and that status subresources of VirtualMachine and VMMigration stay writable by the manager and platform administrators only.
What is enforced, and where¶
| Concern | Enforced by | Section |
|---|---|---|
| A VM is bound to a hypervisor VM only if VirtRigaud created (or adopted) it | The providers, with an owner stamp on the hypervisor VM | VM ownership |
A namespace uses another namespace's Provider, VMClass, VMImage only if it is granted | The controllers (spec.consumerNamespaceSelector) | Cross-namespace references |
| A clone or migration creates a VM in another namespace only if that namespace allows it | The controllers (a Namespace annotation) | Cross-namespace targets |
Only the manager writes the virtrigaud.io markers on a VirtualMachine | A ValidatingAdmissionPolicy in the API server | Admission policy |
A Provider that VMs still use is not deleted from under them | A finalizer on the Provider (ADR-0010) | Provider in-use finalizer |
A prepared image belongs to the VMImage and source it was prepared for | The providers, with an image stamp (ADR-0009) | Identity-safe image preparation |
| A VM's cloud-init user-data is not readable by other local users of a libvirt host | The libvirt provider, with a private seed directory | Private cloud-init seeds |
| Provider pods mount only the migration storage a live migration needs | The manager | Migration storage |
| No disk above an administrator's ceiling is created or grown | The manager (manager.maxDiskSize) | Disk-size limit |
VM ownership¶
A hypervisor identifies a VM by a name or an id that other tenants can also ask for: two VirtualMachines called web in different namespaces want the same vSphere name, a Proxmox VMID can be handed to an unrelated VM after the original is removed, and a VM can exist for reasons that have nothing to do with VirtRigaud. Binding by name or id alone would let one tenant power off, resize or delete another tenant's VM.
The rule, followed by the vSphere, libvirt and Proxmox providers:
VirtRigaud never binds a VirtualMachine to an existing hypervisor VM unless it can prove that this VirtualMachine created (or adopted) it.
How it works¶
- The owner travels with every call. The manager sends the
VirtualMachine's UID, namespace and name asowneronCreate. Only the UID authorizes anything; namespace and name are for audit. - The owner is stamped on the hypervisor VM when it is created.
- A create that finds a VM with the requested name fails closed. It binds to it (and succeeds idempotently) only if the stamp carries the requester's UID; this covers a create retried because the manager lost its
status.idwrite. In every other case the create is refused with a non-retryableConflict.
Where the stamp lives:
| Provider | Stamp | Inspect it |
|---|---|---|
| vSphere | ExtraConfig keys virtrigaud.owner.uid, virtrigaud.owner.namespace, virtrigaud.owner.name (no guestinfo. prefix, so the guest cannot read them) | govc vm.info -e <vm> \| grep virtrigaud.owner |
| libvirt (single-host and clustered) | A <virtrigaud:owner uid=… namespace=… name=…/> element in the domain <metadata>, namespace URI https://virtrigaud.io/xmlns/libvirt/owner/v1 | virsh metadata <domain> --uri https://virtrigaud.io/xmlns/libvirt/owner/v1 |
| Proxmox | One line in the VM description (the "Notes" field): virtrigaud.owner v1 uid=… namespace=… name=… phase=creating\|ready sum=… | qm config <vmid> \| grep virtrigaud.owner |
New libvirt domains are named <namespace>.<name> on the host (VM web in team-a becomes domain team-a.web), so two namespaces' VMs no longer collide. Existing domains keep their names, and every operation after Create uses status.id. Scripts that assumed "domain name equals VirtualMachine name" must read status.id instead.
Which calls are owner-checked¶
| Provider | Create / Clone | Per-VM calls (Describe, Power, Reconfigure, snapshots, disk read and export, Delete) |
|---|---|---|
| vSphere | Stamped; fails closed on a same-named VM in the target folder. Only real templates are clone sources. | Not owner-checked. Delete, Power, Reconfigure, HardwareUpgrade and the snapshot calls refuse a template or an image artifact (VM_IS_IMAGE_ARTIFACT). |
| libvirt, single-host | Stamped; fails closed on a same-named domain. | Not owner-checked. |
| libvirt, clustered | Stamped; fails closed. | Every routed call is owner-checked. See Clustered operations. |
| Proxmox | Stamped; a re-sent Create or Clone re-binds the VM it made earlier instead of making a second VMID. | Owner-checked on every call, so a reused VMID is never read, powered, resized, snapshotted, exported or deleted as if it were the VirtualMachine's. |
The protection needs both the manager and the provider image of v0.4.1. A request without an owner (an older manager) or a provider that ignores it behaves as before. Roll the providers right after the manager.
What you see when a check refuses¶
| Condition | Reason | Meaning | What to do |
|---|---|---|---|
Ready=False, Provisioning=False | ProviderConflict | A VM with that name exists and VirtRigaud cannot prove it owns it. status.id stays empty, so deleting the VirtualMachine never touches the other VM. Re-checked every 2 minutes. | Adopt the VM (below), remove or rename the other VM, or recreate the VirtualMachine under another name. |
Ready=False, Provisioning=False | ValidationError | The provider rejected the request as invalid. For ownership this means a name its tooling could resolve as something else: on vSphere vm-<digits> (the shape of a vCenter object id) or a name containing /, \, % or :. Re-checked every 30 seconds. | Rename the VirtualMachine. |
Ready=False | VMNotOwned | (Proxmox) The VM at this VMID is not this VirtualMachine's. status.id is kept and no VM is created in its place. | See VM status. |
Ready=False | OwnerStampUnreadable | The stamp cannot be parsed or names more than one owner. Nothing is changed. | An administrator fixes or removes the stamp, or detaches the VM. |
Ready=False | VMIsImageArtifact | The hypervisor object is a template or a prepared image, not a VM. | Detach with virtrigaud.io/orphan-on-delete: "true". |
The messages name only the requested VM. They never say which namespace or VirtualMachine owns the other one; the provider log has those details. virtrigaud_errors_total{reason="provider-create-rejected"} counts refused creates.
Adoption stamps what it adopts¶
With virtrigaud.io/adopt-vms on a Provider, VirtRigaud creates a VirtualMachine for each hypervisor VM that no VirtualMachine manages, and stamps the VM with that new VirtualMachine's identity before binding it. So the owner-checked calls accept it, and another Provider object fronting the same endpoint sees it as owned.
Adoption never takes:
- a VM already bound to a
VirtualMachine, in any namespace or under anyProviderobject; - a VM stamped with the UID of a
VirtualMachinethat exists; - a template, an image artifact, or a VM whose stamp is unreadable.
A VM stamped only with UIDs of no VirtualMachine in this cluster is not adopted either, because the stamp may belong to another VirtRigaud installation managing the same vCenter or host. To adopt such VMs on purpose (for example after detaching VMs with orphan-on-delete), on a single-host Provider:
metadata:
annotations:
virtrigaud.io/adopt-vms: "true"
virtrigaud.io/adopt-stamped-vms: "true" # remove again afterwards
Set adopt-stamped-vms only when no other installation manages the same endpoint. A clustered Provider never adopts a stamped VM; it reports it as a previous incarnation (see Backup and restore).
To stop managing a VM without deleting it, annotate and delete the VirtualMachine:
kubectl annotate virtualmachine <vm> -n <ns> virtrigaud.io/orphan-on-delete=true
kubectl delete virtualmachine <vm> -n <ns>
No provider call is made; the hypervisor VM keeps running. Pointing spec.providerRef at a missing Provider no longer works for this: the field is immutable once the VM is bound. On a clustered Provider, a VM in another namespace than the Provider may only be detached or force-deleted if the Provider carries infra.virtrigaud.io/allow-consumer-orphan-on-delete: "true", because a detached domain keeps running outside the capacity accounting.
Limits to know¶
- Proxmox, unstamped VMs. A Proxmox VM created before owner stamps is treated as the
VirtualMachine's own while its PVE name equals theVirtualMachine's name (or the name adoption would give it). Anyone who can create a VM at a freed VMID (VM.Allocateon a shared pool or an inherited path) can exploit that. Do not grantVM.Allocatewhere VirtRigaud's VMs live to anyone but the provider's own tokens and your administrators. ADR-0012 describes how the rule is to be retired; none of that is in v0.4.2. - A stamp is tamper-evident, not tamper-proof. Whoever can edit the VM configuration on the hypervisor (vSphere
AdvancedConfig, libvirt metadata, ProxmoxVM.Config.Options) can rewrite the stamp. The hypervisor's ACLs are the trust boundary. - Give each Proxmox
Providerits own API token, so in-flight create tracking and audit trails are perProvider. createonvmclonesis read access to the source VM. A clone is a full copy of the source disk and boots cloud-init user-data as root; on libvirt theVMClone's own inline user-data runs as root on the copy. Grantcreateonvmclonesonly to subjects trusted with that VM's data. See VM cloning.- vSphere needs one more privilege. Every
Create,Cloneand adoption writes ExtraConfig, so the vCenter account needsVirtualMachine.Config.AdvancedConfig(Virtual machine > Change Configuration > Advanced configuration) on the target folder and resource pool.
Cross-namespace references¶
A VirtualMachine names its Provider, VMClass and VMImage with spec.providerRef, spec.classRef and spec.imageRef, each of which can set a namespace. Without a check, anyone allowed to create a VirtualMachine in team-a could drive team-b's Provider and its hypervisor credentials, clone from team-b's images, or use its classes.
A reference to another namespace's Provider, VMClass or VMImage is refused unless that object shares itself with the referencing namespace. These three kinds have an optional spec.consumerNamespaceSelector (a standard label selector):
spec.consumerNamespaceSelector | Who may reference the object |
|---|---|
| unset (default) | Only its own namespace |
{} | Every namespace |
matchLabels / matchExpressions | Its own namespace plus every namespace whose labels match |
A reference with no namespace, or naming the object's own namespace, always works. The selector is matched against the Namespace object's labels; the API server sets kubernetes.io/metadata.name: <name> on every namespace, so you can name namespaces explicitly. Each object grants for itself: sharing a Provider does not share its classes or images.
A Provider shared with two namespaces by name:
apiVersion: infra.virtrigaud.io/v1beta1
kind: Provider
metadata:
name: vsphere-shared
namespace: virtrigaud-system
spec:
type: vsphere
endpoint: https://vcenter.example.com/sdk
credentialSecretRef:
name: vsphere-creds
runtime:
mode: Remote
image: "ghcr.io/projectbeskar/virtrigaud/provider-vsphere:v0.4.2"
service:
port: 9090
consumerNamespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values: ["team-a", "team-b"]
A VMClass every namespace may use:
apiVersion: infra.virtrigaud.io/v1beta1
kind: VMClass
metadata:
name: shared-small
namespace: virtrigaud-system
spec:
cpu: 2
memory: "4Gi"
consumerNamespaceSelector: {}
A VMImage shared with namespaces labelled virtrigaud.io/tenant=gold:
apiVersion: infra.virtrigaud.io/v1beta1
kind: VMImage
metadata:
name: ubuntu-22-golden
namespace: virtrigaud-system
spec:
source:
vsphere:
templateName: "ubuntu-22.04-cloudimg"
prepare:
onMissing: Fail
consumerNamespaceSelector:
matchLabels:
virtrigaud.io/tenant: gold
A tenant VM that uses all three, from team-a (labelled virtrigaud.io/tenant: gold):
apiVersion: infra.virtrigaud.io/v1beta1
kind: VirtualMachine
metadata:
name: web-01
namespace: team-a
spec:
providerRef: {name: vsphere-shared, namespace: virtrigaud-system}
classRef: {name: shared-small, namespace: virtrigaud-system}
imageRef: {name: ubuntu-22-golden, namespace: virtrigaud-system}
powerState: "On"
The complete example, applied together, is examples/shared-provider-consumer-namespaces.yaml.
Points that catch people out:
- Any selector with no requirements selects every namespace.
{},matchLabels: {}andmatchExpressions: []all mean "every namespace". Only an unset field means "own namespace only". - Negative selectors match namespaces created later.
NotInandDoesNotExistselect every new namespace that lacks the excluded label. Prefer positive selectors. - Sharing a
Providershares everything its credentials can reach. A tenant allowed to use it can write its ownVMImage(no grant needed in its own namespace) naming any template or image file theProvider's account can read. Share aProvideronly with namespaces trusted with that, and scope the hypervisor account accordingly. - Who can grant. Whoever can edit the object sets its selector, and whoever can label a
Namespacedecides whether that namespace matches. On a plain cluster the second is a cluster administrator. Self-service platforms (Capsule, HNC, Rancher projects) may let a namespace owner edit their own namespace's labels; there, usekubernetes.io/metadata.nameselectors or restrict the labels with a Kyverno or Gatekeeper rule. - Selectors match names and labels, not identities. A namespace deleted and recreated under the same name matches again.
When a reference is refused¶
The object reports Ready=False, reason ConsumerNotAllowed, with a message naming the referenced kind, namespace and name and the field that would grant access, plus one Warning event. No provider call is made through the refused reference. For a VirtualMachine this is a hold that reports status.phase: Failed, including for a VM that is running (see the VM status reference); a VMClone or VMMigration already in flight waits instead of failing. The message is the same whether the referenced object does not exist or exists without a grant, so a refusal tells a tenant nothing about another namespace.
What is checked for a VirtualMachine that already exists:
| Operation on a bound VM | Provider grant | VMClass grant | VMImage grant |
|---|---|---|---|
| describe, power on/off | needed | no | no |
| reconfigure to the class | needed | needed | no |
| re-create a VM missing on its hypervisor | needed | needed | needed |
So revoking a Provider share stops everything for the VMs through it (they keep their status.id and their hypervisor VM; nothing is deleted), while revoking an image share never stops VMs created from the image. A VM that is not bound yet needs all three grants before anything is created. VMSnapshot, VMClone and VMMigration check the Provider (and, for a clone or migration, the target's Provider and VMClass) the same way.
To lift a refusal, widen the selector or label the namespace so an existing selector matches. The manager watches both and re-runs the affected objects within seconds; a refused object is also re-checked every 5 minutes.
Deleting a VM whose Provider is refused keeps the finalizer: the manager never deletes a hypervisor VM through a Provider the namespace may not use. Restore access, or set virtrigaud.io/orphan-on-delete: "true" (detach, no provider call), or virtrigaud.io/force-delete: "true" (drop the finalizer).
There is no admission webhook for this check. The controllers enforce it on every reconcile, so a VirtualMachine that references something it may not use is accepted by the API server and then reported as above.
Cross-namespace clone and migration targets¶
A VMClone or VMMigration creates a VirtualMachine, by default in its own namespace. spec.target.namespace can name a different one. That needs the target namespace to allow it, with an annotation listing the source namespaces:
apiVersion: v1
kind: Namespace
metadata:
name: team-b
annotations:
# VMClones / VMMigrations in team-a and team-c may create VMs here.
infra.virtrigaud.io/allowed-source-namespaces: "team-a, team-c"
The value is a comma-separated list of exact namespace names. *, prefixes and a different case do not match. Whoever can update the target Namespace can grant, so keep update/patch on namespaces limited, and restrict the annotation with a tenant policy on self-service platforms. Grants match names: if a listed namespace is deleted and recreated, the new one inherits the grant.
Without the grant the object reports Ready=False, reason TargetNamespaceNotAllowed, and the manager does nothing in the target namespace. The Provider and VMClass pinned onto the target VM must also select the target namespace (consumerNamespaceSelector, above), otherwise the clone or migration waits with ConsumerNotAllowed. The grant is checked again before every step that touches the target namespace, so revoking it stops the next step without changing what already exists.
More rules that apply to cross-namespace targets:
- A libvirt linked clone into another namespace is refused (
LinkedCloneCrossNamespace); use a full clone. - A migration into another namespace through a Proxmox target
Provideris refused atValidating(ProxmoxCrossNamespaceTarget), and through a clustered targetProvideralso (ClusteredTargetUnsupported). - A migration acts only on the target
VirtualMachineit created itself. AVirtualMachinecreated under the target name by someone else fails the migration withTargetVMNotOwned; the other VM is never read, annotated or deleted. Delete or rename it, or choose anotherspec.target.name. VMMigration.spec.source.providerRefmust name theProviderthe source VM runs on. A mismatch fails the migration.- A clone carries the source's cloud-init user-data (on libvirt unless the
VMClonegives its own), provisioning secrets included. Grant a target namespace only to people who may see that user-data.
Admission policy (reserved keys)¶
The manager writes some annotations and labels on the VirtualMachines it creates or adopts: where a migrated VM came from, which VMClone produced it, that it was adopted. Some controllers decide which VirtualMachine a hypervisor VM is bound to from these markers. Without a guard, anyone who can create VirtualMachines in the Provider's or the clone target's namespace could forge them.
The chart installs a ValidatingAdmissionPolicy that lets only the manager's ServiceAccount add or change them. It runs inside the API server (not a webhook), so it does not need webhooks.enabled and keeps working while the manager is down.
It rejects a create of a VirtualMachine carrying a reserved key, and an update that adds a reserved key or gives one a new value, from any identity other than the manager. It does not reject updates that leave the reserved keys as they are (kubectl edit of spec.powerState, an apply of an exported manifest), removing a reserved key, deleting, or status updates. A rejected request fails with Forbidden and names the keys:
... ValidatingAdmissionPolicy 'virtrigaud-reserved-keys' with binding
'virtrigaud-reserved-keys' denied request: annotation(s) virtrigaud.io/migration
are reserved for the VirtRigaud manager and cannot be added or changed
Reserved annotations: virtrigaud.io/migration, virtrigaud.io/migration-claim, virtrigaud.io/migrated-from, virtrigaud.io/imported-disk-id, virtrigaud.io/disk-checksum, virtrigaud.io/migration-completed, virtrigaud.io/migration-completed-at, virtrigaud.io/clone, virtrigaud.io/clone-uid, virtrigaud.io/cloned-from, virtrigaud.io/adopted-host, virtrigaud.io/adopted-id. Reserved label: virtrigaud.io/adopted.
Keys users set themselves are not reserved: virtrigaud.io/orphan-on-delete, virtrigaud.io/force-delete, virtrigaud.io/create-outcome and infra.virtrigaud.io/placement-uid. A VMClone and a VMMigration also drop virtrigaud.io keys from their spec.target.annotations (and the reserved label from spec.target.labels) instead of copying them to the target VM.
Chart values¶
admissionPolicy:
reservedKeys:
enabled: true # default
exemptUsernames: [] # e.g. ["system:serviceaccount:velero:velero"]
exemptGroups: [] # e.g. ["platform-break-glass"]
- Kubernetes 1.30 or later is required. On an older cluster the chart renders nothing, and the adoption and clone markers can be forged by anyone who can create
VirtualMachines in those namespaces; restrictcreateandupdateonvirtualmachinesthere, or reserve the keys with Kyverno or Gatekeeper. Withhelm template, pass--kube-versionwhen you render for an older cluster. - Installing it needs cluster-scoped rights (
validatingadmissionpoliciesandvalidatingadmissionpolicybindings) for the identity that runshelm installorhelm upgrade. - Backup and restore tools (Velero, for example) recreate
VirtualMachines with their annotations and are rejected. Add the tool's identity toexemptUsernamesbefore you restore. - More than one VirtRigaud installation in a cluster: each installation's policy exempts only its own manager. Add the other managers' ServiceAccounts (
system:serviceaccount:<namespace>:<name>) toexemptUsernames. - Cluster administrators are not exempt by default,
system:mastersincluded. List a break-glass group inexemptGroupsif you need one. - Do not set
reservedKeys.enabled: falseunless another admission control reserves the same keys.
The kustomize deployment installs the same policy from config/admission-policy/.
Check it:
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding | grep reserved-keys
# As a tenant: must be refused.
kubectl -n team-a annotate virtualmachine web virtrigaud.io/migration=team-a/forged
The policy and binding are named <release fullname>-reserved-keys, for example virtrigaud-reserved-keys.
Provider in-use finalizer¶
Every Provider carries the finalizer provider.infra.virtrigaud.io/in-use. The manager adds it to existing Providers on the first reconcile after the upgrade. Deleting a Provider that is still needed does not remove it; the Provider waits with its runtime (Deployment, Service, ServiceAccount) still up, so the objects that use it can still be deleted through it. Before this change, deleting a Provider released every VirtualMachine that used it on its next delete and left the hypervisor VMs running unmanaged.
What holds a Provider:
| Object | Holds it while |
|---|---|
VirtualMachine | It is bound through the Provider (or has something on the hypervisor: a status.id, a pending host, or the CreateRequested marker) and still carries the VM finalizer. A VM that never got that far holds nothing. VMs in another namespace count only if the Provider still allows that namespace. |
VMSnapshot | Its delete still has to call SnapshotDelete through the Provider. |
VMClone | In flight (not Ready or Failed) from a source VM that holds it. |
VMMigration | In flight or being deleted, from a source VM that holds it or into this Provider. |
What you see:
DeleteBlocked=True, reason ProviderInUse, with a message giving the number of objects of each kind and the namespaces they are in (at most 10 namespaces, never an object name), and one Warning event ProviderDeleteBlocked. The Provider is released as soon as the last holder goes.
To release it deliberately:
- Delete the VMs first (the normal path), or
- annotate the
Providervirtrigaud.io/force-delete: "true". The runtime is removed and the finalizer released at once, with aWarningeventProviderForceDeletedrecording what still used it. The objects are not touched; VMs in theProvider's namespace are then released without a provider call on their next delete, and the hypervisor VMs stay running, unmanaged. VMs in other namespaces stay held until each is annotatedorphan-on-deleteorforce-delete.
Deleting a namespace that contains a Provider
Kubernetes deletes the provider's pods together with everything else in the namespace, and a finalizer cannot stop that. VMs whose delete arrives after the pod is gone are held (ProviderUnavailable), not orphaned, and the Provider shows DeleteBlocked=True with reason ProviderInUseRuntimeNotKept; the namespace stays Terminating until you act. Delete the VMs, and wait for them to go, before deleting their namespace. A Provider deleted with --cascade=foreground has the same limit; use the default cascade.
Automation that deletes a Provider and waits for it to disappear must now delete its VMs first, or wait longer. Uninstall the manager only after the Providers and VMs are gone; without a manager nothing releases the finalizer. After a rollback to a manager without this change, remove it by hand once no VM uses the Provider:
kubectl patch provider <name> -n <namespace> --type=json \
-p '[{"op":"remove","path":"/metadata/finalizers"}]'
Also, nothing is created through a Provider that is being deleted: right before every Create, Clone and ImportDisk the manager re-reads the Provider from the API server and holds the object with Ready=False/ProviderDeleting if it is going away.
Identity-safe image preparation¶
A prepared image is an artifact on the hypervisor: a vSphere template, a libvirt pool file. It used to be named after the VMImage alone and accepted as "already prepared" if anything of that name existed, so two same-named VMImages in different namespaces collided, and a tenant allowed to use a shared Provider could pre-create the artifact another tenant's VMs are cloned from (ADR-0009).
Now every prepare carries the VMImage's UID, namespace, name and a digest of its spec.source, and:
- the provider names the artifact from that identity, for example
team-a.ubuntu-22.04_3c9e1f0a7b2d4e61(vSphere template, or<pool dir>/team-a.ubuntu-22.04_3c9e1f0a7b2d4e61.qcow2on libvirt), stamps it with the identity, and reuses an existing artifact only if its stamp carries the same UID and digest; - the manager records a prepare only when the provider's answer echoes the stamp it asked for;
- prepare state is kept per
Provider:VMImage.status.providerStatusis keyed by<namespace>/<name>, so aProvidernamedvsphereinteam-bnever satisfies a VM onteam-a/vsphere.
Where the stamp lives:
| Provider | Stamp |
|---|---|
| vSphere | ExtraConfig virtrigaud.image.stampversion, virtrigaud.image.uid, virtrigaud.image.sourcedigest, plus namespace, name, preparedby, preparedat. Inspect with govc vm.info -e <template> \| grep virtrigaud.image. VMs made from the template never carry it. |
| libvirt | A JSON stamp file .<namespace>.<name>_<hash>.virtrigaud-image.json next to the artifact in the pool directory, recording UID, digest, and the artifact's inode and size. |
| Proxmox | Nothing is prepared: URL import (source.http) is refused. Use source.proxmox.templateID with an existing template. |
Source changes and re-created images
A changed spec.source or a re-created VMImage (new UID) gets a new artifact; the old one is left in place. The first VM create after the upgrade prepares each image once more under its new name; artifacts from earlier releases (bare names) are never reused, renamed or deleted.
When the provider refuses¶
The VMImage (and the VM, as WaitingForDependencies) reports one of these reasons. The create is held, not failed.
| Reason | Meaning | What to do |
|---|---|---|
ProviderLacksArtifactIdentity | The Provider imports images but does not report supportsImageArtifactIdentity (an older provider image). Nothing is sent. Retried every 30 seconds. | Upgrade the provider image. |
ArtifactNotConfirmed | The provider's answer did not echo the requested stamp. Retried with a backoff from 30 seconds to 5 minutes. | Upgrade the provider image; check the provider log. |
ArtifactConflict | An artifact exists at the derived name with a stamp that does not match (or none). VirtRigaud never adopts, overwrites or deletes it. Retried every 5 minutes. | Investigate and remove the artifact by hand. |
ArtifactPrepareStalled | Another request has been importing the same artifact for longer than max(2 × spec.prepare.timeout, 2h). | Check the other Provider sharing the image location. |
InvalidSource | The source is rejected (bad URL or checksum mismatch, an OVF the provider refuses, Proxmox URL import). Not retried until the source changes. | Fix spec.source. |
ImportFailed | An asynchronous import failed. Retried after 1 minute, doubling to 30 minutes. | Fix the source or wait. |
SourceDigestMissing, ProviderUIDMissing | With onMissing: Fail or Wait, a recorded entry cannot be matched to the current source or Provider, and those settings forbid the prepare that would re-validate it. | Switch the image to onMissing: Import. |
virtrigaud_image_prepare_artifact_total{provider_type,outcome} counts prepares by created, reused, in_progress and conflict; alert on a rising conflict.
Who may write the image location
The stamp proves provenance only against principals who cannot write the image location. Providers that resolve to the same location (the same vSphere import folder, libvirt pool directory) share one artifact per image and source. Give each tenant's Provider its own import location (spec.defaults.folder on vSphere) and scope its hypervisor account to it, and let nobody else write there.
Reference-style sources (templateName, a libvirt path, a Proxmox template ID) name an existing object and are used as written; they reach whatever the Provider's account can read. libvirt path sources are confined: the file must sit directly in an allowed image directory (VIRTRIGAUD_LIBVIRT_IMAGE_DIRS, default /var/lib/libvirt/images), must not be in use by any domain, and is always copied into the VM's own disk, never attached in place. vSphere templateName must name a real template. Details: Image preparation.
Private cloud-init seeds (libvirt)¶
A libvirt VM's cloud-init seed ISO carries its user-data, which may hold passwords, keys and tokens. Earlier releases built it under the world-listable /tmp at the SSH user's umask. Since v0.4.0 seeds are built in a dedicated directory that only the provider's SSH user can list:
- Where:
VIRTRIGAUD_LIBVIRT_SEED_DIRon the provider pod (set it in theProvider'sspec.runtime.env), default/var/lib/libvirt/virtrigaud-seeds. One absolute directory, applied to every host of a clusteredProvider. It must not be in/tmp,/var/tmp,/run,/dev,/procor/sys. - Permissions: the ISO is
0600(libvirt gives it to the qemu user when the domain starts),user-dataandmeta-dataare removed once the ISO is built, and the SSH user keeps a privatecloud-init.private.isothat clones read. The directory must be owned by the SSH user with no group or other read/write bit (mode0711so the qemu user can traverse it;0700if qemu runs as the SSH user), and every parent must not be writable by others unless sticky. - Fail closed: if the directory is missing and cannot be created, is owned by someone else, or is readable or writable by group or others, every create on that host fails, and so does every clone of a VM that has a seed, with a retryable
refusing to stage cloud-init data, until you fix it. - Every host needs it: each single-host libvirt host, each host of a clustered
Provider, and the target host of a migration into libvirt.
Prepare each host once, as root (replace virtrigaud with the SSH user):
Host preparation as a whole is in Libvirt host preparation.
Operational notes: keep the directory on a local filesystem (not NFS); on a host shared by several Providers with different SSH users, give each its own directory through the environment variable; exclude it from host backups or encrypt and restrict those backups, because every seed holds its VM's user-data in plaintext for as long as the VM exists; and prefer short-lived bootstrap credentials in user-data. SELinux hosts are not tested. If you change the variable later, existing VMs keep their seed in the old directory and Delete no longer removes it there.
Seeds left in /tmp/virtrigaud-cloudinit/ by earlier releases may still hold user-data. Remove the ones no domain references (compare with virsh domblklist --details <domain>).
Cloud-init seeds on Proxmox¶
Since v0.4.1 a Proxmox VM that carries spec.userData gets the whole user-data through a NoCloud seed CD-ROM that the provider builds on the PVE node over SSH. That changes what the Provider's credentials are worth:
- The Provider holds a root SSH credential to the PVE node. The seed's
qm importdiskandqm setrun asroot@pamover SSH, outside the API token's permissions, and PVE attributes the changes to root. Restrict who can read the Provider's Secret (ssh_usermust beroot,ssh_privatekeyorssh_password,known_hosts), and use a key that only this provider uses. Host-key verification is on by default and has no trust on first use. - The user-data is stored in plaintext in the seed volume on the VM's storage for the VM's whole life, as in PVE's own cloud-init drive and libvirt's seed. Keep long-lived secrets out of it.
- The seed never reaches a command line or a log. The three files travel as a tar archive on the standard input of one SSH session, are built in a root-only directory on a tmpfs on the node under a per-VM lock, and are removed after the build. The provider logs the VM, node, storage, slot and the size of the user-data, never its content.
See Proxmox provider: cloud-init.
Customization secrets of a VMClone¶
A VMClone's spec.customization has fields that can hold a credential and that no provider acts on (sysprep.productKey, sysprep.adminPassword.value, sysprep.joinDomain.password.value, sysprep.customCommands, certificates[].data, guestCommands[].command and .arguments, userData.ignition.inline). Since v0.4.1 the manager does not send them to the provider; the Clone request carries an allow-list of what providers apply or report as not applied. The v1beta1 API still stores them in plaintext in the VMClone, so do not put a real credential in them, and rotate any value you set on v0.4.0 or earlier. See VM Cloning.
Logs and staging locations¶
Since v0.4.1 a migration's staging location (an NFS server, export, path and uid/gid; an S3 bucket, key and endpoint) and every image URL are redacted from the provider and manager logs, from provider errors, and from the status, conditions and Warning event of a failed VMMigration. The cause stays readable (MNT3ERR_ACCES, SignatureDoesNotMatch). Logs written by earlier releases keep what they contain: treat them as sensitive. A libvirt create from an image URL no longer puts the URL on the host's command line.
Migration storage¶
Provider pods hold hypervisor credentials, so what is mounted into them is a trust decision. Since v0.4.0, a VMMigration with spec.storage.type: pvc gets its storage mounted like this:
- A PVC is mounted into the source and target
Providerpods only while aVMMigrationin that namespace records it instatus.storagePVCNameand is still in flight. The target mounts it read-only. - The PVC must carry the label
virtrigaud.io/component=migration-storage, beBound, and be either the PVC the migration created (identified by the UID it recorded instatus.storagePVCUID) or the PVC named inspec.storage.pvc.name. Before, any PVC in the namespace with that label was mounted read-write into every provider pod. - For a PVC you name yourself, label it first:
A migration fails at Validating if the named PVC lacks the label or belongs to another VMMigration. - Deleting a VMMigration deletes only the PVC it created. A PVC you named is unmounted and left for you (MigrationPVCRetained event). - Provider pods roll once when a PVC is added or dropped, which interrupts an export or import running at that moment. Upgrade between migrations.
Other migration rules that matter in a shared environment:
- NFS migrations refuse
spec.storage.nfs.uidandgidof0(reasonNFSRootIdentityNotAllowed), because the client chooses the identity it presents and uid 0 is root on an export withoutroot_squash. Use a dedicated non-zero uid and gid that owns the export. - A disk landed by a migration is attached in place only to the
VirtualMachinethe migration created (itsspec.importedDisk.migrationRefnames aVMMigrationin the VM's own namespace). Any other imported-disk path is treated as a base image: confined and copied on libvirt, refused on vSphere and Proxmox (ValidationError). - A failed or deleted
VMMigrationdeletes the disk its import landed on the target, unless the disk is in use (spec.options.cleanupPolicydecides when:OnSuccess(default),Always,Never).
See Migration user guide for the migration workflow itself.
Disk-size limit¶
By default the only bound on a VM disk is the CRD's 64Ti per disk. A tenant who owns a namespace can ask for that much and fill a shared datastore or pool. The administrator's ceiling is opt-in and tenants cannot raise it:
# values.yaml
manager:
maxDiskSize: "500Gi" # whole number of bytes, 1Gi to 64Pi; empty = no ceiling
Use binary units (Gi, Ti): 500G is only about 465.7Gi. With it set:
- A VM whose
VMClassspec.diskDefaults.size(on libvirt, 20Gi when the class sets none) or anyspec.disks[].sizeGiBis above the ceiling is not created: it waits withReady=False, reasonDiskSizeExceedsLimit, and aWarningevent, and is created once its sizes fit. - A
Reconfigurenever grows a disk beyond it. The VM's other changes apply, the over-limit disk is left as it is, and the VM reportsDiskSizeWithinLimit=False(reasonDiskSizeExceedsLimit). - A
VMCloneorVMMigrationis checked against the real sizes of the source's disks as its provider reports them, and the target provider enforces the limit again when it lands a migrated disk. A size that cannot be read refuses the operation (DiskSizeLimitUnverifiable). - VMs that already have larger disks keep running and are never shrunk.
A Provider's owner can lower the ceiling for that Provider (never raise it) with an annotation:
An invalid annotation value is refused by the Provider webhook; if one is stored anyway, nothing is created or grown through that Provider (DiskSizeLimitUnverifiable) until it is fixed. With webhooks.enabled=true and manager.maxDiskSize set, the chart also renders validating webhooks (vvirtualmachine-disksize.kb.io, vvmclass-disksize.kb.io) that refuse a larger size at kubectl apply time; they check the manager's ceiling only. The controllers enforce the limit whether or not webhooks are on.
What the limit does not cover: it bounds the sizes VirtRigaud asks for, per disk. A primary disk that comes from an image already larger than the limit is not checked, and totals per VM or namespace are not limited; use hypervisor storage quotas for those. The clone and migration checks need provider images of v0.4.1: an older image does not report disk sizes, so while a limit applies clones through it are refused.
Upgrade impact¶
Read Upgrading for the full procedure and order. These are the changes from this page that can break an existing setup.
| Change | Who is affected | What happens after the upgrade | Action |
|---|---|---|---|
Cross-namespace Provider/VMClass/VMImage references need a grant | Any VM, clone or migration referencing an object in another namespace | The VM fails closed with Ready=False/ConsumerNotAllowed, including running VMs: no provider calls, no deletes, until granted | After applying the new CRDs and before rolling the manager, set spec.consumerNamespaceSelector on every shared object |
| Cross-namespace clone and migration targets need a grant | VMClone/VMMigration with spec.target.namespace set to another namespace, including ones in flight | Waits with TargetNamespaceNotAllowed | Annotate the target namespace with infra.virtrigaud.io/allowed-source-namespaces |
spec.providerRef is immutable once the VM is bound | Tooling that edits it; the old "re-point then delete" un-adopt trick | The edit is rejected (422) | Use virtrigaud.io/orphan-on-delete |
Provider.spec.type is immutable | Tooling that edits it | The edit is rejected | Create a new Provider |
vSphere create and clone need VirtualMachine.Config.AdvancedConfig | All vSphere users | Creates fail with a vCenter permission error | Grant the privilege |
Create fails closed on a same-named VM it does not own | A create retried after the manager lost status.id, for a VM created before owner stamps; any collision with an unstamped VM | ProviderConflict; VMs already bound (status.id set) are unaffected | Adopt the VM, or remove it |
New libvirt domains are named <namespace>.<name> | Scripts and monitoring keyed on the domain name | Existing VMs keep their names | Read status.id |
| libvirt default cloud-init no longer provisions credentials | libvirt VMs created without spec.userData | No default ubuntu user, key or sudo | Supply your own spec.userData |
| libvirt base images are confined and always copied | Images outside /var/lib/libvirt/images, or attached in place by an earlier release | Such images are refused as a base | Set VIRTRIGAUD_LIBVIRT_IMAGE_DIRS; create a VMImage under a new name for a fresh copy |
| Seed directory must exist on every libvirt host | Every libvirt Provider | Creates and clones fail with refusing to stage cloud-init data until the directory is prepared | Run the install -d command above on each host |
Image prepares carry the VMImage identity | Every import-style image (ovaURL, libvirt url) | One re-import per image and location; a provider image without the capability is held (ProviderLacksArtifactIdentity); Proxmox URL import is refused | Roll provider images with the manager; use a Proxmox templateID |
| Reserved keys policy | Backup tools that restore VMs with annotations; a second VirtRigaud installation | Their writes are rejected with Forbidden (Kubernetes 1.30+) | Add them to admissionPolicy.reservedKeys.exemptUsernames |
Deleting a Provider that is in use waits | Automation that deletes a Provider and expects it to disappear | DeleteBlocked=True/ProviderInUse | Delete the VMs first, or force-delete the Provider |
| Provider pods mount only referenced migration PVCs | pvc migrations; PVCs labelled virtrigaud.io/component=migration-storage | Provider pods roll once; unreferenced labelled PVCs are no longer mounted | Label a PVC before naming it in spec.storage.pvc.name; delete your named PVC yourself |
| NFS migrations refuse uid/gid 0 | VMMigration with nfs.uid: 0 or nfs.gid: 0 | Fails at Validating | Use a non-zero uid/gid |
| Linked clones fail closed | VMClone with options.type: LinkedClone | libvirt linked clones are refused; a libvirt linked clone into another namespace is refused | Use a full clone |
| The disk-size limit | Nobody, unless manager.maxDiskSize is set | No change when unset | Set it if you need it |
Changes of v0.4.1 that touch the security model (from v0.4.0; the full list is in the upgrade notes):
| Change | Who is affected | Action |
|---|---|---|
A VMClone's unused customization secrets are no longer sent to the provider | Anyone who set sysprep.productKey, sysprep.adminPassword.value, sysprep.joinDomain.password.value, sysprep.customCommands, certificates[].data, guestCommands or userData.ignition.inline on v0.4.0 or earlier | Rotate those values; they were sent, and they are still stored in plaintext in the VMClone |
| Proxmox VMs with user-data need a root SSH credential in the Provider's Secret | Proxmox Providers whose VMs carry spec.userData | Add ssh_user, ssh_privatekey or ssh_password and known_hosts before rolling the provider; restrict who can read the Secret |
| Staging locations and image URLs are redacted from logs | Log queries and alerts that matched them | Update them; treat logs from earlier releases as sensitive |
Roll the providers promptly after the manager. Most of the checks above need a v0.4.0 (or later) provider image as well as the manager, and a manager ahead of its providers holds some operations (for example, image prepares are held on a provider that does not report artifact identity, and a Create whose outcome is unknown is not sent again to a provider that does not report owner re-bind).
Further reading¶
The reference documents in the main repository carry the full detail, including every edge case: VM ownership, libvirt domain ownership, cross-namespace references, cross-namespace targets, admission policy, image preparation and upgrading from v0.3.11. For what each condition reason means on a VirtualMachine, see VM status reference.