VirtRigaud Documentation¶
Welcome to the VirtRigaud documentation. VirtRigaud is a Kubernetes operator for managing virtual machines across multiple hypervisors — VMware vSphere, Libvirt/KVM, and Proxmox VE — through a single declarative API.
What VirtRigaud is¶
VirtRigaud lets you declare VMs the same way you declare Deployments. A VirtualMachine custom resource describes the desired VM (size, image, networks, disks, power state); a small set of controllers in the manager reconcile that desire against whichever hypervisor backs the namespace's Provider.
Hypervisor-specific logic does not run inside the manager. Each provider runs as its own gRPC server pod, and the manager talks to it over a stable proto contract (proto/provider/v1/provider.proto). This isolates hypervisor failures, lets providers be upgraded independently, and means a misbehaving provider cannot crash the rest of the operator.
Architecture at a glance¶
┌────────────────────────┐
│ Kubernetes API │
│ VirtualMachine, │
│ Provider, VMClass, │
│ VMImage, VMNetwork- │
│ Attachment, VM- │
│ Snapshot, VMMigration,│
│ VMClone, VMSet, │
│ VMPlacementPolicy, │
│ Host, HostPool │
└───────────┬────────────┘
│ watch / status
┌───────────▼────────────┐
│ virtrigaud-manager │
│ (12 reconcilers) │
│ │
│ per-RPC interceptors: │
│ 1. request-sent │
│ 2. metrics │
│ 3. CircuitBreaker │
└───────────┬────────────┘
│ gRPC (mTLS)
┌─────────────────────────────┼─────────────────────────────┐
│ │ │
┌───────▼──────┐ ┌────────▼─────────┐ ┌────────▼────────┐
│ provider- │ │ provider- │ │ provider- │
│ vsphere │ │ libvirt │ │ proxmox │
│ (govmomi) │ │ (SSH + virsh, │ │ (REST API) │
│ │ │ single host or │ │ │
│ │ │ clustered) │ │ │
└──────────────┘ └──────────────────┘ └─────────────────┘
│ │ │
vCenter / ESXi KVM / QEMU host(s) Proxmox VE host
One manager binary, one manager Dockerfile (consolidated in v0.3.6), and one gRPC client per Provider CR. Each outbound RPC passes through three unary client interceptors: a request-sent interceptor (tells a call that never left the manager from one whose outcome is unknown), a metrics interceptor (per-RPC latency and status-code counter) and a CircuitBreaker interceptor (one breaker per Provider CR). Provider connections require mTLS unless the Provider CR explicitly opts out.
Quick Navigation¶
Getting Started¶
- 15-Minute Quickstart - Get up and running quickly
- Installation Guide - Helm installation instructions
- Basic VM Example - Create your first VM
- Helm CRD Upgrades - Managing CRD updates
Guides¶
- Provider Guides - Overview and provider setup
- Provider Capabilities Matrix - Feature comparison
- Advanced Topics - Advanced VM operations
- Security Model and Multi-Tenancy - Ownership, cross-namespace grants, admission policy
- VM Status Reference - Phases, conditions, reasons and held provider calls
Provider-Specific Guides¶
- vSphere Provider - VMware vCenter/ESXi integration
- Libvirt Provider - KVM/QEMU virtualization
- Proxmox VE Provider - Proxmox Virtual Environment
- Provider Tutorial - Build your own provider
- Provider Versioning - Version management
Advanced Topics¶
- VM Lifecycle Management - Advanced VM operations
- Nested Virtualization - Run hypervisors in VMs
- Graceful Shutdown - Proper VM shutdown handling
- VM Snapshots and Revert - Take a snapshot and revert a VM to it
- Clustered Libvirt (Experimental) - One Provider in front of several libvirt hosts: placement, operations, backup and restore
- Remote Providers - Provider architecture
Operations¶
- Overview - Operations guide
- Observability - Monitoring and metrics
- Security - Security best practices
- Resilience - High availability and fault tolerance
- Upgrade Guide - Version upgrade procedures
- vSphere Hardware Versions - Hardware compatibility
Security Configuration¶
References¶
- Custom Resource Definitions - Complete API reference
- CLI Tools Reference - Command-line interface guide
- CLI API Reference - Detailed CLI documentation
- Metrics Catalog - Available metrics
- Provider Catalog - Available providers
- Full CRD Reference - Generated field-level schema
- Provider gRPC API Guide - How the manager uses the contract and what changed in v0.4.0
- Provider gRPC API - Generated manager-to-provider contract
Development¶
- Overview - Development guide
- Building Locally - Build from source
- Contributing Guide - Contribution guidelines
- Testing Locally - Local testing
Examples Directory¶
- Example README - Overview of all examples
- Advanced Examples - Snapshots, reconfigure, cloning, task tracking
- Clustered libvirt example - Provider, HostPool, Hosts and a first VM
- Shared Provider example -
consumerNamespaceSelectorfor multi-tenant clusters - Security configuration - Credentials, mTLS and network policy examples
Custom Resources¶
VirtRigaud v0.4.2 ships 12 Custom Resource Definitions in the infra.virtrigaud.io/v1beta1 API group:
| Kind | Purpose |
|---|---|
VirtualMachine | Desired state of a single VM (image, class, networks, disks, power state). spec.snapshot.revertToRef reverts it to a VMSnapshot. |
Provider | Connection details for a hypervisor (vSphere / Libvirt / Proxmox) and its provider-server Deployment. spec.topology: cluster (libvirt only) fronts several hosts; spec.consumerNamespaceSelector shares it with other namespaces. |
VMClass | Reusable sizing template (CPU, memory, firmware). Can be shared with other namespaces through spec.consumerNamespaceSelector. |
VMImage | Reference to a hypervisor template / OVA / cloud image. Can be shared with other namespaces through spec.consumerNamespaceSelector. |
VMNetworkAttachment | Logical network the VM attaches to (resolved per-provider). |
VMSnapshot | A point-in-time snapshot of a VirtualMachine. |
VMMigration | Inter-provider VM migration (export → import) through S3 or NFS staging (a legacy PVC backend also exists). |
VMSet | Replica-set of identical VMs. CRD available; controller stub only — Ready=False/ControllerNotImplemented in v0.4.2. |
VMPlacementPolicy | Placement / anti-affinity rules. Applied via VirtualMachine.spec.placementRef; the clustered libvirt scheduler uses it as its constraint language. There is no standalone controller. |
VMClone | One-shot clone of an existing VM (vSphere, Proxmox and Libvirt). Only vSphere makes linked clones; libvirt and Proxmox serve full clones and refuse a linked one (LinkedCloneUnsupported). |
Host | One hypervisor host behind a clustered libvirt Provider, with its live capacity and health in status. |
HostPool | A group of Hosts under one clustered Provider, with scheduling strategy and overcommit policy. status counts total, ready and schedulable hosts. |
The manager also runs a VMAdoption reconciler (not a CRD — it reconciles Provider resources annotated with virtrigaud.io/adopt-vms: "true", discovers VMs the hypervisor already owns, and creates VirtualMachine CRs labelled virtrigaud.io/adopted=true for them, plus an adopted-Ncpu-Nmb VMClass per unique sizing). Adoption skips templates and image artifacts, and, since v0.4.0, VMs whose owner stamp names no VirtualMachine in this cluster. See the generated CRD reference for the full schema.
Version Information¶
This documentation covers VirtRigaud v0.4.2, the current release. The generated CRD and gRPC references are generated from the v0.4.2 tag.
Upgrading from v0.4.1 or a v0.4.2 release candidate
v0.4.2 adds one optional status field (status.placement.resizeReservation), so the CRDs must be installed before the manager, and two manager flags. Upgrade the chart with --reset-then-reuse-values and override any image tag you pinned, roll each Provider's spec.runtime.image yourself (Helm does not change it; roll the vSphere providers together with the manager), and do not roll back to v0.4.1 while a clustered VM holds a resize reservation. Read Upgrading from v0.4.1 to v0.4.2 first.
Upgrading from v0.4.0
v0.4.1 has no CRD schema change and the same roll order, but a Proxmox Provider whose VMs carry user-data now needs SSH credentials in its Secret, libvirt VMClasses with firmware: UEFI now make UEFI VMs, and secrets set in a VMClone's spec.customization on v0.4.0 or earlier should be rotated. Read Upgrading from v0.4.0 to v0.4.1 first.
Upgrading from v0.3.11
v0.4.0 is a security-hardening release with breaking changes (cross-namespace references need a grant, spec.providerRef is immutable once bound, vSphere needs a new vCenter privilege, libvirt hosts need a seed directory, and more). Read the Upgrade Guide and Helm CRD Upgrades before upgrading.
Recent Releases¶
- v0.4.2 — Control-plane scalability, clustered admission safety and Provider fault isolation. A patch release of v0.4.1; changes no API field's meaning and no gRPC message; adds the optional
VirtualMachinestatus fieldstatus.placement.resizeReservationand two manager flags. It ships the code of v0.4.2-rc2. Release notes.- Less control-plane load. Running VMs are polled every 2 minutes (5 when powered off) instead of every 10 seconds, unchanged status is not rewritten, a provider connection is validated once instead of on every reconcile, a clustered Provider's VM counts and Host capacity are read from indexes instead of full scans, and a new VM is described as soon as its
Createis recorded. The figures in the release notes come from a test environment with a fake provider and from microbenchmarks, not from production clusters. - Admission safety on clustered Providers. A placement is released only on evidence; an admitted resize-up is reserved durably (
status.placement.resizeReservation,Reconfiguring=True/ResizeReserved) before it is sent and released only on a confirmedReconfigure; an admission inherited from an earlier manager is checked again before it is executed (InheritedAdmissionOverCapacity); a VM whose size cannot be established is never counted at a guessed size. See Clustered placement. - Fault isolation. One VM's backend failures no longer open its Provider's circuit breaker and block the Provider's healthy VMs (#472, found while qualifying rc1 and fixed in rc2); a Provider that does not answer still opens it. The vSphere provider logs in to vCenter again on its own after an outage. See Resilience.
- Configurable client rate limits.
--client-qpsand--client-burst(chartmanager.clientQPS,manager.clientBurst), defaults unchanged at 20 and 40. - Security update.
golang.org/x/netv0.62.0 in every Go module and binary (CVE-2026-78661, CVE-2026-97045; no vulnerable function reachable from VirtRigaud code); the CRD-hook image shipskubectl1.35.10. Some findings remain: see Security. - Rollback. Do not roll back to v0.4.1 while a clustered VM holds a resize reservation, and roll the manager and the chart back together. See Rolling back to v0.4.1.
- Less control-plane load. Running VMs are polled every 2 minutes (5 when powered off) instead of every 10 seconds, unchanged status is not rewritten, a provider connection is validated once instead of on every reconcile, a clustered Provider's VM counts and Host capacity are read from indexes instead of full scans, and a new VM is described as soon as its
- v0.4.1 — Proxmox cloud-init user-data, libvirt platform features, and CLI tools that perform the operations they document. A patch release of v0.4.0; the CRDs have no schema change.
- Proxmox: the whole user-data reaches the guest. The provider builds a NoCloud seed on the PVE node over SSH, so
users,write_files,packages,runcmdand every other cloud-config key apply, as on libvirt and vSphere. This needs SSH credentials in the Provider's Secret and is breaking for Proxmox users whose VMs carry user-data. The seed is kept out of the boot order; a clone gets a seed of its own and the class's memory; linked clones are no longer advertised; the loser of a VMID race is answered "not performed" and retried. See Proxmox. - libvirt platform. Secure boot, VT-d, nested virtualization and VMs with several network interfaces work on libvirt 10.
firmware: UEFIis honoured now (it was ignored), so check your VMClasses before you roll the libvirt provider; the host needs an OVMF package. See libvirt firmware. - Controller and Provider status. A refused
VirtualMachinecreate is retried on a backoff (30 seconds doubling to 10 minutes) and at once when its inputs change, with new condition andCreateBackoffmessages; a provider's answer survives a status-write conflict;Provider.status.reportedCapabilitiesis read again after a provider roll, andProviderAvailableandstatus.healthyfollow the ready replicas. - libvirt snapshots. A snapshot whose overlay file is already on the host is refused for good, and "not performed" snapshot and revert answers are retried.
- Security. A
VMClone's unused customization secrets are no longer sent to the provider (rotate any real value you set); staging locations are redacted from every provider and manager log; libvirt create-from-URL downloads withcurlwith the URL off the host's command line. - Tooling.
vcts runruns the conformance tests,virtrigaud-loadgenperforms real power, reconfigure, snapshot and clone operations and enforcesvmCount,vrtg-provider init --type proxmoxscaffolds a Proxmox provider, and thesdkandprotoGo modules are tagged with each release. Thevirtrigaud-provider-runtimechart is deprecated. See the upgrade notes.
- Proxmox: the whole user-data reaches the guest. The provider builds a NoCloud seed on the PVE node over SSH, so
- v0.4.0 — Security hardening, clustered libvirt providers and snapshot revert.
- Clustered libvirt providers (experimental, opt-in).
Provider.spec.topology: clusterfronts several libvirt hosts registered asHostresources in aHostPool. A placement scheduler (usingVMPlacementPolicy) picks a host per VM, and every later call is routed to that VM's host and checked against its owner stamp. Create, describe, delete, power, reconfigure, snapshot, full clone and S3/NFS export work across hosts; aVMMigrationcan target a clustered Provider. Host-to-host migration is not implemented. A clustered Provider runs one replica, so a rollout is a short outage. - Snapshot revert.
VirtualMachine.spec.snapshot.revertToRefnow reverts the VM (before, the field was accepted and ignored). The VM must be powered off; progress is instatus.snapshotRevertand theSnapshotRevertedcondition. On libvirt a revert needs a snapshot taken with the VM powered off. - Security hardening. VM ownership stamps and fail-closed ownership checks on vSphere, libvirt and Proxmox; cross-namespace
Provider/VMClass/VMImagereferences needspec.consumerNamespaceSelector; cross-namespace clone and migration targets need a grant; aValidatingAdmissionPolicy(Kubernetes 1.30+) reserves the manager-owned annotations; aProviderthat is still in use is not deleted from under its VMs (ADR-0010); identity-safe image preparation (ADR-0009); private libvirt cloud-init seeds; aVMMigrationacts only on the target VM it created; pod/tmpsize limits; an opt-in administrator disk-size limit (manager.maxDiskSize); shell-quoted and XML-escaped libvirt commands and definitions. - Behaviour changes.
spec.providerRefis immutable once a VM is bound (usevirtrigaud.io/orphan-on-delete: "true"to detach); a libvirt VM withoutspec.userDatagets no default login; libvirt linked clones are disabled;powerOnonVMCloneandVMMigrationis honored (defaultfalse); libvirtReconfigurereports a change that needs a restart as such (Reconfiguring=True/RestartRequired);status.ipslists only reachable addresses;status.phaseis set on every VM. - Proxmox. The bridge, VLAN tag, NIC model and static IP of a VM's networks are applied; the cloud-init drive is placed on the VM's disk storage; imports of a migrated disk are accepted only from the VM's own verified migration stage.
- Operations. A provider call whose outcome is unknown is held for an operator instead of being re-sent blindly (
SnapshotOutcomeUnknown,ExportOutcomeUnknown,ImportOutcomeUnknown); a delete held for an unreachable clustered host says so (DeleteBlocked/HostUnreachable); new metricsvirtrigaud_host_committed_cpuandvirtrigaud_host_committed_memory_mib; the gRPC contract gains host inventory (ListHosts,GetHostInfo),TransferOwner,DeleteStagedDiskandDeleteLandedDisk(see Provider gRPC API).
- Clustered libvirt providers (experimental, opt-in).
- v0.3.11 — NFS cross-hypervisor migration staging backend (ADR-0006 Slice 4).
- Second storage-agnostic
VMMigrationbackend: NFS (#236). Alongside the S3 backend, the disk can now be staged on an NFS export and moved withqemu-img's native transport — no PVC, no relay, no S3 credentials. Lab-validated for libvirt ⇄ libvirt, libvirt ⇄ vSphere and libvirt ⇄ Proxmox VE; vSphere ⇄ Proxmox VE over NFS was not run. - Per-provider NFS transport. libvirt (host-side) and vSphere (pod-side) use qemu-img's
nfs://libnfs driver; Proxmox kernel-mounts the export becausepve-qemu-kvmships no libnfs driver. nfs.uid/nfs.gidare effectively mandatory for cross-provider migrations (AUTH_SYS authorizes by the numeric uid/gid each leg presents); set them to the export owner. The staged object is a single flat file in the export root.- Proxmox now advertises
s3andnfs(not PVC: its disks live on the node, unreachable by a pod-mounted PVC). Every pair among the three hypervisors was validated over S3, and five pairs over NFS. See the VM Migration User Guide.
- Second storage-agnostic
- v0.3.9 — Libvirt online operations + clone + image-prepare + memory snapshots on all three providers.
- Libvirt online CPU/memory reconfigure (#203).
virsh setvcpus/setmem --livewith no power-cycle for VMs created withcpuHotAddEnabled/memoryHotAddEnabledon the VMClass (4× headroom, 64-vCPU cap). Lab-proven. - Libvirt online disk expansion (#201).
virsh blockresize+ best-effort in-guest FS grow. Grow-only; no power-cycle. Lab-proven. - Libvirt memory snapshots (#202). RAM-inclusive checkpoints on running VMs. Stopped VM downgrades to disk-only with a WARN.
- Libvirt clone (#153) + clone hardening (#208/#221). Full and linked (qcow2 overlay) clones, same-provider. UEFI nvram re-pointed per clone; hot-add headroom preserved across class-override clones.
- End-to-end image preparation (#154).
VMImage.spec.preparedrives lazy VM-create-time image import on libvirt, vSphere (OVA/OVF), and Proxmox.prepare.onMissing: Failholds VM creation withWaitingForDependenciesuntil the image is ready. - Proxmox export compression (#219).
ExportDiskhonorsCompressfor qcow2 targets. - Memory snapshots on all three providers. vSphere (#200):
CreateSnapshot(memory=true); libvirt (#202):snapshot-create-as; Proxmox: PVE-native.
- Libvirt online CPU/memory reconfigure (#203).
- v0.3.8 — VMClone MVP + VMSet stub + secure-by-default chart.
- VMClone controller (MVP). Full and linked clones functional via
VMCloneCRD (vSphere and Proxmox; libvirt added in v0.3.9). - VMSet: CRD defined, controller not yet active. A
VMSetresource is accepted but emitsReady=False / Reason=ControllerNotImplemented. - Chart (#173): providers disabled by default. Opt-in via
providers.<type>.enabled=true.
- VMClone controller (MVP). Full and linked clones functional via
- v0.3.7 — mTLS enforcement, multi-arch images, circuit-breaker metrics.
- v0.3.6 — Observability + supply-chain release (CircuitBreaker, G7 metrics, H1 build consolidation, Go 1.26 floor, OTel CVE fixes).
- v0.3.5 — Observability G-track foundation (RPC metrics, error counters, reconcile metrics).
- v0.3.3 — Changelog organisation with versioned release headers.
- v0.2.3 — Provider feature parity: Reconfigure, Clone, TaskStatus, ConsoleURL.
See CHANGELOG.md for the complete version history.
Provider Status¶
| Provider | Status | Maturity | Documentation |
|---|---|---|---|
| vSphere | Production Ready | Stable | Guide |
| Libvirt/KVM (single host) | Production Ready | Stable | Guide |
Libvirt/KVM (clustered, topology: cluster) | Opt-in | Experimental | Guide |
| Proxmox VE | Production Ready | Beta | Guide |
| Mock | Complete | Testing | providers/tutorial.md |
Support¶
- GitHub Issues: github.com/projectbeskar/virtrigaud/issues
- Discussions: github.com/projectbeskar/virtrigaud/discussions
Quick Links¶
- Main README - Project overview
- CHANGELOG - Version history
- Contributing - How to contribute
- License - Apache License 2.0