VM Migration Guide — architecture and internals¶

This page is the architectural deep dive for VirtRigaud's VMMigration flow in v0.3.11. It complements the practical User Guide and the field-by-field API Reference.

If you want to migrate a VM today, start with the User Guide. If you want to understand why the controller is organized the way it is, or you are debugging a failed migration, read on.

Where the code lives¶

Validated direction matrix (v0.3.11)¶

High-level model¶

                    ┌──────────────────────────────────┐
                    │      VMMigration CR (CRD)        │
                    │  spec.source / target / storage  │
                    └──────────────┬───────────────────┘
                                   │ watch + reconcile
                                   ▼
            ┌───────────────────────────────────────────────┐
            │       VMMigrationReconciler                   │
            │  (~2074 LOC, phase machine in switch on       │
            │   migration.Status.Phase;                     │
            │   vmmigration_controller.go:198-218)          │
            └───────────────┬───────────────┬───────────────┘
                            │               │
          ┌─────────────────┘               └──────────────────┐
          ▼                                                     ▼
  Annotate Source + Target Provider CRs          Validate + prepare staging backend
  virtrigaud.io/migration-pvc=<pvc>              (S3 bucket/key; NFS export/file;
  (PVC mode only)                                 or PVC for compat)
          │                                                     │
          └───────────────┬─────────────────────────────────────┘
                          ▼
            ┌──────────────────────────────────┐
            │  Migration proceeds through      │
            │  Snapshotting → Exporting →      │
            │  Transferring → Converting →     │
            │  Importing → Creating →          │
            │  Validating-Target → Ready       │
            └──────────────────────────────────┘

The decisive architectural choice adopted in ADR-0006 (superseding the earlier PVC-only model) is that the transfer medium is storage-backend-agnostic: the disk is staged on an external S3 bucket or NFS export, not on a Kubernetes PVC mounted into provider pods. ADR-0006 captures the rationale:

1. No shared PVC mount required. Source and target providers reach the staging backend independently, which removes the co-location requirement and opens up cross-namespace and cross-cluster topologies.
2. Per-backend credentials, done once. S3 credentials live in a Secret; NFS is network-ACL-gated. Each backend's credentials story is defined once in the CRD, not reimplemented per provider.
3. The disk never crosses CSI. CSI-provisioned PVCs have host-level access constraints; external staging backends do not.

PVC mode is retained for backward compatibility. It is no longer the recommended path.

The phase machine¶

Phases are defined at vmmigration_types.go:363-390. The reconciler dispatches on migration.Status.Phase at vmmigration_controller.go:198-218.

                      ┌─────────┐
                      │ Pending │
                      └────┬────┘
                           │
                           ▼
                   ┌────────────────┐
                   │  Validating    │ ─── any error ──┐
                   └──────┬─────────┘                 │
                          │                           │
                          ▼                           │
                   ┌────────────────┐                 │
                   │ Snapshotting   │ ─── any error ──┤
                   └──────┬─────────┘                 │
                          │                           │
                          ▼                           │
                   ┌────────────────┐                 │
                   │   Exporting    │ ─── any error ──┤
                   └──────┬─────────┘                 │
                          │                           │
                          ▼                           │
                   ┌────────────────┐                 │
                   │  Transferring  │ ─── any error ──┤
                   └──────┬─────────┘                 │
                          │                           │
                          ▼                           │
                   ┌────────────────┐                 │
                   │  Converting    │ ─── any error ──┤
                   └──────┬─────────┘                 │
                          │                           │
                          ▼                           │
                   ┌────────────────┐                 │
                   │   Importing    │ ─── any error ──┤
                   └──────┬─────────┘                 │
                          │                           │
                          ▼                           │
                   ┌────────────────┐                 │
                   │   Creating     │ ─── any error ──┤
                   └──────┬─────────┘                 │
                          │                           │
                          ▼                           ▼
              ┌─────────────────────┐         ┌─────────┐
              │  Validating-Target  │ ────────│ Failed  │
              └──────┬──────────────┘   error └────┬────┘
                     │                             │
                     ▼                             │
                ┌─────────┐                   retry│ (per
                │  Ready  │                   loop │  options.retryPolicy)
                └─────────┘                        │
                                                   ▼
                                             ┌─────────┐
                                             │ Pending │
                                             └─────────┘

Per-phase responsibilities¶

Idempotency¶

Each phase handler is designed to be re-entrant. Concretely, the controller:

• Stores intermediate identifiers in status (snapshot ID, export task ID, import task ID, target VM ID) so a re-reconcile after a manager restart can resume without redoing work.
• For PVC mode, owns the PVC via owner references so the K8s GC reclaims it on VMMigration delete. As of v0.3.8, the ProviderReconciler no longer deletes migration-storage PVCs (#184).
• For S3/NFS mode, the staged object is deleted by the controller's cleanup pass per options.cleanupPolicy, and on finalizer processing.
• Uses K8s annotations on the Provider CRs (virtrigaud.io/migration-pvc, virtrigaud.io/reconcile-trigger) for PVC-mode co-ordination. S3/NFS modes do not annotate providers.

Reconciler instrumentation¶

The reconciler is the most heavily instrumented in the codebase:

• virtrigaud_manager_reconcile_total{name="VMMigration", outcome=…} — every reconcile records one sample. The G3 + K5 double-count fix in PR #106 made this accurate; earlier releases recorded two samples per errored reconcile.
• virtrigaud_manager_reconcile_duration_seconds{name="VMMigration"} — histogram of reconcile latency.
• virtrigaud_errors_total{reason="get_migration", controller="VMMigration"} — per-reason error counter.

Plus the indirect signals from the provider RPCs the migration drives:

• virtrigaud_provider_rpc_requests_total{method=ExportDisk|ImportDisk|SnapshotCreate|…} — per-RPC throughput.
• virtrigaud_circuit_breaker_state{provider_type, provider} — one breaker per Provider CR (G6 / PR #112). A long migration is the most demanding test of CircuitBreaker semantics because a single migration can issue many provider RPCs back to back.

See Resilience for the breaker's behavior in detail.

Provider-side responsibilities¶

The proto contract for the migration-specific RPCs is in proto/provider/v1/provider.proto:

rpc ExportDisk(ExportDiskRequest) returns (ExportDiskResponse);   // proto.proto:296
rpc ImportDisk(ImportDiskRequest) returns (ImportDiskResponse);   // proto.proto:297
rpc GetDiskInfo(GetDiskInfoRequest) returns (GetDiskInfoResponse);// proto.proto:298
rpc SnapshotCreate(SnapshotCreateRequest) returns (SnapshotCreateResponse); // proto.proto:282

Each in-tree provider implements these. The staging backend determines the transport mechanism for ExportDisk/ImportDisk:

Proxmox hybrid data plane¶

The Proxmox provider is a hybrid: a REST API control plane (API token auth) plus an SSH data plane to the PVE node. For S3 migration the node-side qemu-img/qm streams over SSH into the provider pod (the S3 client). The Proxmox Provider Secret must carry both an API token (token_id / token_secret) AND SSH credentials (ssh_user + ssh_password and/or ssh_privatekey) plus a pinned known_hosts. The pod reaches the node by IP (PROVIDER_ENDPOINT = PVE IP) with known_hosts pinned to that IP because in-cluster DNS does not resolve the PVE hostname.

For NFS migration, the PVE node kernel-mounts the export. The NFS AUTH_SYS uid/gid presented by the Proxmox provider is controlled by nfs.uid / nfs.gid via setpriv on the node — set them to the uid/gid that owns the export files.

The staging URL contract¶

The controller and providers exchange staging references via:

• S3: s3://<bucket>/<key> — the provider resolves bucket + endpoint from the migration's S3 config.
• NFS: a path relative to the NFS mount (the kernel-mount or libnfs URL is constructed by the provider from server/export/path).
• PVC (compat): pvc://<pvc-name>/disk.<fmt> — the libvirt provider decodes this at internal/providers/libvirt/server.go:485-489.

For S3 and NFS backends, no data crosses a shared PVC. The source provider writes independently to the staging backend; the target provider reads independently. The controller does not relay bytes.

Why the migration controller is the failure-prone one¶

Per PROJECT_CONTEXT.md, vmmigration_controller.go is the most complex file in the codebase. The reasons are visible from the phase machine:

• It owns external state in two providers and a staging backend. A simple VM reconciler owns external state in one provider. The migration controller has to coordinate a multi-actor dance and survive any of those actors failing.
• Long-running RPCs. Exports and imports for large disks can take hours. The keep-alive tuning in v0.3.51 (GRPC_CONNECTION_AGE_FIX) was a direct response to migrations failing because the SDK server was closing connections at 30s.
• Many failure modes have the same surface. "Phase=Failed, ProviderError" could be a credentials problem, a network partition, an SSH host issue, a disk-space issue on the staging backend, a format-conversion error, a target VM with the same name already existing, etc. The reconciler tries to set helpful status.conditions but operators routinely need to consult the provider logs.

The fieldTesting postmortems (fieldTesting/MIGRATION_*) catalogue the real-world failures the team has shipped fixes for.

ImportedDisk vs ImageRef on the target VM¶

When the migration creates the target VirtualMachine, it does not set spec.imageRef. It sets spec.importedDisk instead:

apiVersion: infra.virtrigaud.io/v1beta1
kind: VirtualMachine
metadata:
  name: my-vm-libvirt
  namespace: applications
  annotations:
    virtrigaud.io/migrated-from: applications/my-vsphere-vm
    virtrigaud.io/migration: applications/my-vm-to-libvirt
    virtrigaud.io/migration-completed: "true"
    virtrigaud.io/migration-completed-at: "2026-05-23T11:14:35Z"
spec:
  providerRef:
    name: libvirt-prod
    namespace: virtrigaud-system
  classRef:
    name: medium-vm
  importedDisk:
    diskID: my-vm-libvirt-disk
    format: qcow2
    source: migration
    migrationRef:
      name: my-vm-to-libvirt
  networks:
    - name: corp-bridge

Key properties:

• importedDisk and imageRef are mutually exclusive on a VirtualMachine spec.
• importedDisk.diskID is the provider-specific disk identifier (e.g. libvirt volume name) — the provider knows how to resolve this to a path.
• importedDisk.migrationRef is purely audit / traceability. The target VM continues to function even after the VMMigration CR is deleted.
• importedDisk.source is one of migration, clone, import, snapshot, manual. The migration controller sets it to migration.

You can also create a VirtualMachine with importedDisk outside the migration flow — useful for adopting an existing disk from a manual import or a clone. Set source: manual in that case.

Roadmap¶

• Cross-cluster migration. v0.3.11 assumes both providers are managed by the same VirtRigaud manager. Cross-cluster (federated) migrations over S3 are architecturally unblocked by ADR-0006 but not yet validated.
• S3 direct mode. transferMode: direct is a roadmap item — the source provider would write directly to S3 without relaying through the pod. Not implemented in v0.3.11; auto resolves to relay.
• Live migration. All v0.3.x migrations are cold (snapshot-based). Live migration within a hypervisor family (vSphere vMotion, libvirt migrate) is a separate feature, not a VMMigration mode.
• Per-Provider CircuitBreaker thresholds. v0.3.11 still uses resilience.DefaultConfig() uniformly. A long migration might want a higher FailureThreshold than a short Describe-driven reconcile.

See also¶

• VM Migration User Guide — practical how-to.
• VM Migration API Reference — every spec field with defaults and validation rules.
• Resilience — CircuitBreaker behavior during long migrations.
• Full CRD Reference — generated CRD documentation.
• docs/adr/0006-storage-backend-agnostic-cross-hypervisor-migration.md — design rationale for the storage-backend-agnostic migration model.
• Field-test postmortems: fieldTesting/MIGRATION_*.md in the main repo.