VM Migration User Guide¶

This page is aligned to VirtRigaud v0.3.11. It is the practical "how do I move a VM?" guide. For the field-by-field CRD reference see API Reference; for the architectural model see VM Migration Guide.

What VirtRigaud v0.3.11 supports¶

The transfer medium is now storage-backend-agnostic (ADR-0006). The disk is staged on either an S3-compatible object store or an NFS export and never traverses a Kubernetes CSI PVC. PVC mode is retained for compatibility only.

Concept in one diagram¶

┌──────────────────────────┐        ┌──────────────────────────┐
│   Source Provider        │        │   Target Provider        │
│   (vsphere / libvirt /   │        │   (vsphere / libvirt /   │
│    proxmox)              │        │    proxmox)              │
│                          │        │                          │
│  export disk             │        │  import disk             │
│  (native format)    ─────┼──────► │  (format convert        │
│                          │        │   via qemu-img)          │
└──────────────────────────┘        └──────────────────────────┘
              │                                   ▲
              └───────────────┬───────────────────┘
                              │ staged on
                              ▼
              ┌───────────────────────────────────┐
              │   Staging backend (one of):       │
              │   • S3 (bucket/prefix)            │
              │   • NFS (export, flat object)     │
              │   • PVC  ← compat-only            │
              └───────────────────────────────────┘
                              │
                        orchestrated by
                              ▼
                ┌────────────────────────────────┐
                │  VMMigration controller        │
                │  (vmmigration_controller.go,   │
                │  phase machine)                │
                └────────────────────────────────┘

Three things happen in sequence, all driven by the migration controller:

1. Validate. Source and target providers are resolved; storage config is validated; the staging object/path is prepared.
2. Export. The source provider writes the disk to the staging backend in its native format.
3. Import. The target provider reads the staged disk, converts format via qemu-img if needed, and creates the target VM.

Quick start: libvirt → vSphere over NFS¶

NFS is the newest validated path (Slice 4, 2026-06-21). This example migrates a VM from a Libvirt host to vSphere, staging the disk on an NFS export.

Prerequisites¶

• A working VirtRigaud install on the cluster (manager + both providers Ready).
• A source VirtualMachine already managed by the Libvirt provider.
• A target Provider CR for vSphere.
• An NFS export accessible by both provider pods and (for Proxmox) by the PVE node. Recommended: one export per tenant (the staged object lives in the export root as a flat file).
• The NFS export owned by a known uid/gid that both provider legs will present. See Storage — NFS for why this matters.

NFS VMMigration¶

apiVersion: infra.virtrigaud.io/v1beta1
kind: VMMigration
metadata:
  name: app-libvirt-to-vsphere-nfs
  namespace: default
spec:
  source:
    vmRef:
      name: app-on-libvirt
    createSnapshot: false
    powerOffBeforeMigration: true   # required for vSphere target; ESXi locks running disks
  target:
    name: app-on-vsphere
    providerRef:
      name: vsphere-prod
    classRef:
      name: standard-2cpu-4gb
    networks:
      - name: primary
        networkRef:
          name: vm-network
    powerOn: false
  storage:
    type: nfs
    nfs:
      server: 172.16.56.13
      export: /export/virtrigaud
      uid: 1000     # uid that owns the export files — all legs must match
      gid: 1000     # gid that owns the export files — all legs must match
    transferMode: auto    # informational for NFS; qemu-img handles transport
  options:
    cleanupPolicy: OnSuccess
    verifyChecksums: true
    timeout: 1h

Apply and watch:

kubectl apply -f app-libvirt-to-vsphere-nfs.yaml

# Watch phase progression
kubectl get vmmigration app-libvirt-to-vsphere-nfs -w

# Detailed status (conditions, events, storageInfo)
kubectl describe vmmigration app-libvirt-to-vsphere-nfs

A successful run progresses Pending → Validating → Snapshotting → Exporting → Transferring → Converting → Importing → Creating → Validating-Target → Ready. End-to-end time depends on disk size + network throughput to the NFS server.

Quick start: vSphere → Libvirt over S3¶

S3 is the relay-based path (Slices 1–3). The provider pod acts as the S3 client; bytes flow: host → pod → S3 bucket → pod → host. transferMode: auto resolves to relay (the only implemented mode; direct is roadmap).

Prerequisites¶

• An S3-compatible bucket (AWS S3, MinIO, Ceph RGW, rustfs, etc.).
• A Secret in the migration namespace carrying S3 credentials.
• Source VM managed by vSphere, target Provider CR for Libvirt.

S3 credentials Secret¶

apiVersion: v1
kind: Secret
metadata:
  name: s3-migration-credentials
  namespace: default
type: Opaque
stringData:
  accessKeyID: <your-access-key>
  secretAccessKey: <your-secret-key>
  # sessionToken: <optional, for temporary credentials>

S3 VMMigration¶

apiVersion: infra.virtrigaud.io/v1beta1
kind: VMMigration
metadata:
  name: my-vm-to-libvirt
  namespace: default
spec:
  source:
    vmRef:
      name: my-vsphere-vm
    createSnapshot: true
    powerOffBeforeMigration: true
    deleteAfterMigration: false   # keep source for rollback

  target:
    name: my-vm-libvirt
    providerRef:
      name: libvirt-prod
      namespace: virtrigaud-system
    classRef:
      name: medium-vm
    networks:
      - name: corp-bridge
    powerOn: false

  options:
    diskFormat: qcow2
    verifyChecksums: true
    timeout: 4h

  storage:
    type: s3
    transferMode: relay       # relay (implemented); auto → relay
    s3:
      bucket: virtrigaud
      endpoint: http://minio.example:9000   # omit for AWS S3
      region: us-east-1
      usePathStyle: true                    # true for MinIO/Ceph/rustfs; false for AWS
      credentialsSecretRef:
        name: s3-migration-credentials

Apply and watch:

kubectl apply -f my-vm-to-libvirt.yaml
kubectl get vmmigration my-vm-to-libvirt -w

Storage¶

S3 staging¶

The S3 backend is the primary staging path for fleet migrations. The provider pod is the S3 client — no CSI PVC is involved. The source exports the disk in its native format; the target owns format conversion (vmdk → qcow2, etc.) on import.

Key configuration fields under spec.storage.s3:

The S3 server hostname is SSRF-checked against the manager flag --migration-storage-allowed-hosts. Loopback and link-local targets are always rejected.

NFS staging (new in Slice 4)¶

NFS staging is the second validated backend. The disk is moved using qemu-img's native NFS support — not a PVC, and not a pod relay. Per-provider transport detail:

The uid/gid requirement¶

Flat-file layout¶

The staged object is a single flat file in the export root:

<export root>/vmmigrations-<namespace>-<name>-<stage>.qcow2

NFS cannot auto-create directories the way S3 key prefixes do. A nested spec.storage.nfs.path (subpath below the export) is supported but the flat file must exist at that path — the controller will not mkdir it. For simplicity, omit path and use the export root with one export per tenant to avoid collisions.

Integrity model¶

qemu-img over NFS emits no in-stream byte checksum, so the dual-checksum model (source SHA256 vs target SHA256) degrades to a target-side qemu-img check / post-import uuid query rather than a full SHA256 match. options.verifyChecksums: true is still recommended — it triggers the target-side check.

Security and hardening (ADR-0006 C6)¶

Key configuration fields under spec.storage.nfs:

The NFS server hostname is SSRF-checked by the same manager flag as S3: --migration-storage-allowed-hosts. Loopback and link-local targets are always rejected.

PVC staging (compatibility only)¶

PVC mode (storage.type: pvc) mounts a ReadWriteMany PVC into both provider pods and is retained for clusters that already use it. It is no longer the recommended path for new migrations.

Sizing guidance (still relevant for PVC users): size the PVC at 1.5×–2× the source disk size to accommodate both the export and any sibling conversion file (qemu-img convert writes a sibling before deleting the original).

See API Reference — storage.pvc for field-level detail.

What happens to the source VM¶

What happens to the target VM¶

On phase=Ready:

• The target VirtualMachine resource exists and is independent of the VMMigration CR. Deleting the VMMigration does not delete the target.
• The target VM is annotated virtrigaud.io/migration-completed=true and virtrigaud.io/migration-completed-at=<RFC3339>.
• The target's spec references the imported disk via importedDisk rather than imageRef. This is the production-ready VM you can manage, scale, snapshot, etc. independently.

On phase=Failed:

• A partially-created target VM is cleaned up by the finalizer (vmmigration_controller.go:1257). The intermediate storage and any provider snapshots are also cleaned up per options.cleanupPolicy.

Failure modes and recovery¶

The migration controller is the most complex reconciler in the codebase (~2074 LOC, vmmigration_controller.go). Known failure classes:

For a recent postmortem catalogue:

ls fieldTesting/MIGRATION_*.md

Common operator-visible failures¶

Forcing a retry without waiting¶

Migrations retry per options.retryPolicy.retryDelay with exponential backoff. If you have fixed the root cause and want to retry immediately:

# Trigger an immediate reconcile by touching an annotation
kubectl annotate vmmigration my-vm-to-libvirt \
    virtrigaud.io/reconcile-trigger="$(date +%s)" --overwrite

Reading migration progress¶

# Phase + percentage
kubectl get vmmigration my-vm-to-libvirt \
  -o jsonpath='{.status.phase}{"\t"}{.status.progress.percentage}{"\n"}'

# Conditions
kubectl get vmmigration my-vm-to-libvirt -o yaml | yq .status.conditions

# Live progress watch (1s interval, jq-formatted)
watch -n 1 'kubectl get vmmigration my-vm-to-libvirt \
  -o json | jq "{phase:.status.phase, pct:.status.progress.percentage, msg:.status.message}"'

Cleanup policies¶

options.cleanupPolicy controls what happens to intermediate state after the migration finishes:

For S3: "cleaned up" means the staged object is deleted from the bucket. For NFS: "cleaned up" means the flat file is deleted from the export. For PVC: the PVC is owned by the VMMigration CR; kubectl delete vmmigration <name> garbage-collects it regardless of cleanupPolicy. The policy controls the in-controller cleanup pass, not the K8s-GC fallback.

Performance guidance¶

Multi-disk VMs¶

The controller handles multi-disk VMs by processing each disk through the export → transfer → import pipeline. You do not need to specify each disk explicitly in spec.target.disks[] for the migration to handle them — that field is only for overriding target-side disk parameters (size, storage hint).

For NFS migrations the staged object is a single flat file in the export root, named vmmigrations-<ns>-<name>-<stage>.qcow2 (the <stage> segment, e.g. export, distinguishes the staging phase). Keep one export per tenant so concurrent migrations do not collide on the export root.

Cross-namespace migrations¶

spec:
  source:
    vmRef:
      name: vm-in-prod
    providerRef:
      name: vsphere-provider
      namespace: infrastructure       # cross-namespace OK
  target:
    name: vm-in-staging
    namespace: staging                 # target lives in 'staging'
    providerRef:
      name: libvirt-provider
      namespace: infrastructure

The migration resource itself can live in either the source or target namespace; the target VM is created in spec.target.namespace regardless. For PVC mode, the intermediate PVC is created in the same namespace as the VMMigration.

Examples directory¶

The in-tree example YAMLs are under examples/migration/ in the main repository. The recommended starting points for v0.3.11:

• vmmigration-nfs.yaml — NFS staging (Slice 4, validated)
• vmmigration-s3.yaml — S3 staging (Slices 1–3, validated)

PVC examples (vmmigration-basic.yaml, vmmigration-cross-namespace.yaml) are retained for compatibility but are no longer the recommended path.

See also¶

• VM Migration API Reference — every field, every default, every validation rule.
• VM Migration Guide — architectural deep dive (phase machine, controller orchestration, storage sequencing).
• Resilience — what happens when the source or target provider flaps during a long migration.
• Libvirt Host Preparation — host setup for Libvirt migrations.
• ADR-0006 — design rationale for storage-backend-agnostic migration.