Proxmox VE Provider¶

The Proxmox provider manages VMs on Proxmox Virtual Environment (PVE) via the native PVE REST API. It is the newest production-grade provider in the VirtRigaud tree and is currently maturing toward general availability.

This page is aligned to VirtRigaud v0.3.11. Capability claims trace back to the provider's GetCapabilities builder in internal/providers/proxmox/capabilities.go and the REST client in internal/providers/proxmox/pveapi/.

Status¶

The Proxmox provider is listed as Production-beta in the capability matrix. It implements the full RPC surface, but ConsoleURL is web-UI-deep-link only (not a standalone VNC ticket — see below), and the production-burn-in time is shorter than for vSphere/libvirt.

Capabilities at a glance¶

Built via capabilities.NewBuilder() in internal/providers/proxmox/capabilities.go:

The "ConsoleURL" claim deserves nuance — see Console access below.

For the full cross-provider matrix and resilience / observability story:

• Provider capabilities matrix
• Operations — Resilience — CircuitBreaker (G6 / v0.3.6) wraps every Proxmox RPC.
• Operations — Observability

RPC support¶

• Validate — client.FindNode() against the cluster.
• Create — POST /api2/json/nodes/{node}/qemu or clone from a template VMID. Cloud-init is attached as IDE2 cloudinit drive.
• Delete — DELETE /api2/json/nodes/{node}/qemu/{vmid}.
• Power — POST /api2/json/nodes/{node}/qemu/{vmid}/status/{start|stop|reset|shutdown}.
• Describe — VM state + guest-agent-derived IPs + console URL deep link + raw PVE provider details.
• Reconfigure — PUT /api2/json/nodes/{node}/qemu/{vmid}/config for hot-plug CPU/memory.
• SnapshotCreate / SnapshotDelete / SnapshotRevert — /snapshot subtree of the VMID; honours includeMemory via vmstate=1.
• CloneCreate — POST /api2/json/nodes/{node}/qemu/{vmid}/clone. Both full=0 (linked) and full=1 (full clone) supported.
• TaskStatus — polls the PVE task UPID (internal/providers/proxmox/pveapi/). Counts toward virtrigaud_provider_tasks_inflight.
• ConsoleUrl — web-UI deep link (see below).

Prerequisites¶

• Proxmox VE 7.0 or later, reachable from the manager / provider pod on port 8006/HTTPS.
• Either an API token or a username + password.
• Storage and bridges configured on the PVE nodes — the provider does not create them.

Authentication¶

The Proxmox provider reads credentials from the Secret mounted at /etc/virtrigaud/credentials inside the provider pod, falling back to environment variables (internal/providers/proxmox/server.go:60-112). Both API tokens and username/password are supported; tokens are strongly preferred for production.

The Secret keys are read as files; the key names matter:

If a token_id + token_secret pair is present, the provider uses token auth. Otherwise it falls back to username/password and acquires a session ticket. Mixed credentials in the same Secret are tolerated (token wins).

API token (recommended)¶

Create the token in PVE:

# As root@pam on a PVE node
pveum user add virtrigaud@pve --comment "VirtRigaud"
pveum user token add virtrigaud@pve vrtg-token --privsep 1
# Grant permissions (see "Minimum permissions" below)
pveum acl modify / --users virtrigaud@pve --roles PVEVMAdmin
# Or use a tighter custom role:
# pveum role add VirtRigaud --privs "VM.Allocate,VM.Audit,VM.Config.CPU,VM.Config.Memory,VM.Config.Disk,VM.Config.Network,VM.Config.Options,VM.Monitor,VM.PowerMgmt,VM.Snapshot,VM.Clone,Datastore.AllocateSpace,Datastore.Audit"
# pveum acl modify / --users virtrigaud@pve --roles VirtRigaud

Provider CR + Secret:

apiVersion: v1
kind: Secret
metadata:
  name: proxmox-credentials
  namespace: virtrigaud-system
type: Opaque
stringData:
  token_id: "virtrigaud@pve!vrtg-token"
  token_secret: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
---
apiVersion: infra.virtrigaud.io/v1beta1
kind: Provider
metadata:
  name: proxmox-cluster
  namespace: virtrigaud-system
spec:
  type: proxmox
  endpoint: "https://pve.example.com:8006"
  credentialSecretRef:
    name: proxmox-credentials
  insecureSkipVerify: false
  runtime:
    mode: Remote
    image: "ghcr.io/projectbeskar/virtrigaud/provider-proxmox:v0.3.11"
    service:
      port: 9443
      tls:
        enabled: true
        secretRef:
          name: provider-proxmox-tls
        insecureSkipVerify: false

Username / password¶

apiVersion: v1
kind: Secret
metadata:
  name: proxmox-credentials
type: Opaque
stringData:
  username: "virtrigaud@pve"
  password: "REPLACE_ME"

TLS / mTLS (v0.3.7+)¶

Starting in v0.3.7, the manager enforces that every Provider CR has a spec.runtime.service.tls block. A Provider without this block fails to reconcile and its status will show TLSConfigured=False, Reason=TLSBlockMissing — no Deployment is created.

For full mTLS details see Security — mTLS.

spec.runtime.service.tls fields¶

TLS material mounts at /etc/virtrigaud/tls inside the provider pod. Both manager and provider pin TLS 1.3. The TLSConfigured status condition reasons are TLSBlockMissing, ExplicitlyDisabled, SecretRefMissing, and Enabled.

Minimum permissions¶

For an API token using a custom role:

Apply at the path you scope (typically /).

Endpoint formats¶

The CRD validates HTTPS scheme.

Node selection¶

By default the provider selects a node automatically. To pin to specific nodes (e.g., for HA constraints or licensing), set PROVIDER_NODE_SELECTOR:

spec:
  runtime:
    env:
      - name: PROVIDER_NODE_SELECTOR
        value: "pve-node-1,pve-node-2"

Storage¶

PVE storage is referenced by ID (e.g., local-lvm, local-zfs, cephfs-pool). The qcow2 disk type requires file-backed storage; raw works on block-backed storage (LVM, ZFS, Ceph). Mismatches return 400 Bad Request from the PVE API — see Troubleshooting.

In the VMImage, point at the storage by ID:

apiVersion: infra.virtrigaud.io/v1beta1
kind: VMImage
metadata:
  name: ubuntu-22-template
spec:
  source:
    proxmox:
      templateName: "ubuntu-22-template"
      storage: "local-lvm"

Networking¶

PVE networking maps to Linux bridges (vmbr0, vmbr1, ...) with optional 802.1Q VLAN tags:

apiVersion: infra.virtrigaud.io/v1beta1
kind: VMNetworkAttachment
metadata:
  name: lan-vmbr0
spec:
  network:
    proxmox:
      bridge: vmbr0
      model: virtio
  ipAllocation:
    type: DHCP
---
apiVersion: infra.virtrigaud.io/v1beta1
kind: VMNetworkAttachment
metadata:
  name: dmz-vlan100
spec:
  network:
    proxmox:
      bridge: vmbr1
      vlanTag: 100
      model: virtio
  ipAllocation:
    type: Static
    address: "10.0.100.50/24"

Multi-NIC, mixed DHCP/static, and per-NIC MAC pinning are all supported via cloud-init.

Cloud-init (cicustom + per-NIC ipconfig)¶

PVE has a native cloud-init implementation. VirtRigaud uses two complementary paths:

1. cicustom — the provider uploads a custom user-data snippet to PVE storage and references it on the VM config (cicustom: user=local:snippets/...).
2. ipconfig0, ipconfig1, ... — for static IP configuration, generated from the VMNetworkAttachment.ipAllocation field. These are PVE's per-NIC cloud-init network config.

The IDE2 cloudinit drive is attached automatically at VM creation; the VM picks it up at first boot.

spec:
  userData:
    cloudInit:
      inline: |
        #cloud-config
        hostname: web-server
        users:
          - name: ubuntu
            sudo: ALL=(ALL) NOPASSWD:ALL
            ssh_authorized_keys:
              - "ssh-ed25519 AAAA..."
        packages:
          - qemu-guest-agent
        runcmd:
          - systemctl enable --now qemu-guest-agent

VM example¶

apiVersion: infra.virtrigaud.io/v1beta1
kind: VMClass
metadata:
  name: small
spec:
  cpu: 2
  memory: "4Gi"
  firmware: UEFI
  diskDefaults:
    type: qcow2
    size: "20Gi"
---
apiVersion: infra.virtrigaud.io/v1beta1
kind: VMNetworkAttachment
metadata:
  name: lan-vmbr0
spec:
  network:
    proxmox:
      bridge: vmbr0
      model: virtio
  ipAllocation:
    type: DHCP
---
apiVersion: infra.virtrigaud.io/v1beta1
kind: VirtualMachine
metadata:
  name: web-server
spec:
  providerRef:
    name: proxmox-cluster
  classRef:
    name: small
  imageRef:
    name: ubuntu-22-template
  powerState: On
  networks:
    - name: lan
      networkRef:
        name: lan-vmbr0
  disks:
    - name: data
      sizeGiB: 40
      type: qcow2
  userData:
    cloudInit:
      inline: |
        #cloud-config
        hostname: web-server
        packages: [nginx, qemu-guest-agent]
        runcmd:
          - systemctl enable --now nginx qemu-guest-agent

Reconfiguration¶

The Proxmox provider supports online (hot-plug) reconfiguration via PUT /api2/json/nodes/{node}/qemu/{vmid}/config:

Snapshots — including memory state¶

Proxmox is the only provider that genuinely supports memory snapshots. The provider passes vmstate=1 to the snapshot API when VMSnapshot.spec.includeMemory: true (internal/providers/proxmox/pveapi/client.go:794-828):

apiVersion: infra.virtrigaud.io/v1beta1
kind: VMSnapshot
metadata:
  name: pre-upgrade
spec:
  vmRef:
    name: web-server
  description: "Snapshot before upgrade"
  includeMemory: true

Memory snapshots are slower (RAM contents are streamed to storage) but allow point-in-time restore of the running state including in-flight transactions.

Cloning¶

Both full and linked clones are supported natively by PVE. The provider sets full=0 for linked clones, full=1 for full clones (default).

apiVersion: infra.virtrigaud.io/v1beta1
kind: VMClone
metadata:
  name: web-server-02
spec:
  source:
    vmRef:
      name: template-vm
  target:
    name: web-server-02
  options:
    type: LinkedClone     # FullClone for an independent copy
    powerOn: true

Linked clones share the parent's disk on copy-on-write storage (ZFS / Ceph / qcow2 with backing); the parent must remain available.

Cross-hypervisor migration¶

Proxmox participates fully in VirtRigaud's storage-backend-agnostic, any-direction cross-hypervisor migration (ADR-0006, #236). It is validated end-to-end as both a source and a target, in both directions, against vSphere and libvirt, over both staging backends — S3 (ADR-0006 Slice 3) and NFS (Slice 4). The disk is staged on the backend and moved with qemu-img; it never traverses a CSI PVC.

The SSH data plane (required for migration)¶

The Proxmox provider is a hybrid: an API-token control plane (the REST API) plus an SSH data plane to the PVE node. The disk is read/written on the node with node-side tools (qemu-img, qm, pvesm), so migration requires the provider pod to reach the PVE node over SSH. The pod reaches the node by IP (PROVIDER_ENDPOINT = the PVE node IP), with known_hosts pinned to that IP, because in-cluster DNS typically does not resolve the PVE hostname.

This means the Proxmox Provider Secret must carry SSH credentials in addition to the API token when you intend to migrate. Add to the Secret described in Authentication:

Without these, control-plane operations (Create/Power/Snapshot/…) still work, but a migration with a Proxmox endpoint fails when it tries to open the SSH data plane.

How each backend moves the disk¶

• S3 — the node-side qemu-img convert -U -O qcow2 streams the disk over SSH into the provider pod, which is the S3 client (the universal relay mode; the disk bytes flow node → pod → S3 → pod → node). -U lets the export read a disk whose VM may still be running. On import, the provider streams the S3 object to a node temp file, runs qemu-img check, creates a diskless shell VM, and attaches the disk with qm importdisk + qm set. The import target storage is operator-configurable (PROVIDER_DEFAULT_STORAGE / PVE_DEFAULT_STORAGE), not hardcoded local-lvm — a directory store is a valid target.

• NFS — uses a kernel NFS mount on the node, not libnfs. pve-qemu-kvm is built without the libnfs block driver, so qemu-img rejects an nfs:// URL with Unknown protocol 'nfs'. The Proxmox provider therefore mounts the export with the node's kernel NFS client (exactly how PVE's first-class NFS storage mounts) and runs qemu-img against the mount. Because the export must be written as the migration's nfs.uid/nfs.gid — but qemu-img must run as root to read the root-owned source — the export is two-stage: qemu-img (as root) flattens the source to a local temp, then the temp is copied onto the export as nfs.uid/nfs.gid via setpriv.

Async task tracking¶

Every long-running PVE operation returns a UPID (Unique Process ID). The provider returns this as a proto TaskRef; the manager polls TaskStatus (which calls the PVE /api2/json/nodes/{node}/tasks/{upid}/status endpoint) with jittered backoff until terminal.

Since v0.3.6, in-flight tasks are tracked by the virtrigaud_provider_tasks_inflight{provider_type="proxmox", provider="<name>"} gauge (G7.3). See Observability.

Console access¶

Describe populates status.consoleURL with a deep link to the PVE web UI's console view (internal/providers/proxmox/server.go:563-569):

https://pve.example.com:8006/#v1:0:=qemu/{vmid}:4:5:=console

This is a web-UI deep link, not a standalone VNC ticket. To use it:

1. Open it in a browser.
2. The PVE UI prompts for login (or single sign-on if you have it).
3. You land on the VM's noVNC console.

A first-class VNC ticket endpoint (where the provider would acquire a one-shot ticket via POST /api2/json/nodes/{node}/qemu/{vmid}/vncproxy and embed it in the URL) is planned for a future release — see the "Proxmox" section in the capability matrix roadmap. The current matrix marks this cell ⚠️ to reflect the gap.

Troubleshooting¶

CircuitBreaker open for the proxmox provider¶

virtrigaud_circuit_breaker_state{provider_type="proxmox", provider="proxmox-cluster"} == 2

The CircuitBreaker (G6 / v0.3.6) has fast-failed enough RPCs to PVE to open the breaker. Common causes:

• Expired or rotated API token.
• TLS certificate validation failure.
• PVE node down / pveproxy service not running.
• API rate-limit (PVE applies per-IP rate limits; verify by curl-ing /version from inside the provider pod).

The breaker self-recovers once the underlying issue is fixed; no manual reset needed.

storage 'qcow2' requires file-backed storage¶

You set disks.type: qcow2 against a non-file storage type (LVM, ZFS, Ceph RBD). Fix one of two ways:

• Switch the disk type to raw.
• Move to a file-backed storage (local, nfs, cephfs).

401 Unauthorized from PVE¶

For token auth, the Authorization header must be PVEAPIToken=user@realm!tokenid=secret. Test from a debug pod:

curl -k "https://pve.example.com:8006/api2/json/version" \
  -H "Authorization: PVEAPIToken=virtrigaud@pve!vrtg-token=$SECRET"

If that returns 401, the token is bad or the user has no read permission at / — see Minimum permissions.

VMID collision in a cluster¶

The provider derives the VMID from a hash of the VM name + a timestamp suffix. In rare cases (rapid create/delete cycles) collisions can occur. If you see VM <vmid> already exists, retry — the next attempt will pick a fresh VMID.

Guest agent IPs missing¶

# On the PVE node:
qm config <vmid> | grep agent
# Should show: agent: enabled=1

If agent: is absent or 0, set it via reconfigure. Inside the guest, install + enable qemu-guest-agent. The provider's Describe queries /api2/json/nodes/{node}/qemu/{vmid}/agent/network-get-interfaces and falls back gracefully when the agent is absent, but status.ips will be empty.

Validation walkthrough¶

# 1. Network reachability
curl -k https://pve.example.com:8006/api2/json/version

# 2. Token works
curl -k "https://pve.example.com:8006/api2/json/nodes" \
  -H "Authorization: PVEAPIToken=virtrigaud@pve!vrtg-token=$SECRET"

# 3. Cluster status
curl -k "https://pve.example.com:8006/api2/json/cluster/status" \
  -H "Authorization: PVEAPIToken=virtrigaud@pve!vrtg-token=$SECRET"

Debug logging¶

spec:
  runtime:
    logLevel: debug

This causes the PVE REST client to log every request/response pair. Tokens / passwords are redacted (internal/providers/proxmox/pveapi/); the raw body is included so you can diff against the PVE API docs.

Performance tips¶

• Token auth over password auth: avoids the session-ticket renewal cycle.
• qcow2 + ZFS only when you need it: ZFS does its own snapshotting; qcow2 on ZFS doubles the overhead.
• Pin the provider pod to a host with fast connectivity to the PVE cluster: the API is chatty (one call per VM operation, plus task polling).
• PROVIDER_NODE_SELECTOR for HA pinning: keeps related VMs on the same node where shared storage is local.

API reference¶

• Full CRD field reference: Generated CRD docs.
• Provider gRPC contract: Generated gRPC docs.
• Capability matrix (all providers): Capabilities.
• Proxmox API reference: Proxmox VE API.

Support¶

• Documentation: VirtRigaud Docs
• Issues: GitHub Issues — label provider/proxmox