mTLS for Manager <-> Provider gRPC¶

This page describes how to enable and operate mTLS for manager↔provider gRPC in v0.3.8: the Provider CR tls block, how the manager and provider enforce it, the SAN allow-list, the secure-by-default/fail-closed behaviour, certificate rotation and its limitation, the Helm providerTLS block for chart-templated providers, the TLSConfigured status condition, and a worked cert-manager example for producing the TLS Secret.

Trust model¶

VirtRigaud uses a single CA per install (ADR-0003 decision #5):

• The manager holds a client certificate + key plus the CA bundle.
• Each provider holds a server certificate + key plus the same CA bundle.
• All provider pods trust the same CA. One VirtRigaud install is one administrative boundary.

The TLS Secret is provisioned by the operator. VirtRigaud reads a Kubernetes Secret containing tls.crt / tls.key / ca.crt; how those bytes are minted — manual openssl, an internal PKI pipeline, Vault, External Secrets, or cert-manager — is the operator's choice. The Helm chart ships no cert-manager Certificate or issuer scaffolding; it only references a Secret you provide. A worked cert-manager example for producing that Secret is given below.

The Provider CR tls block¶

mTLS for a controller-managed provider is configured on the Provider CR under spec.runtime.service.tls (ProviderTLSSpec):

apiVersion: infra.virtrigaud.io/v1beta1
kind: Provider
metadata:
  name: provider-vsphere-prod
  namespace: virtrigaud-system
spec:
  type: vsphere
  endpoint: https://vcenter.internal.example.com
  credentialSecretRef:
    name: vsphere-prod-credentials
  runtime:
    image: ghcr.io/projectbeskar/virtrigaud/provider-vsphere:v0.3.8
    service:
      tls:
        enabled: true                       # default true
        secretRef:
          name: provider-vsphere-prod-tls   # Secret with tls.crt / tls.key / ca.crt
        insecureSkipVerify: false           # default false — dev-only

The referenced Secret may be either a kubernetes.io/tls-typed Secret (with an extra ca.crt key) or a plain Opaque Secret carrying all three keys explicitly. Both shapes are accepted, so a cert-manager-produced Secret needs no translation.

How enforcement works end-to-end¶

Provider CR (spec.runtime.service.tls.enabled=true, secretRef → Secret)
        │
        ▼
ProviderReconciler                                  internal/controller/provider_controller.go
  evaluateTLSPosture() sets the TLSConfigured Condition,
  mounts the Secret on the provider pod at /etc/virtrigaud/tls,
  and passes VIRTRIGAUD_PROVIDER_ALLOWED_SANS to the provider
        │
        ├──────────────► Manager-side gRPC client          internal/runtime/remote/resolver.go
        │                  buildTLSConfig() loads tls.crt/tls.key/ca.crt
        │                  from the Secret and builds a *tls.Config:
        │                    MinVersion = TLS 1.3
        │                    RootCAs    = ca.crt
        │                    Certificates = tls.crt + tls.key (client cert)
        │                    ServerName = provider Service FQDN
        │
        └──────────────► Provider-side gRPC server          sdk/provider/server, middleware
                           serves mTLS:
                             MinVersion = TLS 1.3
                             RequireAndVerifyClientCert
                             ClientCAs  = ca.crt
                           validateTLSPeer() enforces the SAN allow-list

Manager side¶

The manager's resolver (internal/runtime/remote/resolver.go) builds the gRPC client TLS config from the Secret:

• MinVersion = TLS 1.3 (the project floor).
• RootCAs from ca.crt — used to verify the provider's server certificate.
• Certificates from tls.crt + tls.key — the manager's client certificate.
• ServerName anchored to the provider Service FQDN (virtrigaud-provider-<namespace>-<name>.<namespace>.svc.cluster.local), so SNI matches a deterministic SAN you mint into the provider server cert.

Provider side¶

The provider gRPC server (SDK sdk/provider/...) serves mTLS with:

• MinVersion = TLS 1.3.
• RequireAndVerifyClientCert — the TLS stack itself must validate the manager's client cert against the CA bundle loaded into ClientCAs.
• A SAN allow-list enforced by validateTLSPeer (next section).

TLS material is mounted on the provider pod at /etc/virtrigaud/tls (tls.crt, tls.key, ca.crt).

The SAN allow-list¶

On top of TLS chain verification, the provider applies an authorization gate via validateTLSPeer (ADR-0003 decision #5):

The allow-list is delivered to the provider via the VIRTRIGAUD_PROVIDER_ALLOWED_SANS environment variable (comma-joined), set by the controller for controller-managed providers and by the chart for static providers.

Secure-by-default and fail-closed¶

The provider is fail-closed (behaviour shipped in v0.3.7, unchanged in v0.3.8). A provider pod that finds no TLS material at /etc/virtrigaud/tls and does not have VIRTRIGAUD_PROVIDER_INSECURE=true set will hard-exit on startup rather than silently fall back to plaintext, with this error:

TLS material missing at /etc/virtrigaud/tls and VIRTRIGAUD_PROVIDER_INSECURE is not set to "true"; either provision /etc/virtrigaud/tls/{tls.crt,tls.key,ca.crt} or set VIRTRIGAUD_PROVIDER_INSECURE=true to opt into plaintext (audit-flagged)

This is deliberate: a misconfigured upgrade refuses to start rather than regressing to plaintext.

Escape hatches (plaintext fallback)¶

There is no global --insecure-no-tls-providers flag. Plaintext is opt-in per Provider:

• Controller-managed providers: set spec.runtime.service.tls.enabled=false on the Provider CR. The controller then sets VIRTRIGAUD_PROVIDER_INSECURE=true on the provider pod so it boots in audit-flagged plaintext instead of crash-looping. The manager logs a WARNING and the TLSConfigured condition reads False / ExplicitlyDisabled.
• Chart-templated (static) providers: set providerTLS.insecure=true (only honoured when providerTLS.secretName is empty).

In both cases the provider starts plaintext with a loud audit WARNING. Compensating controls (NetworkPolicy + encrypted CNI) remain the operator's responsibility for those Providers — see Network Policies.

The TLSConfigured status condition¶

The controller surfaces a TLSConfigured condition on every Provider CR so auditors can verify TLS posture with kubectl get providers rather than a packet capture:

kubectl get provider provider-vsphere-prod -n virtrigaud-system \
  -o jsonpath='{.status.conditions[?(@.type=="TLSConfigured")]}{"\n"}'

Certificate rotation¶

Leaf certificate/key rotation is hot-reloaded without a pod restart. When TLS material is present, the provider enables a controller-runtime certwatcher on the mounted leaf cert/key by default; Kubernetes' Secret-to-Pod sync (~60s) updates the mounted files and the watcher picks up the new bytes.

To rotate the leaf certificate:

1. Re-mint the leaf cert/key (keeping the same CA).
2. kubectl apply the updated Secret.
3. Watch the TLSConfigured condition stay green; no pod restart needed.

Helm providerTLS block (chart-templated providers only)¶

The providerTLS block in values.yaml governs chart-templated / static provider Deployments only. Controller-managed providers read their TLS posture from the Provider CR's tls block, not from these values.

providerTLS:
  # Name of an externally-provisioned Kubernetes Secret (typically
  # kubernetes.io/tls) containing tls.crt, tls.key, and ca.crt. Empty
  # by default — operators opt in by setting it.
  secretName: ""

  # SAN/CN values the provider accepts from the manager's client cert.
  # Maps to VIRTRIGAUD_PROVIDER_ALLOWED_SANS (comma-joined). Empty
  # (default) is permissive: any cert signed by the configured CA.
  allowedSANs: []

  # Explicit plaintext escape hatch. Maps to VIRTRIGAUD_PROVIDER_INSECURE.
  # Only honoured when secretName is empty. false by default.
  insecure: false

Chart behaviour:

The chart ships no cert-manager Certificate or issuer template. Provision the Secret yourself; cert-manager is a fine way to produce it (see below).

Producing the Secret with cert-manager¶

cert-manager is a convenient way to produce the tls.crt / tls.key / ca.crt Secret. The chart does not create these resources — you apply them yourself. Root the issuer in your organisation's private PKI, not a public CA.

# 1. A private CA issuer (or reuse your org's existing ClusterIssuer).
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: virtrigaud-ca-issuer
  namespace: virtrigaud-system
spec:
  ca:
    secretName: virtrigaud-ca-keypair   # your CA cert + key, provisioned out of band
---
# 2. A Certificate that produces the provider's server Secret.
#    cert-manager writes tls.crt / tls.key, and (with the ca.crt option)
#    the CA bundle into the same Secret — exactly the three keys VirtRigaud reads.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: provider-vsphere-prod-tls
  namespace: virtrigaud-system
spec:
  secretName: provider-vsphere-prod-tls        # referenced by the Provider CR's secretRef
  issuerRef:
    name: virtrigaud-ca-issuer
    kind: Issuer
  commonName: provider-vsphere-prod
  dnsNames:
    - virtrigaud-provider-virtrigaud-system-provider-vsphere-prod.virtrigaud-system.svc.cluster.local
  duration: 2160h     # 90 days
  renewBefore: 720h   # rotate 30 days early; leaf hot-reloads, no restart

For the manual openssl-only recipe (CA, manager client cert, and N provider server certs) and the operator runbook, see the in-repo operator security guide and the Operations → Security page.

Certificate management recommendations¶

For a regulated deployment:

• Use a private CA rooted in your organisation's PKI (a cert-manager Issuer/ClusterIssuer over your CA, or your existing PKI pipeline). Do not use a public CA.
• Use short-lived leaf certs (e.g. 90-day duration, 30-day renewBefore) and rely on leaf hot-reload — no pod restart is needed for leaf rotation.
• Schedule CA rotations as rolling provider restarts, since the CA bundle does not hot-reload. Keep CA lifetimes long enough that this is infrequent and planned.
• Populate AllowedSANs when more than one identity could present a cert signed by the CA, so a provider only accepts its own manager. Leave it empty only when the CA's signing scope is exactly your manager's identity.
• Keep TLS Secrets distinct from credential Secrets — they use different keys and different mount paths (/etc/virtrigaud/tls vs /etc/virtrigaud/credentials) and should be managed independently.

Breaking change: v0.3.6 → v0.3.7 (historical)¶

See also¶

• Operations -> Security — overall v0.3.8 security posture, including the resolved-gap inventory.
• Bearer Token Authentication — the other half of the gRPC-channel and hypervisor-credential auth story.
• Network Policies — defence-in-depth, and the required compensating control for any Provider you run with tls.enabled=false.
• Resilience — per-Provider CircuitBreaker limits the blast radius of a degraded or spoofed provider.