Skip to main content

vault-unseal

Overview​

vault-unseal is a sidecar container that automatically unseals Vault whenever a pod starts, eliminating manual unsealing after restarts.

The Problem​

When a Vault pod restarts, the Vault process is sealed on startup. Without automatic unsealing:

  • Every Vault pod restart requires manual vault unseal commands
  • Vault restarts during deployments, node maintenance, or failures cascade into cluster-wide secret access failures

How It Works​

The sidecar solves this by operating as a simple state machine:

  1. Wait for unseal key file: Polls until the file appears at the configured path
  2. Wait for Vault readiness: Retries until Vault API responds
  3. Check seal status: Queries Vault's seal status
  4. Unseal (if needed): Reads the key file and calls vault unseal if Vault is sealed
  5. Stay running: Waits for pod shutdown, surviving restarts and maintaining Vault access

The key file is typically provisioned by an init container, Secret projection, or pre-placed on the host before the pod starts. Vault seals itself on startup by default; vault-unseal then unseals the vault automatically.

Installation​

Prerequisites​

  • Kubernetes cluster with Vault deployed as a StatefulSet
  • Unseal key provisioning mechanism (init container, Secret, pre-placed on host, or external source)

Quick Start​

Add vault-unseal as a sidecar to your Vault StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  namespace: vault
spec:
  # ... standard StatefulSet fields ...
  template:
    spec:
      containers:
        - name: vault
          image: vault:latest
          env:
            - name: VAULT_ADDR
              value: "http://127.0.0.1:8200"
          volumeMounts:
            - name: unseal-key
              mountPath: /vault/unseal
        - name: vault-unseal
          image: ghcr.io/benfiola/homelab-helper:latest
          args: ["homelab-helper", "vault-unseal"]
          env:
            - name: VAULT_ADDR
              value: "http://127.0.0.1:8200"
            - name: VAULT_UNSEAL_KEY
              value: "/vault/unseal/unseal-key"
          volumeMounts:
            - name: unseal-key
              mountPath: /vault/unseal
              readOnly: true
      volumes:
        - name: unseal-key
          # See "Providing the Unseal Key" below
          emptyDir: {}

The sidecar requires:

  • Network access to Vault's API (e.g., localhost on VAULT_ADDR)
  • Read access to the unseal key file at the path specified by VAULT_UNSEAL_KEY

Configuration​

CLI Flags and Environment Variables​

The utility is invoked as:

homelab-helper vault-unseal [flags]
FlagEnv VarRequiredDefaultDescription
--vault-addrVAULT_ADDR✓Vault API address (e.g., http://127.0.0.1:8200)
--vault-unseal-keyVAULT_UNSEAL_KEY✓Path to the unseal key file
--log-levelLOG_LEVELinfoLog level (debug, info, warn, error)
--log-formatLOG_FORMATtextLog format (text or json)

Providing the Unseal Key​

The unseal key must be written to the path specified by VAULT_UNSEAL_KEY so the sidecar can read it and unseal Vault. Choose one approach:

Option 1: Kubernetes Secret (Development)​

Store the unseal key in a Kubernetes Secret and project it as a file into the pod. This is convenient for development but insecure for production as anyone with access to Secrets can unseal Vault. For production, use Option 2 or 3 or a dedicated key management system.

Option 2: Init Container with External Source (Production)​

Use an init container to fetch the unseal key from a secure external system (AWS Secrets Manager, HashiCorp Consul, encrypted file store, HSM) during pod startup and write it to a shared volume. This keeps the key outside the cluster and automates provisioning for each pod restart.

Option 3: Persistent Volume (Manual Key Placement)​

Use a persistent volume (PVC, hostPath, or other backend) bound to the pod. Manually write the unseal key to the volume once; it persists across pod restarts and node reboots. vault-unseal will find it and unseal Vault automatically on each startup.

Monitoring​

Monitor logs to verify the sidecar is working:

kubectl logs -n vault vault-0 -c vault-unseal -f

Troubleshooting​

Unseal fails with error​

Symptom: Logs show an error when attempting to unseal Vault.

Check:

  • Verify the unseal key is correct. An incorrect key will cause unsealing to fail.
  • Inspect the key file location specified by VAULT_UNSEAL_KEY to confirm the key is present and readable.
  • Verify VAULT_ADDR points to a healthy, reachable Vault instance.

"Waiting for vault unseal key" logs continue indefinitely​

Symptom: The sidecar keeps retrying and never finds the unseal key file.

Check:

  • Verify the file exists at the path specified by VAULT_UNSEAL_KEY
  • Check file permissions—the sidecar must have read access to the file
  • If using a volume (Secret, PVC, hostPath), verify the volume is mounted correctly: kubectl describe pod vault-0 | grep -A5 unseal-key

Sidecar logs show errors​

Check logs for the specific error:

kubectl logs vault-0 -c vault-unseal -f

If unclear, enable debug logging: --log-level=debug

Security Considerations​

Unseal Key Protection​

The unseal key is a high-value secret. Treat it with the same care as Vault root tokens.

Storage:

  • Never commit unseal keys to version control or container images
  • For production, use a secure external system (AWS Secrets Manager, HashiCorp Consul, encrypted store, HSM)
  • For development only, Kubernetes Secrets are acceptable
  • Rotate unseal keys periodically: vault operator rekey

Access Control:

  • Limit which workloads can mount the unseal key volume (RBAC, network policies)
  • Use Pod Security Standards to restrict sidecar capabilities
  • Run the sidecar with read-only filesystem where possible

Audit Logging​

  • Enable Vault audit logging to track unseal operations
  • Monitor audit logs for unexpected unsealing outside pod restart windows
  • Alert on repeated unseal failures or unusual timing patterns

See Also​