Skip to main content

vault-push-secrets

Overview​

vault-push-secrets periodically exports Vault secrets, roles, and policies to a simplified schema stored in Google Cloud Storage. Its purpose is enabling rapid cluster recovery by providing a single source to restore Vault during bootstrap.

The Problem​

Using Vault UI as the source of truth works for day-to-day operations, but creates friction during cluster resets:

  • Vault is initialized with no data; you must populate it before applications can start
  • Manually re-creating dozens of secrets across multiple paths is error-prone and time-consuming

Why This Utility​

vault-push-secrets solves this by:

  1. Periodically exporting Vault state (default: every 10 minutes) to a simplified schema that normalizes the common 1:1 relationship between roles, policies, and secrets
  2. Simplifying bootstrap by exporting roles and policies alongside secrets, allowing you to restore all three from a single file
  3. Always maintaining the latest state — the same GCS location is overwritten on each push, ensuring the latest export is always at a predictable location

What Gets Exported​

The export includes:

  • Secrets: All key-value pairs from the configured Vault secret mount
  • Roles: Kubernetes auth roles with their bound service accounts, namespaces, and policy references
  • Policy References: Tracked permissions per role (read specific secrets, read all secrets, read roles, read policies)

The schema is intentionally simplified: it assumes 1:1 role↔policy↔secret relationships, with explicit handling for exceptions (roles bound to any namespace, roles with access to all secrets).

How It Works​

Architecture​

vault-push-secrets is a Deployment running a continuous loop:

  1. Authenticate to Vault

    • Primary: Use the pod's Kubernetes service account token (mounted at /var/run/secrets/kubernetes.io/serviceaccount/token)
    • Fallback: Use explicit VAULT_AUTH_TOKEN if provided (overrides Kubernetes auth)

  2. Export Vault Data

    • List all secrets in VAULT_SECRETS_MOUNT
    • List all Kubernetes auth roles in VAULT_AUTH_MOUNT
    • For each role, extract: bound service account, bound namespace, associated policy name
    • For each policy, parse its rules to determine: which secret paths it grants read access to, whether it can read roles, whether it can read policies

  3. Build Simplified Schema

    • Normalize the export to the schema (see reference below)
    • Track exceptions: namespace-agnostic roles (namespace: ""), roles with wildcard secret access (secret: "")

  4. Calculate Checksum

    • Serialize the schema to JSON
    • Calculate SHA256 hash
    • Compare against the last-uploaded checksum
    • Skip upload if unchanged (saves GCS API calls and storage writes)

  5. Upload if Changed

    • Serialize schema to YAML
    • Upload to GCS at the configured GCS_DESTINATION
    • Overwrite the previous export (no history maintained)
    • Update the in-memory checksum

  6. Loop

    • Sleep for configured interval (default: 10 minutes)
    • Repeat from step 1

Workflow​

[Every 10 minutes]
       ↓
Authenticate to Vault (SA token or explicit token)
       ↓
Export all secrets and Kubernetes auth roles/policies
       ↓
Normalize to simplified schema
       ↓
Calculate checksum
       ↓
Checksum unchanged from last upload?
  Yes → Skip upload, sleep, repeat
  No → Continue
       ↓
Serialize to YAML, upload to GCS
       ↓
Update checksum, sleep, repeat

Key Concepts​

  • Simplified Schema for Homelab Bootstrap: The export normalizes complex Vault configurations into a flat structure designed to be consumed by the homelab project's bootstrap automation. The schema assumes a 1:1 role↔policy↔secret relationship; when this holds, restoration is straightforward. For implementation details, see the appsSecretsSchema in the homelab repository.

  • Exceptions: Some roles don't fit the 1:1 model:

    • Namespace-Agnostic Roles (namespace: ""): A role accepted from any namespace. Useful for services that run in multiple namespaces but share the same secret access.
    • Wildcard Secret Access (secret: ""): A role with read access to all secrets in the mount. Typically used for highly privileged roles (like the pusher itself).

  • Checksum Deduplication: Only uploads when Vault data changes. Avoids redundant GCS API calls and storage writes, reducing costs.

  • Latest-Only Export: Each upload overwrites the previous file at the same GCS location. There is no backup history or point-in-time recovery; the export always contains the current Vault state.

Installation​

Prerequisites​

  • Kubernetes cluster with Vault deployed
  • Google Cloud Storage bucket and GCS service account with write permissions
  • Helm 3.x
  • Vault configured with:
    • Kubernetes auth enabled on VAULT_AUTH_MOUNT
    • Secret mount on VAULT_SECRETS_MOUNT
    • A Kubernetes auth role that the pusher's service account can assume

Deploy with Helm​

Add the repository and install:

helm repo add homelab-helper https://benfiola.github.io/homelab-helper
helm repo update
helm install vault-push-secrets homelab-helper/vault-push-secrets \
  --namespace vault \
  --create-namespace \
  --values values.yaml

The Helm chart creates:

  • Deployment running the pusher
  • ServiceAccount for Kubernetes auth to Vault
  • ClusterRole and ClusterRoleBinding granting list permissions on service accounts (needed to enumerate role bindings)

Verify Installation​

Check the deployment is running:

kubectl get deployment -n vault vault-push-secrets
kubectl logs -n vault -l app.kubernetes.io/name=vault-push-secrets -f

Usage​

Monitoring Exports​

The pusher runs automatically every 10 minutes. Monitor logs to verify behavior:

kubectl logs -n vault -l app.kubernetes.io/name=vault-push-secrets -f

Log Lines:

LineMeaning
authenticating with vaultAttempting Kubernetes or token auth
exporting secretsReading from Vault
secrets unchanged, skipping uploadChecksum matched previous export; no upload needed (normal)
secrets changed, uploadingData changed; new export being uploaded
secrets successfully pushedExport complete; timestamp of push included

Restoring During Bootstrap​

The homelab project provides the homelab bootstrap CLI command to automate Vault restoration from the exported state. Refer to the homelab documentation for bootstrap usage.

Configuration​

CLI Flags and Environment Variables​

The utility is invoked as:

homelab-helper vault-push-secrets [flags]
FlagEnv VarRequiredDefaultDescription
--gcs-credentialsGCS_CREDENTIALS✓Path to GCS service account JSON key file
--gcs-destinationGCS_DESTINATION✓GCS path; format: gs://bucket/path/file.yaml
--intervalINTERVAL10mExport interval (e.g., 10m, 1h)
--vault-addrVAULT_ADDRhttp://localhost:8200Vault API endpoint
--vault-auth-mountVAULT_AUTH_MOUNT✓Kubernetes auth mount path (e.g., kubernetes)
--vault-auth-roleVAULT_AUTH_ROLE*Kubernetes auth role name (required if not using token)
--vault-auth-tokenVAULT_AUTH_TOKEN*Vault token (overrides Kubernetes auth if provided)
--vault-secrets-mountVAULT_SECRETS_MOUNT✓Secret engine mount path (e.g., secret)

* Either --vault-auth-role or --vault-auth-token must be provided.

Helm Chart Values​

config:
  vaultAddr: "http://vault:8200"
  vaultAuthMount: "kubernetes"
  vaultAuthRole: "vault-push-secrets"
  vaultSecretsMount: "secret"
  gcsDestination: "gs://bucket/path/vault-export.yaml"
  gcsCredentialsSecret: "gcs-credentials"
  gcsCredentialsKey: "credentials.json"
  logLevel: "info"

deployment:
  image:
    tag: "" # Defaults to chart version
  resources:
    null
    # Example:
    # limits:
    #   cpu: 200m
    #   memory: 256Mi
    # requests:
    #   cpu: 100m
    #   memory: 128Mi

serviceAccount:
  name: "" # Defaults to chart name

Export Schema Reference​

The export is a YAML document with this structure:

secrets:
  my-app:
    db_host: "postgres.example.com"
    db_user: "myapp"
    db_password: "secret123"
  another-app:
    api_key: "key456"

roles:
  my-app:
    namespace: "default"
    secret: "my-app"
    service-account: "my-app"
    roles: false
    policies: false

  vault-push-secrets:
    namespace: ""
    secret: ""
    service-account: "vault-push-secrets"
    roles: true
    policies: true

Secrets: Each key is a secret name; the value is its key-value data from Vault.

Roles: Each key is a role name with these fields:

  • namespace: Kubernetes namespace the role is bound to. Empty string ("") means any namespace.
  • secret: Secret path the role can read (relative to VAULT_SECRETS_MOUNT). Empty string ("") means all secrets.
  • service-account: Kubernetes service account name.
  • roles: Whether the role can read Kubernetes auth roles.
  • policies: Whether the role can read ACL policies.

Troubleshooting​

Authentication to Vault Fails​

Symptom: Logs show "failed to authenticate with vault"

Diagnose:

If using Kubernetes auth:

# Verify the auth role exists
vault auth list
vault read auth/kubernetes/role/vault-push-secrets

# Test the role from a pod using the same service account as vault-push-secrets
kubectl run debug --rm -it --image=alpine:latest --serviceaccount=vault-push-secrets -n vault -- sh
# Inside pod:
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -X POST http://vault.vault.svc.cluster.local:8200/v1/auth/kubernetes/login \
  -d '{"jwt":"'$TOKEN'","role":"vault-push-secrets"}'

If using token auth:

# Verify token is valid
vault token lookup <TOKEN>

# Verify token has read permissions
vault token lookup -format=json <TOKEN> | jq .data.policies

GCS Upload Fails​

Symptom: Logs show "failed to upload to cloud storage"

Diagnose:

Verify the GCS credentials and bucket are accessible from your local machine:

# Set up credentials locally
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs-service-account-key.json

# Verify bucket access
gsutil ls gs://my-bucket/

# Test write permissions
gsutil -m cp /etc/hostname gs://my-bucket/test-write.txt
gsutil rm gs://my-bucket/test-write.txt

Common causes:

  • GCS service account lacks storage.objects.create permission on the bucket
  • Bucket doesn't exist or is in a different project
  • GCS destination path is malformed (should be gs://bucket/path/file.yaml)

Export is Incomplete​

Symptom: Downloaded export contains fewer secrets or roles than expected

Diagnose:

Verify that the expected data exists in Vault:

# Check current Vault state matches expected secrets
vault kv list secret/
vault kv list secret/my-app/

# Check current roles match expected
vault list auth/kubernetes/role/

Common causes:

  • Secrets or roles were added to Vault after the last export cycle (the export is always point-in-time accurate)
  • Secrets exist on a different mount

See Also​