Skip to main content

router-policy-sync

Overview​

router-policy-sync is a Kubernetes controller that synchronizes Cilium network policies with your upstream router (MikroTik) firewall rules. It automates the administration of external internet access to cluster resources by automatically discovering gateway IPs and keeping firewall rules in sync with your in-cluster network policies.

The Problem​

Managing external access to cluster resources requires coordinating multiple control planes:

  1. Router firewall rules: Define which external IPs can reach your cluster
  2. In-cluster network policies: Define which pods can receive that traffic
  3. Gateway discovery: Where does the gateway IP come from to route traffic?

Manually keeping these synchronized is error-prone and tedious:

  • Update the Cilium policy, remember to update the firewall rule
  • Redeploy a gateway pod with a new IP, manually update the NAT rule
  • Delete a source IP from the policy, manually remove from the router
  • Forget a step, and you either expose services unintentionally or block legitimate traffic

This controller eliminates the friction by automatically keeping your router firewall rules in sync with Cilium network policies. Define your ingress policy once in Cilium, annotate it, and the controller automatically syncs rules to your router and discovers the gateway IP. Everything stays synchronized.

How It Works​

Architecture​

router-policy-sync uses two reconcilers working together:

  1. CiliumPolicyReconciler: Watches Cilium CiliumClusterwideNetworkPolicy resources for the sync annotation (router-policy-sync.homelab-helper.benfiola.com/sync-with-router). When a Cilium policy is annotated, it creates a corresponding cluster-scoped RouterPolicy custom resource with the same name as the Cilium policy.

  2. RouterPolicyReconciler: Watches RouterPolicy resources. When a policy is created or changes, it:

    • Extracts allowed source IPs from the referenced Cilium policy's FromCIDR fields (validates that all are /32 ranges)
    • Discovers the gateway IP and service by finding pods matching the Cilium policy's endpoint selector, then finding their LoadBalancer service
    • Extracts allowed protocols and ports from the discovered LoadBalancer service's port definitions
    • Syncs the source IPs, protocols, and ports to the MikroTik router (firewall filters, NAT rules, and address lists)
    • Reports reconciliation status back to the RouterPolicy

Workflow​

Annotate CiliumClusterwideNetworkPolicy with sync-with-router
       ↓
CiliumPolicyReconciler detects annotation
       ↓
Creates RouterPolicy (cluster-scoped, same name as Cilium policy)
       ↓
RouterPolicyReconciler detects new RouterPolicy
       ↓
Extracts allowed source IPs from Cilium policy's FromCIDR fields
       ↓
Discovers gateway IP and service:
  - Finds pods matching Cilium policy's endpoint selector
  - Finds LoadBalancer service for each pod
  - Extracts service IP from status.loadBalancer.ingress
       ↓
Extracts protocols and ports from LoadBalancer service's port definitions
  - Groups ports by protocol (tcp, udp, etc.)
       ↓
Validates IPs against reserved CIDR ranges
       ↓
Syncs to MikroTik:
  - Creates/updates firewall filter per protocol (comma-separated ports)
  - Creates/updates firewall NAT rule per protocol/port combination
  - Syncs address lists (desired source IPs from policy)
  - Cleans up stale rules on each sync
       ↓
Updates RouterPolicy status (Ready condition, timestamps)
       ↓
Remove annotation from CiliumClusterwideNetworkPolicy
       ↓
CiliumPolicyReconciler detects annotation removal
       ↓
Deletes RouterPolicy
       ↓
RouterPolicyReconciler detects deletion
       ↓
Cleans up firewall rules on router

Key Concepts​

  • RouterPolicy: A cluster-scoped custom resource that represents a single Cilium policy being synced to the router. The RouterPolicy name matches the Cilium policy it represents, and it is owned by that policy (garbage collected when the policy is deleted).

  • Firewall Address List: A named list of IPs on the MikroTik router. The controller keeps this in sync with the source IPs extracted from the referenced Cilium policy's FromCIDR fields.

  • Firewall Filter: One rule per protocol (tcp, udp, etc.), referencing the address list. Each rule contains comma-separated ports for that protocol. Filters are synced during each reconciliation; stale rules (for protocols no longer in the policy) are automatically removed.

  • Firewall NAT Rule: One destination NAT rule per protocol/port combination. The NAT rule translates traffic from allowed source IPs on a specific port to the discovered gateway IP address (obtained from the LoadBalancer service status). NAT rules are synced during each reconciliation; stale rules are automatically removed.

  • Source IP Validation: All FromCIDR entries in the Cilium policy must be /32 ranges (individual IPs). Any CIDR that is not a /32 will cause the sync to fail. Additionally, IPs must not fall within reserved CIDR ranges (0.0.0.0/8, 127.0.0.0/8, etc.).

  • Port and Protocol Validation: All ports in the discovered LoadBalancer service's port definitions must be valid (e.g., integers between 1 and 65535). Protocols must be valid (tcp, udp, etc.).

  • Firewall Marker: A pre-existing firewall filter rule that acts as an anchor point. New filter rules are inserted at this marker's position, ensuring your rules appear in the correct chain order (see Installation section).

How IP Discovery Works​

The controller discovers the gateway IP by:

  1. Reading the endpoint selector from the Cilium policy's ingress rules
  2. Finding all pods across the cluster that match the endpoint selector labels
  3. For each matching pod, finding its LoadBalancer service (by checking service selectors against pod labels)
  4. Extracting the IP from service.status.loadBalancer.ingress[*].ip
  5. Using the first discovered IP as the NAT target

This means your Cilium policy's endpoint selector should target the pods that host the LoadBalancer service exposing your gateway or application.

Installation​

Prerequisites​

  • Kubernetes cluster with Gateway API installed (v1.0.0+)
  • A Envoy Gateway API controller
  • Cilium deployed and configured for network policies
  • MikroTik router with REST API enabled
  • Helm 3.x

Firewall Marker Setup​

Marker Filter Required

As a controller, we cannot confidently determine where to insert firewall filter rules. Guessing wrong could be a serious security problem (e.g., accidentally blocking legitimate traffic or exposing the cluster). You must create a marker rule to explicitly tell the controller where to insert its rules.

Create a marker firewall filter rule on your MikroTik router. The marker should be positioned in the chain where you want the controller's rules inserted—typically before any deny-all policies that terminate the chain.

The marker rule must have the comment router-policy-sync::marker so the controller can locate it. You can create it via your router's standard firewall filter commands (web UI, API, or CLI).

Verification:

Before deploying the controller, verify the marker exists:

/ip firewall filter print where comment="router-policy-sync::marker"

You should see one rule in the output with the correct comment.

Deploy with Helm​

Add the repository and install:

helm repo add homelab-helper https://benfiola.github.io/homelab-helper
helm repo update
helm install router-policy-sync homelab-helper/router-policy-sync \
  --namespace router-policy-sync \
  --create-namespace \
  --set config.mikrotikBaseURL="https://192.168.1.1/rest" \
  --set config.mikrotikUsername="admin" \
  --set config.mikrotikPasswordSecret="router-credentials" \
  --set config.mikrotikPasswordKey="password"

Create a secret containing the MikroTik password:

kubectl create secret generic router-credentials \
  --from-literal=password="your-mikrotik-password" \
  -n router-policy-sync

The chart deploys:

  • A Deployment running the controller
  • A ServiceAccount with necessary RBAC permissions
  • ClusterRole and ClusterRoleBinding for Cilium and pod/service resource access
  • Custom Resource Definition (RouterPolicy)

Verify Installation​

Check the deployment is running:

kubectl get deployment -n router-policy-sync router-policy-sync
kubectl logs -n router-policy-sync -l app.kubernetes.io/name=router-policy-sync

Usage​

Syncing a Cilium Policy​

To synchronize a Cilium network policy with your router, annotate it:

kubectl annotate ciliumclusterwidenetworkpolicy my-policy \
  router-policy-sync.homelab-helper.benfiola.com/sync-with-router="" \
  --cluster

The controller will immediately:

  1. Create a RouterPolicy resource (cluster-scoped) with the same name as the Cilium policy
  2. Extract allowed source IPs from the Cilium policy's FromCIDR fields (must be /32)
  3. Discover the gateway IP and service by finding pods matching the policy's endpoint selector and their LoadBalancer service
  4. Extract allowed protocols and ports from the LoadBalancer service's port definitions
  5. Sync firewall rules to MikroTik (filter, NAT, and address lists)
  6. Report status in the RouterPolicy

Updating the Allowlist​

To update the allowed source IPs, edit the Cilium policy directly:

kubectl edit ciliumclusterwidenetworkpolicy my-policy

Update the FromCIDR fields in the ingress rules. All CIDRs must be /32 ranges (individual IPs):

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: my-policy
spec:
  endpointSelector:
    matchLabels:
      app: my-gateway
  ingress:
    - fromCIDR:
        - "203.0.113.10/32"
        - "198.51.100.5/32"
        - "192.0.2.100/32"

To update protocols or ports, edit the LoadBalancer service's port definitions:

kubectl edit service my-gateway-lb

Update spec.ports with the desired port and protocol configuration. Changes are synced to the router immediately. The controller will create/update firewall rules for each protocol and clean up any rules for protocols no longer present in the service.

Monitoring Status​

Check the RouterPolicy status:

kubectl get routerpolicy
kubectl describe routerpolicy my-policy

The status shows:

  • Conditions: Ready condition with success/failure details
  • lastReconciledTime: When the policy was last synced
  • observedGeneration: Tracks which spec generation was processed

Example status:

status:
  conditions:
    - type: Ready
      status: "True"
      reason: Synced
      message: "Router policy synced with firewall (protocols: [{tcp [80 443]} {udp [53]}], policy IPs: [203.0.113.10], gateway IP: 10.0.0.5)"
      observedGeneration: 1
  lastReconciledTime: "2024-01-15T10:30:00Z"
  observedGeneration: 1

Stopping Sync​

To stop syncing a Cilium policy and clean up all associated firewall rules:

kubectl annotate ciliumclusterwidenetworkpolicy my-policy \
  router-policy-sync.homelab-helper.benfiola.com/sync-with-router- \
  --cluster

The controller will:

  1. Delete the RouterPolicy (via owner reference / garbage collection)
  2. Remove firewall filter, NAT, and address list entries from the router

All resources are cleaned up within seconds.

Configuration​

CLI Flags and Environment Variables​

The controller is invoked as:

homelab-helper router-policy-sync [flags]

Available flags and their environment variable equivalents:

FlagEnvironment VariableDefaultDescription
--health-addressHEALTH_ADDRESS:8081Address for health/readiness probes (/healthz, /readyz)
--metrics-addressMETRICS_ADDRESS:8080Address for Prometheus metrics endpoint (/metrics)
--leader-electionLEADER_ELECTIONfalseEnable leader election for HA deployments
--kubeconfigKUBECONFIG""Path to kubeconfig; uses in-cluster config if empty
--mikrotik-base-urlMIKROTIK_BASE_URL(required)Base URL of MikroTik REST API (e.g., https://192.168.1.1/rest)
--mikrotik-usernameMIKROTIK_USERNAME(required)Username for MikroTik API authentication
--mikrotik-passwordMIKROTIK_PASSWORD(required)Password for MikroTik API authentication
--sync-intervalSYNC_INTERVAL5mInterval for periodic reconciliation to detect drift

Helm Chart Values​

config:
  mikrotikBaseURL: "https://192.168.1.1/rest"
  mikrotikUsername: "admin"
  mikrotikPasswordSecret: "router-credentials"
  mikrotikPasswordKey: "password"
  syncInterval: "5m"

deployment:
  image:
    # Override image tag (defaults to chart version)
    tag: ""

  # Number of controller replicas (use >1 with --leader-election for HA)
  replicas: 1

  # Resource limits/requests (optional)
  resources:
    null
    # Example:
    # limits:
    #   cpu: 200m
    #   memory: 256Mi
    # requests:
    #   cpu: 100m
    #   memory: 128Mi

RouterPolicy Reference​

Overview​

RouterPolicy is a cluster-scoped resource created automatically by the CiliumPolicyReconciler when a Cilium policy is annotated with the sync annotation. Users typically do not create RouterPolicy resources directly; they are managed by the controller.

Spec​

The RouterPolicy spec is intentionally empty. All policy information is derived from the referenced Cilium policy.

apiVersion: router-policy-sync.homelab-helper.benfiola.com/v1
kind: RouterPolicy
metadata:
  name: my-policy
spec: {}

Status​

The RouterPolicy status communicates reconciliation state and any errors.

status:
  conditions:
    - type: Ready
      status: "True"
      reason: Synced
      message: "Router policy synced with firewall (policy IPs: [203.0.113.10], gateway IP: 10.0.0.5)"
      observedGeneration: 1
  lastReconciledTime: "2024-01-15T10:30:00Z"
  observedGeneration: 1

Status Fields:

FieldTypeDescription
conditionsarrayArray of condition objects; contains one Ready condition
lastReconciledTimeRFC3339 stringTimestamp of last successful reconciliation
observedGenerationintegerTracks which spec generation was last processed

Condition Reasons:

ReasonStatusMeaning
SyncedTruePolicy successfully synced to router
ValidationFailedFalseCilium policy validation failed (invalid IPs or CIDRs)
DiscoveryFailedFalseFailed to discover gateway IP from LoadBalancer
SyncFailedFalseRouter sync failed

Troubleshooting​

Sync Not Happening​

Symptom: RouterPolicy created but Ready condition is False with reason SyncFailed, DiscoveryFailed, or ValidationFailed.

Check controller logs:

kubectl logs -n router-policy-sync -l app.kubernetes.io/name=router-policy-sync

Common causes:

  • Validation failed (IPs): The Cilium policy's FromCIDR entries are not all /32 ranges. Check the policy and ensure each CIDR is an individual IP (e.g., 203.0.113.10/32).
  • Validation failed (ports/protocols): The LoadBalancer service has invalid port definitions. Check that all ports are valid integers (1-65535) and protocols are valid (tcp, udp, etc.).
  • Invalid IP in policy: An IP in the Cilium policy's FromCIDR is malformed or in a reserved range. Check that all IPs are valid IPv4 addresses and not in reserved ranges (0.0.0.0/8, 127.0.0.0/8, etc.).
  • Gateway IP discovery failed: No pods matched the endpoint selector, or the matching pods don't have a LoadBalancer service. Verify that:
    • The endpoint selector in the Cilium policy correctly matches your pods
    • There is a LoadBalancer service that selects those pods
    • The LoadBalancer service has an IP assigned in status.loadBalancer.ingress
    • The LoadBalancer service has ports defined in spec.ports
  • MikroTik connection error: Cannot reach the router or credentials are wrong. Verify the base URL, username, and password are correct.
  • Firewall marker missing: The router-policy-sync::marker filter rule doesn't exist on the router. Create it per the Installation section.

Enable Debug Logging​

Run controller with debug logging:

kubectl set env -n router-policy-sync deployment/router-policy-sync LOG_LEVEL=debug

Then tail the logs:

kubectl logs -n router-policy-sync -l app.kubernetes.io/name=router-policy-sync -f

Automatic Remediation​

The controller periodically resyncs RouterPolicies according to the --sync-interval flag (default 5 minutes). This means if firewall rules or address lists are manually deleted from the router, they will be automatically recreated on the next sync cycle. This provides automatic drift detection and remediation without manual intervention.

You can adjust the sync interval in the Helm chart or via the environment variable to balance between responsiveness and API load on the router.

Limitations​

  • IPv4 only: The controller currently supports only IPv4 addresses.
  • /32 CIDR requirement: All source IPs in the Cilium policy's FromCIDR must be /32 ranges (individual IPs). CIDR blocks and ranges are not supported.
  • Single LoadBalancer IP: The controller uses the first IP discovered from the pod's LoadBalancer service. If multiple IPs are present, only the first is used for NAT.
  • Firewall filter chain: The marker filter must be in the firewall filter chain where you want rules inserted. If you use multiple chains, you'll need multiple markers in each chain.
  • MikroTik API dependency: The controller requires REST API access to your router. If the router is unreachable, syncs will fail until connectivity is restored.
  • Pod-to-service matching: The endpoint selector in your Cilium policy should match pods that are exposed by a LoadBalancer service. If no service is found, IP discovery will fail.
  • Stale rule cleanup: During each reconciliation, the controller removes firewall filter and NAT rules that are no longer desired (e.g., if you remove a port from the service). This cleanup only happens during active reconciliation; manual rule deletion on the router will be recreated if the service still defines those ports.

See Also​