Skip to main content

How to integrate Hashicorp Vault with Ververica Platform

Jun Qin
  • Updated

Question

My organization is interested to use Hashicorp Vault to manage sensitive information. How can I use it together with Ververica Platform? 

Answer

Note: This section applies to Ververica Platform 2.2 - 2.8 Enterprise Edition.

The Vault Agent Injector service in Hashicorp Vault leverages the Kubernetes mutating admission webhook to inject secrets into Kuberntes pods via a sidecar container. Applications can remain Vault unaware as the secrets are stored on the file-system within their container.

Assuming we have set up Ververica Platform. It runs in the Kubernetes namespace `vvp`, and the Flink jobs will run in the Kubernetes namespace `flink`. We will follow this Hashicorp Vault tutorial but adapt it to our setup. For a detailed explanation of those Hashicorp Vault commands, please refer to their tutorial. 

Install Hashicorp Vault in our Kubernetes Cluster

helm repo add hashicorp https://helm.releases.hashicorp.com 

helm install vault hashicorp/vault --set "server.dev.enabled=true,server.extraEnvironmentVars.VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200" 

kubectl get pods

Note: The second command above is different from the one in the original tutorial. This is due to a bug  in Hashicorp Vault. 

Create a Secret in Vault

Let us assume we want to create a secret containing username and password at the path: `vvp/database/config`. Login to the `vault-0` pod:

kubectl exec -it vault-0 -- /bin/sh

and then run:

vault secrets enable -path=vvp kv-v2

vault kv put vvp/database/config username=vvpuser password=Top-Secret0

vault kv get vvp/database/config

Enable Kubernetes Authentication

Login to the `vault-0` pod again if you exited already. With the following commands we create a role `vvp-flink-job` and bind the service account `default` and other ones in the namespace `flink` to the policy `read-vvp-secret` which is created by the second command:

vault auth enable kubernetes

vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault policy write read-vvp-secret - <<EOF
path "vvp/data/database/config" {
    capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/vvp-flink-job \
    bound_service_account_names=default,job-*-flink-ha-*manager \
    bound_service_account_namespaces=flink \
    policies=read-vvp-secret \
    ttl=24h

Note: `bound_service_account_names` in the last command is the service account of Flink pods. If you do not use Kubernetes HA in Ververica Platform, the service account is `default`. Otherwise, the service account is `job-<jobid>-flink-ha-jobmanager` and `job-<jobid>-flink-ha-taskmanager`. Here we cover both cases by using wildcards. See Create a Role in Vault.

Note: `bound_service_account_namespaces` in the last command is the namespace where the Flink pods run. Similar to `bound_service_account_names`, comma-separated multiple namespaces and wildcard can be used here if you want to cover multiple namespaces. See Create a Role in Vault.

 

Add Annotations to Flink Pods

Now we create a Deployment in Ververica Platform and add the following annotations through the YAML editor:

spec:
  template:
    spec:
      kubernetes:
        pods:
          annotations:
            vault.hashicorp.com/agent-inject: true
            vault.hashicorp.com/role: vvp-flink-job
            vault.hashicorp.com/agent-inject-secret-database-config.txt: vvp/data/database/config

When the Flink pod is started, log in to the Flink pod, you should see a file `/vault/secrets/database-config.txt` containing the secret we defined. To get that secret, your job simply reads it from the file.

kubectl exec -it -n flink <JM/TM_pod> -- cat /vault/secrets/database-config.txt

Related Information

Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar

Create a Role in Vault

 

Related articles