Question

How can I use the secrets stored in Azure Key Vault if I deploy Ververica Platform on the Azure Kubernetes Service (AKS)?

Answer

Note: This article applies to Ververica Platform 2.0-2.8.

You can use the Azure Key Vault provider for Secrets Store CSI driver to access Azure Key Vault. It allows getting secret contents stored in an Azure Key Vault instance and use the Secrets Store CSI driver interface to mount them into the Ververica Platform pods or into your Flink job pods running on Ververica Platform. At the high level, the procedure consists of four steps:

Identify the `clientId` and the `principalId` you want to use to access Azure Key Vault
In Azure, grant Key Vault access permissions to the identified `clientId` and `principalId`
In AKS, create a SecretProviderClass that specifies the parameters to connect to Key Vault
For the Kubernetes pods, configure volumes with the created storage class and mount them

We will walk through these steps in detail in the sections below.

0. Prepare a Key Vault

Before we dive into the details, let us first create a Key Vault and a secret in Azure to play with. Skip this section if you have already created a key vault and set a secret.

Azure Key Vault supports two different permission models: vault access policy and azure role-based access control. We show both scenarios here, but you can choose one of them to proceed with. Both work with the four Key Vault access modes described in the next section.

Note: The remainder of the article uses the following variables:

`$tenantId`: your tenantId in Azure
`$resourceGroup`: the name of the resource group the AKS cluster is in
`$nodeResourceGroup`: the name of the resource group of the AKS cluster nodes are in
`$clusterName`: the name of the AKS cluster
`$keyVaultName`: the name of the key vault
`$secretName`: the name of the secret in the key vault
`$secretFile`: the full path to the file containing the secret contents

Option A: Create a Key Vault with vault access policy

# create key vault and store its id in $keyVaultId
az keyvault create --resource-group ${resourceGroup} --name ${keyVaultName}
keyVaultId=$(az keyvault show --resource-group ${resourceGroup} 
             --name ${keyVaultName} --query id -o tsv)
# set a secret
az keyvault secret set --vault-name ${keyVaultName} --name ${secretName} \
  --file ${secretFile}

Option B: Create a Key Vault with Azure role-based access control

# create a RBAC enabled key vault and store its id in $keyVaultId
az keyvault create --resource-group ${resourceGroup} \
  --name ${keyVaultName} --enable-rbac-authorization
keyVaultId=$(az keyvault show --resource-group ${resourceGroup} 
             --name ${keyVaultName} --query id -o tsv)

# add the required role assignment to yourself in order to set a secret
az role assignment create --role "Key Vault Administrator" \
  --assignee <yourlogin> --scope ${keyVaultId}
# now set a secret
az keyvault secret set --vault-name ${keyVaultName} --name ${secretName} \
  --file ${secretFile}

* <yourlogin> above is the username you used to log in to the Azure portal.

1. Identify `clientId` and `principalId`

The Azure Key Vault Provider offers four modes for accessing a Key Vault instance. The following four subsections show how to identify the `clientId` and `principalId` for each of these modes. You can choose one to proceed with. While the Service Principal mode requires a Kubernetes secret to work, the Pod Identity mode and the User/System-assigned Managed Identity modes can access a Key Vault instance without the need for a Kubernetes secret. The System-assigned Managed Identity mode needs only the `principalId`.

Service Principal Mode

If your AKS cluster was created in the following way:

az aks create --name $clusterName ... \
  --service-principal $clientId \
  --client-secret $clientSecret

and you know the values of `clientId` and `clientSecret`, then you can use this `clientId`. Otherwise, you can also create a new service principal. For example:

SP=$(az ad sp create-for-rbac --skip-assignment --name <svcPrincipalName>)
clientId=$(echo $SP | jq -r '.appId')
clientSecret=$(echo $SP | jq -r '.password')

* <svcPrincipalName>: use a meaningful service principal name of your choice.

When using service principals to access Key Vault, the `clientId` is also the `principalId`.

To use a service principal to access a Key Vault instance, you will need to store the `clientId` and the `clientSecret` into a Kubernetes secret in the same namespace as the referencing pods. As you will see later, the secret will be used by the Secrets Store CSI driver to connect to the Key Vault instance.

kubectl create secret generic <svcPrincipalSecretName> \
  --from-literal clientid=$clientId \
  --from-literal clientsecret=$clientSecret \
  --namespace <namespace>

Pod Identity Mode

Refer to the Azure Key Vault Provider doc on the specific steps to set up pod identity for Key Vault access. In this mode, you can get the `clientId` and the `principalId` from the created identity:

az identity create --resource-group $nodeResourceGroup --name <identityName>
clientId=$(az identity show --resource-group $nodeResourceGroup 
  --name <identityName> --query clientId -o tsv)
principalId=$(az identity show --resource-group $nodeResourceGroup 
  --name <identityName> --query principalId -o tsv)

* <identityName>: use a meaningful identity name of your choice.

User-assigned Managed Identity Mode

Similar to the pod identity mode, the `clientId` and the `principalId` are from the created identity.

System-assigned Managed Identity Mode

To have a system-assigned managed identity for an AKS virtual machine scale set (vmss), run:

nodePoolVMSS=$(az vmss list --resource-group $nodeResourceGroup 
  | jq -r '.[0].name')
az vmss identity assign --resource-group $nodeResourceGroup \
  --name $nodePoolVMSS
principalId=$(az vmss identity show --resource-group $nodeResourceGroup 
  --name $nodePoolVMSS --query principalId -o tsv)

In this case, we only have the `principalId` from the generated identity. `clientId` is not needed in this mode.

2. Grant Access Permissions to the identified `clientId` and `principalId`

For Key Vault with vault access policy:

az keyvault set-policy --name ${keyVaultName} --spn ${clientId} \
  --secret-permissions get

Here we grant only the secret permission `get`. You can add or reduce permissions if necessary. In the System-assigned Managed Identity mode, because we do not get a `clientId`, we use with `--object-id $principalId` instead:

az keyvault set-policy --name ${keyVaultName} --object-id ${principalId} \
  --secret-permissions get

For Key Vault with Azure role-based access control:

az role assignment create --role "Key Vault Secrets User" \
  --assignee $principalId --scope $keyVaultId

Here we assign the role "Key Vault Secrets User". Change to another role if desired.

3. Create a `SecretProviderClass`

First, install the Azure Key Vault Provider for Secrets Store CSI Driver:

# add helm repo if not done yet
helm repo add csi-secrets-store-provider-azure \
  https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts

# install
helm install csi-secrets \
  csi-secrets-store-provider-azure/csi-secrets-store-provider-azure

Verify the CSI driver is installed and the pods are running

% kubectl get pod
NAME                                                 READY   STATUS    RESTARTS   AGE
csi-secrets-csi-secrets-store-provider-azure-vqpwn   1/1     Running   0          94s
csi-secrets-secrets-store-csi-driver-86kft           3/3     Running   0          94s

Depending on the access mode you use above, you need to supply different parameters when creating the `SecretProviderClass`:

Service Principal Mode:

cat << EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: <secretProviderName>
  namespace: <namespace>
spec:
  provider: azure
  parameters:
    keyvaultName: ${keyVaultName}
    objects:  |
      array:
        - |
          objectName: ${secretName}
          objectType: secret
    tenantId: ${tenantId}
EOF

* <secretProviderName>: Use a meaningful secret provider name of your choice.
* <namespace>: the namespace of the pod which needs access to the Key Vault

Otherwise, you need to modify the YAML file above by adding the following settings under `spec.parameters`:

Pod Identity Mode:

    usePodIdentity: "true"

User-Assigned Managed Identity Mode:

    useVMManagedIdentity: "true"
    userAssignedIdentityID: $clientId

System-Assigned Managed Identity Mode:

    useVMManagedIdentity: "true"
    userAssignedIdentityID: ""

4. Configure and Mount Volumes in Pods

Now you can reference this SecretProviderClass in a volume and mount the volume in a pod. To access Key Vault via the Service Principal in the jobmanager pod, for example, add the following into your deployment spec:

spec:
  templates:
    spec:
      kubernetes:
        jobManagerPodTemplate:
          spec:
            containers:
              - name: flink-jobmanager
                volumeMounts:
                  - mountPath: /azkvsecret
                    name: azkvsecret
            volumes:
              - name: azkvsecret
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: <secretProviderName>
                  # only needed in the Service Principal Mode
                  nodePublishSecretRef:  
                    name: <svcPrincipalSecretName>

If you want to access Key Vault in the Ververica Platform pod, add the following into the `values.yaml` file, then use the file to setup/upgrade Ververica Platform via helm:

volumeMounts:
  - name: azkvsecret
    mountPath: /azkvsecret
    readOnly: true  
volumes:
  - name: azkvsecret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: <secretProviderName>
      nodePublishSecretRef: # only needed in the service principal mode
        name: <svcPrincipalSecretName>

Accessing Key Vault via a Pod Identity or a User/System-assigned Managed Identity is the same as above, except that there is no need to add `nodePublishSecretRef` to the volume specification because the `SecretProviderClass` already contains the information about which identity to use.

The Pod Identity Mode also requires labeling the pods with the label `aadpodidbinding` (see the Pod Identity access mode for more details). You can label Flink jobmanager pods as follows:

spec:
  templates:
    spec:
      kubernetes:
        jobManagerPodTemplate:
          metadata:
            labels:
              aadpodidbinding: <the selector specified in AzureIdentityBinding>

For Ververica Platform pods, you can use the following Values file to add extra labels:

extraLabels:
  aadpodidbinding: <selector you specified in AzureIdentityBinding>

Important: this `extraLabels` feature is only available in Ververica Platform 2.5 or later. For older versions, you would need to add the label manually.

Now, when the Ververica Platform or Flink job pods are started, you should see the secret in the mounted volume.