How to use SecretHub on GKE

This guide will show you how to provision an application running on Google Kubernetes Engine with the secrets it needs.

Using the SecretHub GCP Identity Provider in combination with GKE Workload Identity, your app will be able to load secrets simply by taking on a Kubernetes service account without having to deal with another key.

To make life easy, you can use the demo app from the Getting Started guide, but you can also follow along with your own app.

Before you begin

Before you start using SecretHub with GKE, make sure you have:

1. Followed the Getting Started guide.
2. Installed gcloud with application default credentials configured.
3. Installed kubectl and connected it to a GKE cluster.

Depending on whether you’re following allong with Terraform or the CLI, set up the following:

export GCP_PROJECT_ID="demo-project-1234567890"

variable "gcp_project_id" {
  description = "GCP project ID"
}

variable "gke_cluster" {
  description = "Name of your GKE cluster, must be in `gcp_location`"
}

variable "gke_cluster_location" {
  description = "GCP location where the `gke_cluster` runs"
}

variable "secrethub_repo" {
  description = "SecretHub repo that contains the demo `username` and `password` secrets"
}

provider "google" {
  project = var.gcp_project_id
}

data "google_client_config" "provider" {}

data "google_container_cluster" "cluster" {
  name     = var.gke_cluster
  location = var.gke_cluster_location
}

provider "kubernetes" {
  load_config_file       = false
  host                   = "https://${data.google_container_cluster.cluster.endpoint}"
  token                  = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(data.google_container_cluster.cluster.master_auth[0].cluster_ca_certificate)
}

provider "secrethub" {
  credential = file("~/.secrethub/credential")
}

Step 1: Enable workload identity on your GKE cluster

To use the GCP Identity Provider on GKE, you first need to make sure Workload Identity is enabled on your GKE cluster, so that Kubernetes service accounts can be coupled to Cloud IAM service accounts.

You can either enable it on an existing cluster or create a new cluster with Workload Identity already enabled by default.

Step 2: Create a GCP service account for your app

For every app that you deploy on GKE, it’s recommended to create a GCP Service Account that represents the app, so you can apply least privilege.

gcloud iam service-accounts create demo-app \
  --display-name="SecretHub demo app"

gcloud iam service-accounts list \
  --filter="name:demo-app" \
  --format="value(email)"

export GCP_SERVICE_ACCOUNT_EMAIL="demo-app@demo-project-1234567890.iam.gserviceaccount.com"

resource "google_service_account" "demo_app" {
  account_id  = "demo-app"
  description = "SecretHub demo app"
}

Step 3: Create and link Kubernetes Service Account

To couple the GCP service account to a pod on GKE, you need to create a Kubernetes service account and link it to your GCP service account by adding an annotation that contains the service account email:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-app
  annotations:
    iam.gke.io/gcp-service-account: ${GCP_SERVICE_ACCOUNT_EMAIL}
EOF

gcloud iam service-accounts add-iam-policy-binding \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[default/demo-app]" \
  ${GCP_SERVICE_ACCOUNT_EMAIL}

resource "kubernetes_service_account" "demo_app" {
  metadata {
    name = "demo-app"

    annotations = {
      "iam.gke.io/gcp-service-account" = google_service_account.demo_app.email
    }
  }
}

resource "google_service_account_iam_binding" "demo_app" {
  service_account_id = google_service_account.demo_app.name
  role               = "roles/iam.workloadIdentityUser"

  members = [
    "serviceAccount:${var.gcp_project_id}.svc.id.goog[default/${kubernetes_service_account.demo_app.metadata[0].name}]",
  ]
}

Step 4: Create a Cloud KMS key

For encryption and decryption of your secrets, you need to create a Cloud KMS key.

gcloud kms keyrings create secrethub \
  --location="global"

gcloud kms keys create demo-app \
  --location="global" \
  --keyring="secrethub" \
  --purpose="encryption"

gcloud kms keys add-iam-policy-binding demo-app \
  --location="global" \
  --keyring="secrethub" \
  --member="serviceAccount:${GCP_SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/cloudkms.cryptoKeyDecrypter"

resource "google_kms_key_ring" "secrethub" {
  name     = "secrethub"
  location = "global"
}

resource "google_kms_crypto_key" "demo_app" {
  name     = "demo-app"
  key_ring = google_kms_key_ring.secrethub.id
}

resource "google_kms_crypto_key_iam_binding" "crypto_key" {
  crypto_key_id = google_kms_crypto_key.demo_app.id
  role          = "roles/cloudkms.cryptoKeyDecrypter"

  members = [
    "serviceAccount:${google_service_account.demo_app.email}",
  ]
}

Step 5: Create a SecretHub service account

With the KMS key and the GCP-linked Kubernetes service account in place, you can now create a SecretHub service account:

secrethub service gcp init your-username/demo \
  --description demo-app \
  --service-account-email ${GCP_SERVICE_ACCOUNT_EMAIL} \
  --kms-key "projects/${GCP_PROJECT_ID}/locations/global/keyRings/secrethub/cryptoKeys/demo-app" \
  --permission read

resource "secrethub_service_gcp" "demo_app" {
  repo                  = var.secrethub_repo
  service_account_email = google_service_account.demo_app.email
  kms_key_id            = google_kms_crypto_key.demo_app.id
}

resource "secrethub_access_rule" "demo_app" {
  account_name = secrethub_service_gcp.demo_app.id
  dir          = var.secrethub_repo
  permission   = "read"
}

Do note that to be able to run this, you need to have GCP credentials present with encryption rights on the KMS key you pass in.

After running it, you may have noticed that unlike the usual SecretHub service accounts, the SecretHub GCP service account does not output a credential.

That’s because apps running on GCP don’t need it anymore: as long they take on the specified service account on GCP, they can automatically load their secrets from SecretHub.

Step 6: Create Kubernetes Deployment

Next up, create a pod specification for your app (e.g. app.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: app
  name: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      run: app
  template:
    metadata:
      labels:
        run: app
    spec:
      serviceAccountName: demo-app
      containers:
      - name: demo-app
        image: secrethub/demo-app
        env:
        - name: DEMO_USERNAME
          value: secrethub://your-username/demo/username
        - name: DEMO_PASSWORD
          value: secrethub://your-username/demo/password
        - name: SECRETHUB_IDENTITY_PROVIDER
          value: gcp
        ports:
        - containerPort: 8080

In the env stanza the environment variables DEMO_USERNAME and DEMO_PASSWORD (needed by the demo app) are configured to reference the secrets that need to be loaded.

As you can see, there is no extra key that needs to be managed to be able to load secrets. Assigning the serviceAccountName and setting SECRETHUB_IDENTITY_PROVIDER to gcp is all that’s needed on the pod-side to authenticate.

The secrethub/demo-app image already includes the SecretHub CLI and secrethub run entrypoint, so the app is portable across different platforms. If you don’t want to include it in the image, you can use the SecretHub Mutating Webhook.

Step 7: Use the mutating webhook

If you don’t want to include the SecretHub binary in your image, you can use the SecretHub Mutating Webhook to automatically mount the CLI from a volume and wrap the entrypoint.

To use it, you need to install the webhook on your cluster once, which can be done in multiple ways.

Once the webook is installed, add an annotation to the pod with the container that needs to be mutated: secrethub.io/mutate: demo-app. If there’s multiple containers that need to be mutated, pass them as a comma-separated list: secrethub.io/mutate: app1,app2.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: app
  name: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      run: app
  template:
    metadata:
      labels:
        run: app
      annotations:
        secrethub.io/mutate: demo-app
    spec:
      serviceAccountName: demo-app
      containers:
      - name: demo-app
        image: secrethub/demo-app
        command: ["demo", "serve"]
        args: ["--host", "0.0.0.0", "--port", "8080"]
        env:
        - name: DEMO_USERNAME
          value: secrethub://your-username/demo/username
        - name: DEMO_PASSWORD
          value: secrethub://your-username/demo/password
        - name: SECRETHUB_IDENTITY_PROVIDER
          value: gcp
        ports:
        - containerPort: 8080

Step 8: Deploy the app

Finally, to deploy the app, run:

kubectl apply -f app.yaml

Now let’s see if it all worked. Run the following command to forward data from your local 8080 port to one of the pods in the deployment:

kubectl port-forward deployment/demo-app 8080

Visit: http://127.0.0.1:8080 and you should see the app running:

See also

• SecretHub Mutating Webhook
• Docker Guide
• GKE Workload Identity