How to use SecretHub on GKE
This guide will show you how to provision an application running on Google Kubernetes Engine with the secrets it needs.
Using the SecretHub GCP Identity Provider in combination with GKE Workload Identity, your app will be able to load secrets simply by taking on a Kubernetes service account without having to deal with another key.
To make life easy, you can use the demo app from the Getting Started guide, but you can also follow along with your own app.
Before you begin
Before you start using SecretHub with GKE, make sure you have completed the following steps:
- Set up SecretHub on your workstation.
- Install
gcloud
with application default credentials configured. - Install
kubectl
and connected it to a GKE cluster.
Depending on whether you’re following allong with Terraform or the CLI, set up the following:
Export your GCP project ID to your shell, so you can just copy-paste the commands in this guide later on:
export GCP_PROJECT_ID="demo-project-1234567890"
To follow this guide using Terraform, you need to declare the following variables:
variable "gcp_project_id" {
description = "GCP project ID"
}
variable "gke_cluster" {
description = "Name of your GKE cluster, must be in `gcp_location`"
}
variable "gke_cluster_location" {
description = "GCP location where the `gke_cluster` runs"
}
variable "secrethub_repo" {
description = "SecretHub repo that contains the demo `username` and `password` secrets"
}
Then, configure the following providers:
provider "google" {
project = var.gcp_project_id
}
data "google_client_config" "provider" {}
data "google_container_cluster" "cluster" {
name = var.gke_cluster
location = var.gke_cluster_location
}
provider "kubernetes" {
load_config_file = false
host = "https://${data.google_container_cluster.cluster.endpoint}"
token = data.google_client_config.provider.access_token
cluster_ca_certificate = base64decode(data.google_container_cluster.cluster.master_auth[0].cluster_ca_certificate)
}
provider "secrethub" {
credential = file("~/.secrethub/credential")
}
Step 1: Enable workload identity on your GKE cluster
To use the GCP Identity Provider on GKE, you first need to make sure Workload Identity is enabled on your GKE cluster, so that Kubernetes service accounts can be coupled to Cloud IAM service accounts.
You can either enable it on an existing cluster or create a new cluster with Workload Identity already enabled by default.
Step 2: Create a GCP service account for your app
For every app that you deploy on GKE, it’s recommended to create a GCP Service Account that represents the app, so you can apply least privilege.
gcloud iam service-accounts create demo-app \
--display-name="SecretHub demo app"
Take note of the service account email it generates, e.g. demo-app@demo-project-1234567890.iam.gserviceaccount.com
:
gcloud iam service-accounts list \
--filter="name:demo-app" \
--format="value(email)"
Export the value to your shell, e.g.:
export GCP_SERVICE_ACCOUNT_EMAIL="demo-app@demo-project-1234567890.iam.gserviceaccount.com"
resource "google_service_account" "demo_app" {
account_id = "demo-app"
description = "SecretHub demo app"
}
Step 3: Create and link Kubernetes Service Account
To couple the GCP service account to a pod on GKE, you need to create a Kubernetes service account and link it to your GCP service account by adding an annotation that contains the service account email:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: demo-app
annotations:
iam.gke.io/gcp-service-account: ${GCP_SERVICE_ACCOUNT_EMAIL}
EOF
Next, you need to make the link in Cloud IAM too:
gcloud iam service-accounts add-iam-policy-binding \
--role="roles/iam.workloadIdentityUser" \
--member="serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[default/demo-app]" \
${GCP_SERVICE_ACCOUNT_EMAIL}
resource "kubernetes_service_account" "demo_app" {
metadata {
name = "demo-app"
annotations = {
"iam.gke.io/gcp-service-account" = google_service_account.demo_app.email
}
}
}
resource "google_service_account_iam_binding" "demo_app" {
service_account_id = google_service_account.demo_app.name
role = "roles/iam.workloadIdentityUser"
members = [
"serviceAccount:${var.gcp_project_id}.svc.id.goog[default/${kubernetes_service_account.demo_app.metadata[0].name}]",
]
}
Step 4: Create a Cloud KMS key
For encryption and decryption of your secrets, you need to create a Cloud KMS key.
Start by creating a keyring:
gcloud kms keyrings create secrethub \
--location="global"
Then, create a KMS key for your app:
gcloud kms keys create demo-app \
--location="global" \
--keyring="secrethub" \
--purpose="encryption"
Lastly, make sure the GCP service account you created earlier has decryption rights on the KMS key:
gcloud kms keys add-iam-policy-binding demo-app \
--location="global" \
--keyring="secrethub" \
--member="serviceAccount:${GCP_SERVICE_ACCOUNT_EMAIL}" \
--role="roles/cloudkms.cryptoKeyDecrypter"
resource "google_kms_key_ring" "secrethub" {
name = "secrethub"
location = "global"
}
resource "google_kms_crypto_key" "demo_app" {
name = "demo-app"
key_ring = google_kms_key_ring.secrethub.id
}
resource "google_kms_crypto_key_iam_binding" "crypto_key" {
crypto_key_id = google_kms_crypto_key.demo_app.id
role = "roles/cloudkms.cryptoKeyDecrypter"
members = [
"serviceAccount:${google_service_account.demo_app.email}",
]
}
Step 5: Create a SecretHub service account
With the KMS key and the GCP-linked Kubernetes service account in place, you can now create a SecretHub service account:
secrethub service gcp init your-username/demo \
--description demo-app \
--service-account-email ${GCP_SERVICE_ACCOUNT_EMAIL} \
--kms-key "projects/${GCP_PROJECT_ID}/locations/global/keyRings/secrethub/cryptoKeys/demo-app" \
--permission read
The (optional) --permission read
flag will grant the service read access to every secret in the repo.
For more granular access, see the acl set
command.
resource "secrethub_service_gcp" "demo_app" {
repo = var.secrethub_repo
service_account_email = google_service_account.demo_app.email
kms_key_id = google_kms_crypto_key.demo_app.id
}
resource "secrethub_access_rule" "demo_app" {
account_name = secrethub_service_gcp.demo_app.id
dir = var.secrethub_repo
permission = "read"
}
Do note that to be able to run this, you need to have GCP credentials present with encryption rights on the KMS key you pass in.
After running it, you may have noticed that unlike the usual SecretHub service accounts, the SecretHub GCP service account does not output a credential.
That’s because apps running on GCP don’t need it anymore: as long they take on the specified service account on GCP, they can automatically load their secrets from SecretHub.
Step 6: Create Kubernetes Deployment
Next up, create a pod specification for your app (e.g. app.yaml
):
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: app
name: demo-app
spec:
replicas: 1
selector:
matchLabels:
run: app
template:
metadata:
labels:
run: app
spec:
serviceAccountName: demo-app
containers:
- name: demo-app
image: secrethub/demo-app
env:
- name: DEMO_USERNAME
value: secrethub://your-username/demo/username
- name: DEMO_PASSWORD
value: secrethub://your-username/demo/password
- name: SECRETHUB_IDENTITY_PROVIDER
value: gcp
ports:
- containerPort: 8080
In the env
stanza the environment variables DEMO_USERNAME
and DEMO_PASSWORD
(needed by the demo app) are configured to reference the secrets that need to be loaded.
As you can see, there is no extra key that needs to be managed to be able to load secrets.
Assigning the serviceAccountName
and setting SECRETHUB_IDENTITY_PROVIDER
to gcp
is all that’s needed on the pod-side to authenticate.
The secrethub/demo-app
image already includes the SecretHub CLI and secrethub run
entrypoint, so the app is portable across different platforms.
If you don’t want to include it in the image, you can use the SecretHub Mutating Webhook.
Step 7: Use the mutating webhook
If you don’t want to include the SecretHub binary in your image, you can use the SecretHub Mutating Webhook to automatically mount the CLI from a volume and wrap the entrypoint.
To use it, you need to install the webhook on your cluster once, which can be done in multiple ways.
Once the webook is installed, add an annotation to the pod with the container that needs to be mutated: secrethub.io/mutate: demo-app
.
If there’s multiple containers that need to be mutated, pass them as a comma-separated list: secrethub.io/mutate: app1,app2
.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: app
name: demo-app
spec:
replicas: 1
selector:
matchLabels:
run: app
template:
metadata:
labels:
run: app
annotations:
secrethub.io/mutate: demo-app
spec:
serviceAccountName: demo-app
containers:
- name: demo-app
image: secrethub/demo-app
command: ["demo", "serve"]
args: ["--host", "0.0.0.0", "--port", "8080"]
env:
- name: DEMO_USERNAME
value: secrethub://your-username/demo/username
- name: DEMO_PASSWORD
value: secrethub://your-username/demo/password
- name: SECRETHUB_IDENTITY_PROVIDER
value: gcp
ports:
- containerPort: 8080
Step 8: Deploy the app
Finally, to deploy the app, run:
kubectl apply -f app.yaml
Now let’s see if it all worked.
Run the following command to forward data from your local 8080
port to one of the pods in the deployment:
kubectl port-forward deployment/demo-app 8080
Visit: http://127.0.0.1:8080
and you should see the app running:
