Secrets Management for AWS EKS
This guide will show you how to provision an application running on EKS with the secrets it needs. To make life easy, you can use the demo app from the Getting Started guide or deploy your own custom app and follow along.
You will be able to grant your app access to the required secrets just by having your pods assume an IAM role, using the AWS Identity Provider.
Before you begin
Before you start using SecretHub with AWS EKS, make sure you have completed the following steps:
- Set up SecretHub on your workstation.
- Configure your AWS credentials.
- Install
kubectl
. - Install
eksctl
, the official CLI for EKS.
- Set up SecretHub on your workstation.
- Configure your AWS credentials.
- Install
kubectl
. - Configure the AWS and SecretHub providers.
- Declare the following variables:
variable "vpc_id" {
description = "The VPC the EKS cluster lives in."
}
variable "subnets" {
type = list(string)
description = "The subnets the EKS cluster lives in. A minimum of 2 AZs is required."
}
Step 1: Create Cluster
First, create a Kubernetes cluster:
eksctl create cluster --name secrethub-demo-cluster
data "aws_eks_cluster" "cluster" {
name = module.cluster.cluster_id
}
data "aws_eks_cluster_auth" "cluster" {
name = module.cluster.cluster_id
}
provider "kubernetes" {
host = data.aws_eks_cluster.cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.cluster.token
load_config_file = false
version = "~> 1.9"
}
module "cluster" {
source = "terraform-aws-modules/eks/aws"
cluster_name = "secrethub-demo-cluster"
cluster_version = "1.17"
subnets = var.subnets
vpc_id = var.vpc_id
worker_groups = [
{
instance_type = "t2.medium"
asg_max_size = 1
}
]
}
Step 2: Enable IAM Roles for Service Accounts
Next, to allow your pods to assume IAM roles and have secrets provisioned to them, enable IAM Roles for Kubernetes Service Accounts (IRSA):
eksctl utils associate-iam-oidc-provider --cluster secrethub-demo-cluster --approve
locals {
oidc_issuer = data.aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}
resource "aws_iam_openid_connect_provider" "cluster" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.cluster.certificates.0.sha1_fingerprint]
url = local.oidc_issuer
}
data "tls_certificate" "cluster" {
url = local.oidc_issuer
}
Step 3: Create IAM Role & Service Account
The first thing to set up for your app is an IAM role, along with a Kubernetes Service Account that’s coupled to it.
Use eksctl
to create a new Kubernetes service account named demo-app
, along with a corresponding IAM Role:
eksctl create iamserviceaccount \
--name demo-app \
--cluster secrethub-demo-cluster \
--attach-policy-arn <your-policy-arn> \
--approve
eksctl
requires at least 1 IAM policy to be attached to the role it creates, so if you don’t have an IAM policy for your app, go to the Create Policy page on the AWS Console and create one, e.g.:
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"sts:GetCallerIdentity"
],
"Resource": "*"
}
}
To get the name of the created IAM Role, run:
kubectl get sa demo-app -o yaml
The role’s ARN is the the value of the eks.amazonaws.com/role-arn
key.
Take note of it, as you will need it in the next step.
resource "aws_iam_role" "secrethub_demo" {
name = "SecretHubDemoEKSRole"
description = "Role for SecretHub demo app"
assume_role_policy = data.aws_iam_policy_document.demo_app_assume_role.json
}
resource "kubernetes_service_account" "demo_app" {
automount_service_account_token = true
metadata {
name = "demo-app"
annotations = {
"eks.amazonaws.com/role-arn" = aws_iam_role.demo_app.arn
}
}
}
data "aws_iam_policy_document" "demo_app_assume_role" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = ["${aws_iam_openid_connect_provider.cluster.arn}"]
}
condition {
test = "StringEquals"
variable = "${replace(local.oidc_issuer, "https://", "")}:sub"
values = ["system:serviceaccount:default:demo-app"]
}
}
}
Step 4: Create a KMS key
Next, to allow your secrets to be encrypted and decrypted, set up a KMS key.
- Go to the Create Customer Managed Key page on the AWS Console.
- Enter an alias (e.g.
MyServiceKey
) and optionally a description for the key and click Next. - Add any tags you like and click Next.
- Select any users or roles you would like as Key Administrators and click Next. Make sure your own IAM user or a role you have access is selected or that you select it on the next page as a Key User.
- Select the role created in the previous step as a Key User, set any other preferred Key Users and then click Next. A role or user you have access to should either be a Key User or a Key Administrator.
- Create the KMS key by clicking Finish.
- Take note of the id of the newly created key (e.g.
1234abcd-12ab-34cd-56ef-1234567890ab
), you’ll need it in the next step.
resource "aws_kms_key" "secrethub_auth" {
description = "KMS key to facilitate SecretHub authentication"
}
data "aws_iam_policy_document" "secrethub_auth" {
statement {
actions = ["kms:Decrypt"]
resources = [aws_kms_key.secrethub_auth.arn]
effect = "Allow"
}
}
resource "aws_iam_policy" "secrethub_auth" {
name = "SecretHubAuth"
description = "Allow SecretHub authentication using KMS"
policy = data.aws_iam_policy_document.secrethub_auth.json
}
resource "aws_iam_role_policy_attachment" "demo_app_secrethub_auth" {
role = aws_iam_role.demo_app.name
policy_arn = aws_iam_policy.secrethub_auth.arn
}
Step 5: Setup SecretHub Service Account
With the IAM role and KMS key in place, we can go ahead and create a SecretHub service account for the app.
Run the following command and you’ll be prompted for the name of the role, the id or ARN of the KMS key and the region the KMS key is in:
secrethub service aws init your-username/demo --permission read
Setting --permission read
automatically creates an access rule to give the newly created service account read access on the demo
repo.
variable "secrethub_username" {
description = "Your SecretHub username"
}
resource "secrethub_service_aws" "demo_app" {
repo = "${var.secrethub_username}/demo"
role = aws_iam_role.secrethub_demo.name
kms_key_arn = aws_kms_key.secrethub_auth.arn
}
resource "secrethub_access_rule" "demo_app" {
account_name = secrethub_service_aws.demo_app.id
dir = "${var.secrethub_username}/demo"
permission = "read"
}
As you may have noticed, secrethub service aws init
– in contrary to the generic secrethub service init
command – does not output a credential.
That’s because applications on AWS do not need it anymore: as long they take on the specified role, they can automatically get their secrets from SecretHub.
Step 6: Create Kubernetes Deployment
Next, to configure your deployment you can write a demo-app.yml
file similar to this one:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: app
name: demo-app
spec:
replicas: 1
selector:
matchLabels:
run: app
template:
metadata:
labels:
run: app
spec:
serviceAccountName: demo-app
containers:
- image: secrethub/demo-app
name: demo-app
env:
- name: DEMO_USERNAME
value: "secrethub://your-username/demo/username"
- name: DEMO_PASSWORD
value: "secrethub://your-username/demo/password"
- name: SECRETHUB_IDENTITY_PROVIDER
value: "aws"
ports:
- containerPort: 8080
The line serviceAccountName: demo-app
configures the pods to use the previously created Kubernetes service account, which has access to the required secrets.
This image runs the demo app, with the secrethub run
command and exposes it on port 8080
. You can follow our Docker guide to use a custom docker image for your app.
In the env section the environment variables DEMO_USERNAME
and DEMO_PASSWORD
(used by the demo app) are configured to reference the previously created secrets.
Finally, all you need to do is tell the app it’s running on AWS by setting the SECRETHUB_IDENTITY_PROVIDER
environment variable to aws.
This will ensure the IAM Role of the pod is used for authenticating to SecretHub.
Step 7: Use the mutating webhook
If you don’t want to include the SecretHub binary in your image, you can use the SecretHub Mutating Webhook to automatically mount the CLI from a volume and wrap the entrypoint.
To use it, you need to install the webhook on your cluster once, which can be done in multiple ways.
Once the webook is installed, add an annotation to the pod with the container that needs to be mutated: secrethub.io/mutate: demo-app
.
If there’s multiple containers that need to be mutated, pass them as a comma-separated list: secrethub.io/mutate: app1,app2
.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: app
name: demo-app
spec:
replicas: 1
selector:
matchLabels:
run: app
template:
metadata:
labels:
run: app
annotations:
secrethub.io/mutate: demo-app
spec:
serviceAccountName: demo-app
containers:
- name: demo-app
image: secrethub/demo-app
command: ["demo", "serve"]
args: ["--host", "0.0.0.0", "--port", "8080"]
env:
- name: DEMO_USERNAME
value: secrethub://your-username/demo/username
- name: DEMO_PASSWORD
value: secrethub://your-username/demo/password
- name: SECRETHUB_IDENTITY_PROVIDER
value: aws
ports:
- containerPort: 8080
Step 8: Deploy app
Finally, to deploy the SecretHub demo app, run:
kubectl apply -f demo-app.yml
Step 8: Verify that it works
Now let’s see the app running!
Run the following command to forward data from your local 8080
port to one of the pods in the deployment:
kubectl port-forward deployment/demo-app 8080
Then visit: http://127.0.0.1:8080
and you should see the app running:
