Decouple Application Secrets from Your CI/CD Pipeline

Pipeline without application secrets in Git (GitHub, GitLab, Bitbucket, etc.), CI (Travis CI, CircleCI, Jenkins, etc.), Artifact repos (Docker Hub, JFrog, S3, etc.), Orchestrators (Kubernetes, ECS, Nomad, etc.)

The pipelines we set up to deliver the software we create — consisting of infrastructure as code, configuration management, CI services, orchestrators, and more — tend to share a fundamental flaw: They pass application secrets down the delivery pipeline, and along the way leave traces of your most critical infrastructure secrets.

Secrets end up in all sorts of JSON files, container definitions, jobspecs, ConfigMaps, and much more. Before you know it, they’re somewhere in an S3 bucket, in Consul, etcd, or Redis, or they get written to a forgotten EBS drive. You’ll have secrets lying around with hardly any access rules, without any auditability, and most of the time in plaintext!

You’re just one kubectl describe, aws ecs describe-task-definition, or docker inspect away from your plaintext secrets. Having commands like these surely is great for debugging your deployments, but it also shows how these tools are not really built for keeping secrets secret.

Pipeline with secrets sprawl
Secrets end up throughout the pipeline, in places varying from version-controlled files to CI environment variables, and sometimes even in build artifacts.

While there are ways to reduce or patch potential leakage, in practice, you’ll always be playing catch-up. If you include a new tool in your pipeline, you’ll also need to familiarize yourself with a new environment to secure.

These methods often take more work and specific knowledge than necessary, are prone to error (making an S3 file public is just a single tick away), or are just not up to par with the proper security standards for storing secrets.

Plus, it’s not always clear right off the bat what these tools do with your secrets under the hood. If you’re on Kubernetes, you are aware that any ConfigMap or Secret resource just gets stored on etcd, right?

This automatically means that — whether you like it or not — etcd becomes a treasure trove of secrets. Have you taken the necessary measures to treat it as one?

Our Solution

Postpone the moment of exposing secrets until your app actually needs them, instead of carrying them along the journey of your entire pipeline.

The only universally applicable and sustainable solution to really make sure you don’t spill your application secrets at some point along the way is to introduce them all the way at the end of the application delivery: at runtime.

Postpone the moment of exposing secrets until your app actually needs them, instead of carrying them along the journey of your entire pipeline.

So how does this work in practice? From your Git repo, to your CI system, your build artifacts, and all the way down to your orchestrator, only use references to your secrets. Store the actual values in a secure and central place, and let your app fetch them the moment it starts.

To help you implement this concept in your existing workflow – without making you go back to the drawing board to reimagine everything you’ve already built – we’ve created secrethub run.

Pipeline free of secrets
A pipeline free of secrets.

Say that you have a Node.js app that needs to connect to a Postgres database and to a third-party payment service like Stripe.

Assuming you’ve written the values to SecretHub, you’ll need to tell secrethub run which secrets your app needs. To do this, create a secrethub.env file and use SecretHub paths instead of the actual values:

POSTGRES_PASSWORD = {{ company/app/prod/postgres/password }}
STRIPE_SECRET_KEY = {{ company/app/prod/stripe/secret_key }}

Notice how this file is free of secret values, so you can safely check it into source control.

Then, wrap your app’s launch command in secrethub run:

secrethub run -- node app.js

Every time your app starts, the actual secret values behind the specified paths get fetched, decrypted, and placed into environment variables that are only bound to the child process.

This means your orchestrator cannot get to them anymore, and neither can docker inspect.

It also means that when the app exits, there are zero local copies of your secrets anywhere in your infrastructure. On each consecutive run, they have to be fetched and decrypted again.

This in turn adds a new entry to the audit log on SecretHub, showing that the secret has been read by a specific app or person.

Note: Environment variables can still be read from /proc/<pid>/environ, but you’d have to be the process owner or have root access on the host.

Same Code, Different Secrets

secrethub run is built for supporting multiple environments. We highly recommend you structure your SecretHub secrets in the same way across all your environments: e.g. app/dev/db/password and app/prod/db/password.

If you do so, you can easily switch to a different set of secrets through the use of variables in your secrethub.env file. You could have variables for your environment, stage, region, tenant, or anything else you see fit:

POSTGRES_PASSWORD = {{ company/app/$environment/postgres/password }}
STRIPE_SECRET_KEY = {{ company/app/$environment/stripe/secret_key }}

Or, if you prefer a higher level of abstraction (or maybe long paths just aren’t really your thing):

POSTGRES_PASSWORD = {{ $app/postgres/password }}
STRIPE_SECRET_KEY = {{ $app/stripe/secret_key }}

You can then set these variables as arguments on the command line:

secrethub run -v environment=prod -- node app.js

Alternatively, you can use environment variables prefixed with SECRETHUB_VAR:

SECRETHUB_VAR_environment=prod secrethub run -- node app.js

If you follow this principle, switching from dev to prod can consist of a single variable change.

An Extra 4 MB Download, but Zero Code Changes

Yes, to use secrethub run, you do need to install an extra binary in your container. But don’t worry: It’s a single lightweight binary, it has zero dependencies, and it can easily be installed (e.g. using apk).

We’ve also made sure that you don’t have to change your app code to get proper secrets management. Instead, your app just reads out environment variables, without knowing anything about SecretHub (though if you want to, you’re still welcome to integrate the SecretHub client SDK 🙂).

Plug the Leak in Your Logs

Using secrethub run comes with a nice extra perk: It keeps an eye on your stdout and stderr in case your app accidentally logs any secrets, and it will redact those values for you:

$ secrethub run -- node app.js
[INFO] Launching app...
[INFO] Connecting to db with config: {
   host: localhost
   port: 5432
   user: postgres
   password: <redacted by SecretHub>

This way, you’re keeping secrets not only out of your software delivery pipeline, but also out of your logging systems.

All the Way Down 🐢🐢🐢…

To authenticate to SecretHub and decrypt the fetched secrets, your apps are going to need a SecretHub service credential. Even though you’ve freed your pipeline from application secrets, there is now a new secret that has replaced it. What to do with that extra turtle?

Fortunately, we’ve come up with a fantastic way to avoid having to deal with the extra turtle, thereby solving the bootstrap problem!

In the next blog post, we will cover how to effortlessly and securely accomplish turtleless secrets management on AWS by leveraging STS and KMS under the hood.