Announcing Secrets Management for Terraform

Usage of secrethub_secret Terraform resource

At SecretHub we love Terraform. We use it for everything that has even a hint of repetitiveness, from our entire AWS account to GitHub repositories and Stripe configurations. However, as with every love affair, the sweatpants come out and the makeup takes a hike once the honeymoon is over.

If you’re like us and have used Terraform in production for a while, you’ve likely noticed that managing secrets in Terraform can be a pain:

  • Secrets are defined in plaintext code that is checked into version control.
  • Secrets end up in your .tfstate file, making it hard to share securely.
Secrets, secrets everywhere meme
Secrets sprawl everywhere when you use Terraform in production...

While version control systems are usually protected pretty well, they have not been created with the intention of providing the highest level of security for your secrets. Version control focuses on sharing information, not keeping it secret.

As it turns out, we’re not alone. There’s a whole bunch of people that feel the same pain and have come up with creative ways to work around it, including manually editing the .tfstate file.

There are many ways to skin this cat, but today we would like to share with you a new way to keep secrets out of Terraform projects. One that doesn’t involve tedious workarounds and gives you all the security features you need straight out of the box: encryption, versioning, access controls, and audit logs.

In short, you’ll see you how to:

👋 Goodbye secret .tfvars files

The most common way to keep secrets out of version control is to define them as variables in a separate .tfvars file:

variable "db_password" {}

resource "heroku_app" "your_app" {
  name   = "your-app"
  region = "us"

  sensitive_config_vars = {
    DB_PASSWORD = var.db_password

The secret variables can now be sourced from a secret .tfvars file that is added to your .gitignore and typically contains all the secrets for your Terraform project:

db_password = "yoursecretpassword"

As you can see, no more hardcoded secrets in the code itself. However, this creates its own set of problems:

  • How do you share and synchronize the secret .tfvars files securely out of band with your team?
  • When someone leaves, how do you revoke access and how do you update secrets?
  • How do you track down who accessed which secrets at what point in time?
  • You’re one .gitignore mistake away from checking your secrets into version control, which happens to the best of us.

With companies usually accumulating many Terraform projects over time, these issues only get worse.

That’s why we’ve created a Terraform provider that lets you inject secrets into Terraform code simply by referencing a path on SecretHub.

What is SecretHub?

SecretHub enables you to define a secret once, encrypt it locally and store it on a centralized server that never sees your plaintext secrets. Then it allows you to inject secrets securely into applications at runtime, removing the limitations and risks of placing them in static configuration code.

Terraform Provider

To create secrets and store them on SecretHub, you can use the secrethub_secret resource, either with the generate option to auto-generate a value or using the value option to pass the secret value in directly:

resource "secrethub_secret" "db_password" {
  path = "your-org/your-repo/prod/db/password"

  generate {
    length = 22

resource "secrethub_secret" "db_user" {
  path  = "your-org/your-repo/prod/db/user"
  value = "mysqluser"

You pass them into other resources:

resource "aws_db_instance" "your_db" {
  allocated_storage    = 10
  storage_type         = "gp2"
  engine               = "mysql"
  engine_version       = "5.7"
  instance_class       = "db.t2.micro"
  name                 = "your-db"
  username             = secrethub_secret.db_user.value
  password             = secrethub_secret.db_password.value
  parameter_group_name = "default.mysql5.7"

To load these secrets into your apps outside of Terraform, you can use one of SecretHub’s injection mechanisms that best suits your tool.

Literal secrets like a Stripe API Secret Key are best written to SecretHub using the SecretHub CLI on the moment of obtaining it:

$ secrethub write your-org/your-repo/prod/stripe/secret_key
Please type in the value of the secret, followed by an [ENTER]:

These secrets can be accessed by using the secrethub_secret data source:

data "secrethub_secret" "stripe_secret_key" {
  path = "your-org/your-repo/prod/stripe/secret_key"

Putting it all together, a typical setup can look like this:

variable "environment" {}

locals {
  secrethub_dir = "your-org/your-repo/${var.environment}"

resource "secrethub_secret" "db_password" {
  path = "${local.secrethub_dir}/db/password"

  generate {
    length = 22

resource "secrethub_secret" "db_user" {
  path  = "${local.secrethub_dir}/db/user"
  value = "mysqluser"

data "secrethub_secret" "stripe_secret_key" {
  path = "stripe/secret_key"

resource "heroku_app" "your_app" {
  name   = "your-app"
  region = "us"

  sensitive_config_vars = {
    DB_PASSWORD       = secrethub_secret.db_password.value
    DB_USER           = secrethub_secret.db_user.value
    STRIPE_SECRET_KEY = data.secrethub_secret.stripe_secret_key.value

You can run and share Terraform code without ever having to see actual secrets. Encryption, versioning, access control and audit logs are all provided automatically.

Share .tfstate files securely

Now that you know how to keep secrets out of Terraform code, the next place where secrets end up is the .tfstate file.

Maintaining an entirely secretless .tfstate file is practically unavoidable because of the way Terraform is built. Terraform always needs to keep track of the configured state and secrets are a vital part of that configuration. That’s why most providers write all configuration to the .tfstate file, including sensitive configuration values like your database password.

One does not simply keep secrets out of .tfstate files meme
Secrets inevitably end up in the .tfstate file, making it a secret too.

So what does this mean for you? It basically means that you have to treat your .tfstate file as a secret as well.

A common way AWS users deal with this is to set up an S3 bucket, enable versioning and add KMS on top for encryption at rest. Then you’d have to carefully craft strict IAM policies to control access to the .tfstate and all the secrets contained inside. To be able to track down who has read the .tfstate file at what point in time, you would configure another service like CloudTrail. Finally, you would need to set up DynamoDB to get state locking so you don’t all end up corrupting the .tfstate file when concurrent writes happen.

Obviously, this requires quite some work and largely depends on which infrastructure you run and toolchain you’re used to.

That’s why we’ve created a state backend for Terraform that simply writes your .tfstate to SecretHub as well. Just like any other secret, this means you get all the security features that your .tfstate deserves without any additional setup.

Terraform State Backend

To use SecretHub as a state backend, run the SecretHub HTTP Proxy in a Docker container:

docker run -p --name secrethub -v $HOME/.secrethub:/secrethub secrethub/http-proxy

You can then configure your Terraform project’s backend settings to use a local HTTP backend:

terraform {
  backend "http" {
    address = "http://localhost:8080/v1beta/secrets/raw/your-org/your-repo/terraform.tfstate"

That’s all there is to it. With just a couple of lines of configuration your .tfstate is stored securely on a remote backend. Encryption, versioning, access control and audit logs are all provided.

Note: Terraform’s provider ecosystem is very pluggable, but unfortunately Terraform backends are not. The only way to make it somewhat pluggable is through an HTTP backend.

We are also working on a native state backend, but it currently requires you to use our fork of Terraform so we’re sticking with the more pluggable HTTP backend for now.

Keep provider credentials out of the .tfstate

Nearly every Terraform provider out there needs a set of secrets to configure it. Now I can practically hear you thinking: why not use the secrethub_secret data source for that too?

And you’re absolutely right… most of the time. Sometimes, provider credentials are specific to the person or machine executing the Terraform code. When that happens, you may want to exclude the provider credentials from your .tfstate altogether.

While Terraform generally stores resources and data sources in the .tfstate, it automatically excludes provider configuration from the .tfstate. To take advantage of this, but still be able to manage the credentials with SecretHub, you can leverage the run command of the SecretHub CLI and source Terraform variables from the environment instead.

Consider the provider configuration below:

variable "aws_access_key" {}

variable "aws_secret_key" {}

provider "aws" {
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
  region     = var.aws_region

To source those Terraform variables from SecretHub, create an secrethub.env file with environment variables prefixed with TF_VAR and use SecretHub’s template syntax to point to a secret on SecretHub:

TF_VAR_aws_access_key = {{ your-org/your-repo/prod/aws/access-key }}
TF_VAR_aws_secret_key = {{ your-org/your-repo/prod/aws/secret-key }}

Then, wrap your Terraform command in the secrethub run command:

secrethub run -- terraform apply

On every run, the SecretHub CLI fetches your secrets, decrypts them and injects them as ephemeral environment variables only for the terraform apply process. After the process has completed, the local copies of your secrets are gone.

Of course, you are not limited to TF_VAR variables and can use the same approach to inject any secrets into the environment you wish.

Eliminate secret sprawl in CI/CD

Leveraging SecretHub’s Terraform integration shrinks your external Terraform variables down to just a single value: the SecretHub credential.

This greatly reduces secret sprawl when running Terraform projects in a CI/CD environment. For all your application secrets you get access control, secure sharing, audit logs and versioning.

To run Terraform with SecretHub in a CI/CD environment, we recommend you use a service account instead of your personal account. You can use the SecretHub CLI to create a new service account for your CI:

secrethub service init your-org/your-repo --description circleci --permission read

The credential that it outputs is the only secret your CI needs, which you can configure with a protected environment variable SECRETHUB_CREDENTIAL:


Next steps

So to wrap up, you’ve seen a way to handle secrets in Terraform that gives you:

  • No more secrets defined in plaintext code.
  • No more secret .tfvars files that you need to manage securely out of band.
  • No more hassle setting up a .tfstate backend with all security features it needs.
  • No more keeping track of all your personal provider credentials.
  • No more CI environments polluted with tons of secret variables.

Instead, you can sit back and enjoy encryption, versioning, access control and audit logs straight out of the box.

Enough with the sales pitch, you make up your own mind. Kick the tires for a bit and let us know what you think. To help you get started, have a look at these detailed step-by-step guides: