Hiding Secrets in Terraform

Terraform is a popular tool amongst DevOps folks that allows you to set up your infrastructure as code. It is powerful and fairly easy to get to grips with.

One of the features that makes it really nice is you can make changes and run a check to see what physical changes your code changes will make. By default Terraform will make the least amount of changes it can get away with.

This means if you are setting up 5 different AWS things (some EC2 instances, an Elasticache instance, and maybe some S3 buckets) and the change you make only effects one of the EC2 instances, then Terraform isn’t going to touch your Elasticache stuff.

Local tfstate stores sensitive information

How Terraform goes about the business of comparing the changes in code to the changes in your infrastructure is that it generates what it calls a tfstate.

This is just a set of files that state exactly what is currently running, if what your code would generate changes what is running, it will know that is a thing that needs to be changed.

Unfortunately, in order to set up most of these services you need usernames and passwords to be set - and since you can potentially change these passwords via Terraform then it stands to reason that Terraform is going to need be able to compare your old credentials with possible new ones.

To facilitate this it stores all settings, including usernames, passwords, port numbers and literally everything else in these tfstate files, in plain text.

This wasn’t something I’d have expected as the default behaviour. The documentation does suggest that you use a thing called Remote State (more on that later)

Terraform also puts some state into the terraform.tfstate file by default. This state file is extremely important; it maps various resource metadata to actual resource IDs so that Terraform knows what it is managing. This file must be saved and distributed to anyone who might run Terraform. It is generally recommended to setup remote state when working with Terraform. This will mean that any potential secrets stored in the state file, will not be checked into version control

But that wasn’t always the case, an earlier version of the documentation had the wording;

Terraform also put some state into the terraform.tfstate file by default. This state file is extremely important; it maps various resource metadata to actual resource IDs so that Terraform knows what it is managing. This file must be saved and distributed to anyone who might run Terraform. We recommend simply putting it into version control, since it generally isn’t too large.

No mention of doing anything special to remove secrets from this file.

Lots of people have credentials in Github

Even though using Remote State is now considered the way to proceed, it wasn’t always and as a result quite a few folk have their Terraform tfstate files floating around public Github Repositories.

I’m not the only person to be surprised by this, in fact there has been an open ticket since 2014 with people stating their surprise and sharing how they’ve worked around the issues.

What follows is me distilling a lot of the great information in that ticket into some things you should consider, and adding my own personal opinion to each suggestion.

I’ve already spoiled the ending by sharing that Remote State is the way forward, but there is every chance that the solution you need doesn’t require Remote State, or for some reason precludes you from using it.

Use a dummy password on create and update after

One solution is to only ever use a dummy password and then execute some at a later date which will update the password to the one you actually care about.

Here is a sample script showing how you could do that;

You could even have the code generate a random password to use to update these credentials and then have the random password securely stored somewhere.

  resource "aws_db_instance" "my_database" {
    password = "temp_password"

    provisioner "local-exec" {
      command = "bash -c 'DBPASS=$$(openssl rand -base64 16); aws rds modify-db-instance --db-instance-identifier ${self.id} --master-user-password $${DBPASS} --apply-immediately"
    }
  }

Use something like Git Crypt

Several folk in that Github Issue had suggested Git Crypt. I think this is a lovely solution for the more general problem of “I have some files which I would quite like to version control but would really rather not have their contents exposed to the world by mistake”.

From their README;

git-crypt enables transparent encryption and decryption of files in a git repository. Files which you choose to protect are encrypted when committed, and decrypted when checked out. git-crypt lets you freely share a repository containing a mix of public and private content. git-crypt gracefully degrades, so developers without the secret key can still clone and commit to a repository with encrypted files. This lets you store your secret material (such as keys or passwords) in the same repository as your code, without requiring you to lock down your entire repository.

I’m not sure why you would favour this over Remote State, but I’m certainly going to keep this project in mind for future use.

Use an extra tool like Terrahelp

Someone has written a Go Terraform CLI tool which can encrypt/decrypt .tfstate files.

Personally I feel like you would only use this now if it offered other things to make Terraforming your project a lot easier. At the time it was originally created I imagine it was a godsend though!

Use Remote State

Remote State allows Terraform to store the state in a remote store. Terraform supports storing state in places like Terraform Enterprise, Consul, S3, and more.

Here are the docs for setting up S3 as a remote state.

In a project I recently implemented this one there was only one bit of setup I had to do, one code addition I had to make and then I had to run one command.

The setup was creating a bucket on AWS S3, it should not be readable or writeable by anyone accept the user you will be using for Terraform.

The code I added was;

  terraform {
    backend "s3" {
      bucket = "my-new-bucket"
      key = "state/key"
      region = "eu-west-1"
    }
  }

This simply tells Terraform to use S3 as the backend provider for doing things like storing tfstate files.

Finally I ran terraform init which was a requirement because Terraform had noticed that I had changed from storing locally to storing in S3.

Once that was done I could delete my local tfstate files safe in the knowledge my details were safely stored on S3.