How secret are your secrets?

15.03.2024 — devops, tech, terraform, iac, feature — 3 min read

WHAT DO YOU MEAN?

Terraform is a trusted IaC tool borned 9 years ago by Hashicorp, which was open-sourced until they feels greedy and change its license to a non-open source in 2023.

Recently, during a casual beer session with my colleagues, we have a debate on how safe our secret are when we referring it inside terraform code base.

There are some gray areas around how Terraform manage secrets, and whether or not TF save the provided secrets in it state files (tfstate). Some naively believes that TF keeping that secrets away from the tfstate, and some doubts TF would store them in plain-text. My optimistic view of Terraform led me to believe that it would handle secrets responsibly—perhaps by saving only reference IDs to the actual resource storing the secret value (such as SSM parameter names or Secret Manager IDs).

To put my assumptions to the test, I promptly set up a Terraform workspace and conducted a simple experiment. The results helped solidify my understanding.

Case 1:

Terraform version: 1.4.6

AWS provider: 4.67.0

Method: Using 2 SSM parameters to store user and SSH private key

Please note that SSM parameter is created separately from IaC.

1data "aws_ami" "al2023_latest" {
2  most_recent = true
3
4  filter {
5    name   = "name"
6    values = ["al2023-ami-2023*x86_64"] #al2023-ami-2023.3.20231218.0-kernel-6.1-x86_64 
7  }
8
9  filter {
10    name   = "virtualization-type"
11    values = ["hvm"]
12  }
13
14  owners = ["137112412989"] # Canonical
15}
16
17data "aws_ssm_parameter" "private_key" {
18  name = "/devbox/private_key"
19}
20
21
22data "aws_ssm_parameter" "user" {
23  name = "/devbox/user"
24}
25
26data "template_file" "init_script" {
27  template = templatefile("../../scripts/init_tmpl.sh",
28    {
29      user             = data.aws_ssm_parameter.user.value,
30      user_ssh_pub_key = data.aws_ssm_parameter.private_key.value
31    }
32  )
33}
34
35resource "aws_instance" "devbox_sandbox" {
36  ami                         = data.aws_ami.al2023_latest.id
37  availability_zone           = "ap-southeast-1a"
38  ebs_optimized               = false
39  instance_type               = "t3a.micro"
40  monitoring                  = false
41  key_name                    = "my-devbox-key"
42  subnet_id                   = "subnet-abcdef"
43  vpc_security_group_ids      = ["sg-abcdef"]
44  associate_public_ip_address = true
45  source_dest_check           = true
46  iam_instance_profile        = "my-devbox-role"
47  root_block_device {
48    volume_type           = "gp3"
49    volume_size           = 30
50    delete_on_termination = true
51  }
52
53  user_data = data.template_file.init_script.rendered
54  tags = {
55    "Name" = "my-devbox_sandbox"
56  }
57}

And after planning & applying, here is the interesting part. Once Terraform is allowed to retrieve the secret value (via provided credentials), it will store that value in the tfstate. Depending on how your backend is set up, the tfstate will either be locally stored or an S3 object. If you examine the contents of the tfstate, you’ll find that my private key, which was previously used for setting up SSH to my devbox, is stored in plain text.

Not believing my eyes, I continued with another experiment, this time using Secret Manager, in the hope of a different outcome.

Case 2:

Terraform version: 1.4.6

AWS provider: 4.67.0

Method: Using Secret Manager to store a key-value object that contain user and ssh private key

Please note that Secret is created separately from IaC.

1data "aws_ami" "al2023_latest" {
2  most_recent = true
3
4  filter {
5    name   = "name"
6    values = ["al2023-ami-2023*x86_64"] #al2023-ami-2023.3.20231218.0-kernel-6.1-x86_64 
7  }
8
9  filter {
10    name   = "virtualization-type"
11    values = ["hvm"]
12  }
13
14  owners = ["137112412989"] # Canonical
15}
16
17data "aws_secretsmanager_secret_version" "devbox" {
18  secret_id = "secretmanager/devbox"
19}
20
21data "template_file" "init_script" {
22  template = templatefile("../../scripts/init_tmpl.sh",
23    {
24      user             = jsondecode(data.aws_secretsmanager_secret_version.devbox.secret_string)["user"],
25      user_ssh_pub_key = jsondecode(data.aws_secretsmanager_secret_version.devbox.secret_string)["private_key"]
26    }
27  )
28}
29
30resource "aws_instance" "devbox_sandbox" {
31  ami                         = data.aws_ami.al2023_latest.id
32  availability_zone           = "ap-southeast-1a"
33  ebs_optimized               = false
34  instance_type               = "t3a.micro"
35  monitoring                  = false
36  key_name                    = "my-devbox-key"
37  subnet_id                   = "subnet-abcdef"
38  vpc_security_group_ids      = ["sg-abcdef"]
39  associate_public_ip_address = true
40  source_dest_check           = true
41  iam_instance_profile        = "my-devbox-role"
42  root_block_device {
43    volume_type           = "gp3"
44    volume_size           = 30
45    delete_on_termination = true
46  }
47
48  user_data = data.template_file.init_script.rendered
49  tags = {
50    "Name" = "my-devbox_sandbox"
51  }
52}

Very similar approach from the previous test case, I planned and applied, with expectation of a more secure bahaviour from both Terraform and AWS. Yet, I looked at the exact same issue, lying there is a plain text private key.

This question has been lingering in the back of my mind for quite some time, and I’ve been procrastinating on figuring it out. Surprisingly, it turned out not to be as complicated as I had imagined.After spending around 30 minutes searching, I discovered that I’m not the only one with this question. There’s an issue that was opened—guess when?—almost 9 years ago, discussing this very topic and still no official response from Hashicorp 🤔.

So, it is a expected behaviour from terraform that every sensitive credentials/secrets that is referred by using data_source, can and will be stored plain text in tfstate file after successful execution.

CONCLUDE

Even though Terraform has not explicitly addressed this issue, there are numerous ways to secure our sensitive data. Let’s explore some solutions:

1. Configure Remote Storage for tfstate:

The most straightforward approach is to use remote storage for your Terraform state (e.g., Amazon S3). Ensure that the S3 object is protected with encryption and a bucket policy that restricts access via IAM roles and trusted identities. Only authorized administrators and engineers should have access to this file.

2. Separate Procedure for Secret Management:

Instead of hard-coding secrets directly into your Terraform code, consider using a separate procedure to set or update secret values. For example, when creating a new EC2 instance (as mentioned earlier), avoid pre-rendering the private key in the user_data using Terraform. Instead, use a bash script to fetch and update the secret once the instance is successfully launched. Grant the necessary permissions to access the secret (e.g., an SSM parameter) via the instance profile.

In all honesty, this issue doesn’t spell disaster for your infrastructure. However, it’s an intriguing topic, and I hope my article sheds light on it.