Automate ephemeral workspaces in HCP Terraform using the TFE provider

Over the last several years, organizations worldwide have adopted Terraform as their primary tool for cloud infrastructure provisioning. To make the management and adoption of Terraform accessible, so that organizations can focus on building their applications, HashiCorp released HCP Terraform that handles the heavy lifting for scaling via secure state file management, CI/CD, and cross-stack resource references. Of all the new logical constructs created to enable an organization’s HCP Terraform adoption, such as organization, project, and workspaces, in this note, we’ll be exploring workspaces, and particularly ephemeral workspaces.
This note extends the concepts from my previous HCP Terraform notes. If you’re new to HCP Terraform, I recommend reading these foundational posts first:

Provision AWS Resources with GitHub and HCP Terraform– Learn the fundamentals of HCP Terraform, including creating organizations, projects, and workspaces via the UI. This post covers the essential GitHub VCS integration and demonstrates the complete workflow from repository setup to AWS resource provisioning.

Getting Started with Ephemeral Workspaces in HCP Terraform– Discover how to configure ephemeral workspaces manually through the HCP Terraform console, including auto-destroy settings and activity-based decommissioning.

While the previous notes focused on manual setup via the UI, this note takes an infrastructure-as-code approach, automating the entire lifecycle of the ephemeral workspace using the TFE provider. I divided this note into two parts that, together, demonstrate the creation and use of ephemeral workspaces.

In part 1, I used the TFE provider to programmatically create and configure an ephemeral workspace, which includes:
– AWS OIDC authentication setup,
– HCP Terraform workspace creation, and
– automation with GitHub Actions integration.

In part 2, I used the workspace to deploy actual AWS cloud infrastructure resources. Here I focused on:
– How to use ephemeral workspaces, and
– How ephemeral workspace automatically destroys resources.

The complete code is available in my GitHub repository: kunduso/terraform-ephemeral-workspace. The repository is organized with the 01-workspace-management folder containing all the code for Part 1 (workspace creation and automation), and the 02-infrastructure folder containing the code for Part 2 (deploying actual AWS infrastructure using the ephemeral workspace).

Before we proceed with Part 1 (workspace automation), ensure you meet the following prerequisites.
Prerequisite-1: An active HCP Terraform organization – You’ll need an HCP Terraform account with a Plus or Enterprise plan to create an ephemeral workspace.
Prerequisite-2: An AWS account – You’ll need an active AWS account with permissions to create IAM roles and policies.
Prerequisite-3: AWS IAM Role for GitHub Actions – An IAM role with OIDC trust policy that allows GitHub Actions to securely assume the role and create AWS cloud resources without long-lived credentials. For detailed setup instructions, please check: Securely integrate AWS credentials with GitHub Actions using OpenID Connect. I followed the same note and stored the role ARN as a GitHub Actions secret named IAM_ROLE, which is referenced in the GitHub Actions workflow file.

Part 1: Automating Ephemeral Workspace Creation

1. Create an HCP Terraform API Token
An HCP Terraform API token is a secure credential that allows external tools and scripts to authenticate with the HCP Terraform API to manage resources programmatically. This token enables the TFE provider to create and configure workspaces, projects, and other HCP Terraform resources on your behalf, making it essential for infrastructure-as-code automation workflows. To create a new token:

1.1 Log in to your HCP Terraform organization
1.2 Navigate to Manage -> Settings -> Security -> API tokens
1.3 Click “Create a team token”
1.4 Select a Team (admins), give your token a description (e.g., “Ephemeral Workspace Management”), and select the expiration
1.5 Copy and securely store the token as a GitHub Actions secret. I named it TF_API_TOKEN in my GitHub repository and referenced it in the GitHub Actions workflow file.

2. Set up GitHub VCS Integration in HCP Terraform
GitHub VCS integration in HCP Terraform automatically syncs workspaces with GitHub repositories, enabling GitOps workflows in which code commits trigger infrastructure changes. This integration connects HCP Terraform to your GitHub account via OAuth, allowing it to pull Terraform configurations from specified repositories, folders, and branches, and optionally trigger runs when changes are detected. To create the integration:

2.1 In your HCP Terraform organization, go to Settings → VCS Providers
2.2 Click “Add VCS Provider” and select “GitHub”
2.3 Follow the OAuth setup process to authorize HCP Terraform access to GitHub.

Once connected, note the OAuth Token ID from the VCS provider settings. This OAuth Token ID will be used in the oauth_token_id variable. For a detailed explanation, please check the official guide from Hashicorp: Set up GitHub.com OAuth VCS provider. I stored the token as a GitHub Actions secret named OAUTH_TOKEN_ID.

3. Configure AWS OIDC Provider for HCP Terraform
AWS OIDC (OpenID Connect) provider enables HCP Terraform to authenticate with AWS using temporary credentials instead of long-lived access keys. This setup creates an IAM role associated with the ephemeral workspace, allowing it to provision AWS resources. I followed the recommendation from: implementing access management for your AWS organization and added the code to oidc_iam.tf file.

4. Create an HCP Terraform Project
Before creating the ephemeral workspace, we need to set up an HCP Terraform project to host it. I used the TFE provider to create the project programmatically. Here is the code snippet to create a project:

5. Create the ephemeral workspace
With the project created, we can now create the ephemeral workspace using the TFE provider. Below is the code snippet to create an ephemeral workspace with auto-destroy configuration:

This tfe_workspace resource creates an ephemeral workspace with several key configurations. The organization and project_id are required to associate the workspace. Setting the auto_destroy_activity_duration property or auto_destroy_at (not shown in this repository) makes the workspace an ephemeral workspace. The duration can be configured to a value based on your needs – in this example, I used “1h” for demonstration purposes. For complete configuration options, check the official docs at Terraform registry tfe provider.

6. Configure GitHub Actions CI/CD Pipeline
To automate the deployment process, I set up a GitHub Actions workflow that runs on code pushes to the 01-workspace-management folder. Some salient features of the pipeline are:

– Triggers on push to any branch, but runs terraform apply only when the branch is main.
– Configures Terraform CLI with HCP Terraform authentication using the cli_config_credentials_token parameter, which automatically creates the necessary credentials file for the TFE provider to authenticate with HCP Terraform API
– Creates a terraform.auto.tfvars file dynamically during the workflow run, injecting the HCP Terraform API token as the tfe_token variable for use within the Terraform configuration
– Configures AWS credentials using OIDC authentication with the aws-actions/configure-aws-credentials action, which assumes the IAM role created earlier without requiring long-lived AWS access keys. The action uses the role-to-assume parameter to specify the IAM role ARN and establishes a secure, temporary session for AWS resource management.

Check the terraform.yml GitHub Actions workflow stored in this repository for details.
Once the code was merged into the main branch and deployed using the GitHub Actions pipeline, I checked the workspace-manager workspace in the organization and found that the last run was successful.

If you are wondering what a workspace-manager is, since I did not discuss it earlier, this is the management workspace that executes the TFE provider code to create ephemeral workspaces. It’s a persistent workspace that serves as the control plane for managing the lifecycle of ephemeral workspaces, while the ephemeral workspaces themselves are temporary and auto-destroy after their specified duration. The workspace property is referred under the terraform { cloud {} } properties.

Then I navigated to Workspaces and searched for the workspace that I created.

It was created and configured with all the correct properties.

And that brings us to the end of Part 1. By now, you must have understood how to configure an ephemeral workspace programmatically using the TFE provider, complete with AWS OIDC authentication, VCS integration, and automated CI/CD deployment. In Part 2, I used this workspace to deploy actual AWS infrastructure to observe how the ephemeral nature automatically cleans up resources after the specified duration.

Part 2: Using the Ephemeral Workspace to Deploy AWS Infrastructure

Once the ephemeral workspace was created and configured (as we learnt in Part 1), deploying infrastructure becomes straightforward. The ephemeral workspace monitors the connected GitHub repository (02-infrastructure folder) and automatically triggers runs when changes are detected. When you push changes to a branch and create a PR to main, HCP Terraform creates a speculative plan showing what resources will be created or modified.
The code for this part is under the 02-infrastructure folder of the GitHub repository. There is only one configuration to set to use the ephemeral workspace, which is under the terraform { cloud {} } property.
Here is the code to set that.

Please note that the workspace name must match the workspace created in part 1. In this repository, I call the VPC module and pass a few variables to provision a VPC with public and private subnets in two availability zones, along with an internet gateway and NAT gateways for internet connectivity.

Once changes are merged into the main branch, the ephemeral workspace automatically triggers a run to provision AWS cloud resources, as shown below. It also sets a timer to destroy the resources based on the specified condition.

The most compelling feature of ephemeral workspaces is their automatic cleanup of resources. With auto_destroy_activity_duration = "1h" configured, the workspace automatically destroys all resources after 1 hour of inactivity, measured from the last run. When the timeout is reached, HCP Terraform initiates a destroy run, as shown in the image below, that executes terraform destroy to remove all provisioned AWS resources, ensuring no orphaned infrastructure remains.

This automatic cleanup prevents resource sprawl and eliminates the risk of forgetting to manually destroy temporary infrastructure, making ephemeral workspaces ideal for short-lived environments.

That brings us to the end of this note. Today, you learned how, by automating both the creation and destruction of temporary environments, you can confidently deploy infrastructure for testing, demonstrations, and proofs-of-concept without the risk of accumulating costs or orphaned resources.