🔥Let’s Do DevOps: Writing Modular, Centralized, Custom Terraform GitHub Actions 🚀
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can…
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
Hey all!
Back in the Azure DevOps (ADO) days, I grew our terraform pipelines from a single POC to several hundred. Around about 40 pipelines I got really annoyed updating each pipeline with the innovations I was creating, and decided to start templating them out. On ADO that was incredibly easy — you could reference even single steps in a file, and they’d be pulled into the template — it worked amazingly.
Things aren’t as easy over at GitHub, but they can also be made to work well!
As part of this project, I was maintaining almost a dozen pipelines across a half dozen repos, and every time I’d come up with an innovation or bug-fix, I’d have to roll it out everywhere — but no longer! I implemented centralized Actions that I wrote, that each repo in our Organization can reference, and now I make changes in one place.
Let’s talk about Reusable Workflows, which I didn’t use here, and why.
If you want to skip right to the code, scroll all the way to the bottom for the GitHub Repo link! 🚀 🥳
Way Easier: Reusable Workflows
GitHub’s newest innovation to permit easily sharing Actions among many repos is called a Reusable Workflow. This permits a Job (not a Step) to call another Action, and have it run and track it. I explored this pretty thoroughly in this story:
🔥Let’s Do DevOps: GitHub Reuseable Actions — Theory, Matrix, Concurrency, Make it all Dynamic 🚀
It works great! However, note what I said in brackets above — it must be called as a Job, not as a Step. So if we want to call the Action lots of times in a matrix pattern (super common and effective for Terraform Validation, in my experience), we can set all the permutated variables in the include block.
Which makes perfect sense and works great — until you have someone who isn’t you add an environment. When I add a new env, say prd-s001 , is the tenant ID the same? The subscription? The storage container name?
As I worked with a teammate to copy the includes block, and customize some of the fields with values I had to look up, I realizes that I’d need to create a doc and maybe train folks on how to use this pattern — they didn’t know which values to change, and when.
That works for a computer, but not for a human.
Your automation should be easy for your teammates to use. If not, you’ve failed
Inside the Matrix
There are common rules for each of these repos — for instance, dev-s001 and dev-s002 and all the other dev- environments should set az_client_id to be the same value. We can easily script that. I ended up putting all of those values into a first step and exporting them.
Now there’s no ambiguity about what variables are set for different types of environments. In fact, I was able to remove ALL the customizations that my team would need to make to our environments. They literally just name the environment properly (dev-, stg-, or prd-) and this automation sets ALL other values. That’s huge.
| # Set terraform state file info based on solution_name | |
| if [[ $(echo "$solution_name" | grep 'dev-') ]]; then | |
| tf_storage_resource_group_name='dev-RG' | |
| tf_storage_account_name='devstorageacct' | |
| elif [[ $(echo "$solution_name" | grep 'stg-') ]]; then | |
| tf_storage_resource_group_name='stg-RG' | |
| tf_storage_account_name='stgstorageacct' | |
| fi | |
| # Set approval environment based on solution_name | |
| if [[ $(echo "$solution_name" | grep 'dev-') ]]; then | |
| TF_APPLY_ENV=dev-shared | |
| elif [[ $(echo "$solution_name" | grep 'stg-') ]]; then | |
| TF_APPLY_ENV=stg-shared | |
| fi |
To make these variables available to downstream steps we need to write them to the special $GITHUB_ENV file as a map. For instance, variable_name=variable_value. I like to use bash’s tee -a command since it can write (append) to multiple locations at once if required. Here we just write to one location, but you could easily write to the special $GITHUB_OUTPUT file also in case you need to access this value from a downstream job also.
| # Write envs to file, must write or variable isn't available in other jobs | |
| echo "location=$location" | tee -a $GITHUB_ENV | |
| echo "tf_storage_resource_group_name=$tf_storage_resource_group_name" | tee -a $GITHUB_ENV | |
| echo "tf_storage_account_name=$tf_storage_account_name" | tee -a $GITHUB_ENV | |
| echo "tf_storage_container_name=$tf_storage_container_name" | tee -a $GITHUB_ENV | |
| echo "tf_state_filename=$tf_state_filename" | tee -a $GITHUB_ENV | |
| echo "TF_APPLY_ENV=$TF_APPLY_ENV" | tee -a $GITHUB_ENV |
Next we need to call our centralized Validate Action as a Step. I’d love to use reusable workflows, but those can’t be called as a Step, they can only be an entire Job. We could potentially have a second Job in this Action that reads and maps the variables info, but I decided to go for the simpler and clearer option — write an Action.
The Action lives within Org Name kymidd and in Repo azure-terraform-validate-action, and we look for a file named action.yml at tag main. All of that is customizable except the action.yml part — Actions automatically use only a few names to see if the file exists there. If it’s not named properly, it won’t use it.
Check out the block starting at line 3, with. This section says that within the Action, send these values that are stored in the env context, or the Environmental Value context. Since we wrote them with tee -a, those values are available to this Step.
| - name: Terraform Validate | |
| uses: KyMidd/azure-terraform-validate-action@main | |
| with: | |
| location: ${{ env.location }} | |
| terraform_version: ${{ env.tf_version }} | |
| az_tenant_id: ${{ env.az_tenant_id }} | |
| az_client_id: ${{ env.az_client_id }} | |
| az_subscription_id: ${{ env.az_subscription_id }} | |
| tf_storage_resource_group_name: ${{ env.tf_storage_resource_group_name }} | |
| tf_storage_account_name: ${{ env.tf_storage_account_name }} | |
| tf_storage_container_name: ${{ env.tf_storage_container_name }} | |
| tf_state_filename: ${{ env.tf_state_filename }} |
Writing an Action
Actions themselves are interesting — they look a lot like an entire Actions Workflow file — they have inputs, steps, conditionals, names, step ids, etc., but there are a few crucial differences that indicate an Action can’t run by itself:
The
onsection that causes an Action to be triggered. Since these Actions only operate when called, they can’t trigger themselves based on a repo eventinputsandoutputsare at the top level. Since these Actions are intended to be a part of another Actions Workflow, they might need to accept input and return outputsusingsection, which specifies adocker,javascript, orcompositeruntime for the action.compositeis compatible with all (most?) action calls, and is what I’ll be using below
A notable difference from using a Reusable Workflow is that every step in the Action (even if there are dozens) will be combined for the caller — they’ll see only one step in their workflow. So make sure to clearly call out sectioned outputs if you have it! We’ll do that below in our own Action, too.
Here’s the start of an Action — we assign a name and description (line 1–2), and then start setting inputs on line 3. Each input is something we can receive from the calling Action Workflow — we can set on each one a description, whether it is required or optional (e.g., line 6), and a default value (e.g., line 13).
| name: Terraform Validate | |
| description: Terraform Validation without staging files for apply | |
| inputs: | |
| location: | |
| description: Targeted location/environment | |
| required: true | |
| solution_name: | |
| description: Name of the solution | |
| required: true | |
| terraform_version: | |
| description: Version of Terraform to use | |
| required: true | |
| default: 1.4.5 | |
| az_tenant_id: | |
| description: 'Azure Tenant ID' | |
| required: true |
You can optionally set outputs also — these are items that will be exported from this context and surfaced to the calling Action Workflow. For instance, in this example, we’re surfacing a boolean value (true or false) called TF_CHANGES that is retrieved from a step called tf-changes-test with an output value of TF_CHANGES.
| outputs: | |
| TF_CHANGES: | |
| description: (bool) - Whether there are changes to apply | |
| value: ${{ steps.tf-changes-test.outputs.TF_CHANGES }} | |
| TF_APPLY_ENVIRONMENT: | |
| description: (string) - Environment to apply to | |
| value: ${{ steps.set-tf-apply-env.outputs.ENVIRONMENT }} |
Next we actually set what this Action does. We’re going to write an Action that uses all sorts of steps from different places, so we’ll use type composite (line 2). Our first step is a bash step — and one notable difference from the Actions Workflow syntax and this Actions syntax is that we are required to set the shell (line 5) — vs an Actions Workflow where the shell is implicit and assumed.
Remember that all of these steps (there are many following this one) are shown as a single Step to the calling Action Workflow, so having banners that separate different commands (like on line 7–11) can be incredibly useful and clarifying.
The Action syntax is otherwise very similar to an Actions Workflow — note on line 19–21 how we’re writing any values to the $GITHUB_ENV file so those values are available to downstream steps, just like we’d do in an Action Workflow.
| runs: | |
| using: "composite" | |
| steps: | |
| - name: Export Azure Vars | |
| shell: bash | |
| run: | | |
| echo "" | |
| echo "########################" | |
| echo "## Export Azure Vars" | |
| echo "########################" | |
| echo "" | |
| # Set ARM values | |
| ARM_TENANT_ID=${{ inputs.az_tenant_id }} | |
| ARM_SUBSCRIPTION_ID=${{ inputs.az_subscription_id }} | |
| ARM_CLIENT_ID=${{ inputs.az_client_id }} | |
| # Export ARM values | |
| echo "ARM_TENANT_ID=$ARM_TENANT_ID" | tee -a $GITHUB_ENV env.vars | |
| echo "ARM_SUBSCRIPTION_ID=$ARM_SUBSCRIPTION_ID" | tee -a $GITHUB_ENV env.vars | |
| echo "ARM_CLIENT_ID=$ARM_CLIENT_ID" | tee -a $GITHUB_ENV env.vars |
Third-party Action calls are supported, for example line 10–13 calls the hashicorp/setup-terraform Action to install Terraform on our box.
These types of actions don’t have arbitrary syntax to write a banner, so my simple and hacky solution is to have a preceding step for every third-party call to write the banner out, for example line 1–8.
Find them in the Marketplace
All 3 of these Actions — Terraform Validate, Terraform Plan, and Terraform Apply can be found in the Marketplace!
Kyler's Terraform Validate - GitHub Marketplace
Terraform Validation without staging files for applygithub.comKyler's Terraform Plan - GitHub Marketplace
Terraform Plan, stages files for applygithub.comKyler'sTerraform Apply - GitHub Marketplace
Terraform Applygithub.com
Or Find Them on GitHub
Or you can find them in their github repos here:
GitHub — KyMidd/azure-terraform-validate-action at v1.0.0
Contribute to KyMidd/azure-terraform-validate-action development by creating an account on GitHub.github.comGitHub — KyMidd/azure-terraform-plan-action at v1
Contribute to KyMidd/azure-terraform-plan-action development by creating an account on GitHub.github.comGitHub — KyMidd/azure-terraform-apply-action at v1
Contribute to KyMidd/azure-terraform-apply-action development by creating an account on GitHub.github.com
Summary
In this write-up we talked about the two kinds of referential Actions we can write — Reusable Workflows, which contain all the information required to run, but can only be called as Jobs, not as Steps, and Actions, which contain only a subset of the information required to run, and can be called only as a Step from another Action Workflow.
We showed how Actions are different from an Actions Workflow, and then talked about how they’re the same and you can customize to your heart’s content. We even published some general-purpose Terraform workflows into the GitHub Marketplace for you to consume if you’d like!
Good luck out there!
kyler