🔥Let’s Do DevOps: Compile and Test Local Terraform Provider

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I’ve been working with Hashi’s open-source community recently to test out some patches for a long-standing issue with the Terraform AzureRM provider that broke every part of building and managing Azure FrontDoor resources on the Azure Cloud platform. I’ve written extensively about it, and a good write-up is here:

Terraform AzureRM Provider Has a Breaking Bug, Azure and Hashi Won’t Fix

Part of helping test new patches is downloading branches of the Terraform provider in the public repo, building the binary executable for your architecture using Go, then staging it, and telling Terraform where to find it.

The process is valuable for testing early features or fixes and is far from self-explanatory, so I thought I’d try to write it up in human English. Let’s dive right in!

Git Clone

If you’re a Microsoft SysAdmin, you need to know how the Registry works. As a network engineer, you need to understand Arp and routing protocols. If you’re a DevOps engineer there’s an incredible (seriously incredible) broad skillset out there, but a building block you’ll see everywhere is git.

Git is a tool to permit many (hundreds if necessary!) of developers to work on the same codebase concurrently. When open-source tools like Terraform or the AzureRM (or AWS) provider need updates, contributors create a branch with the updated code, then create a pull request for review.

This PRs and branches are public (for public repos), so you can download their code and test it! In fact, this kind of testing is often very valuable for developers to make sure their fix works for your situation.

Okay, on to real work. I recently had to test out a new version of the AzureRM provider, so let’s clone down the master branch. This is where code would live if a PR for a new feature was merged, but potentially not released in the weekly binary drop. If you wanted to clone a branch, you’d start on that branch but all other commands will be the same.

First, head over to the public repo on GitHub for the AzureRM provider.

Over on the right side, you’ll see “Code” as a green button, click it and then find the SSH repo link. If you don’t see the SSH repo link, sign into (or create!) your GitHub account.

The string will look something like this:

git@github.com:terraform-providers/terraform-provider-azurerm.git

On your local computer where you’ll be doing this building, go to a folder location where you want to clone this repo to. I created a folder called azurerm_provider in the Downloads folder to store it. Then run the following command. As you can see, it’s just git clone and then the repo’s path.

git clone git@github.com:terraform-providers/terraform-provider-azurerm.git

Git: Switch Branch if Needed

If you wanted to switch to a branch here, you’d run the following: git checkout (branch name). For example, one of the current branches today is called test-shim-hdinsight, link.

This is shown here as an example, but I’m going to continue on with the master branch.

If you ran these and want to switch back to master, run git checkout master.

Go: Build the Binary

Now, we’ve downloaded a bunch of source code, but Terraform is built using Go, a language that originated at Google and released as an open-source project in 2009. This language can’t be run purely with source code, we’ll need to compile the code into a binary file that our OS can execute.

Go isn’t installed by default on Mac, Linux, or Windows, so we’ll need to go get the binary.

Terraform itself provides a great walkthrough of where to find these binaries and how to install them. Particularly if you’re on Windows this will be useful for you since I’m on a mac, the steps won’t look or be exactly the same.

You’ll also need to have terraform installed — it’s used for validations. If you’re on mac I highly recommend tfswitch — it’s fantastic for quickly swapping between terraform versions, links.

When the binary is built, it won’t be placed in the repo directly, which confused me at first. It’ll be placed in your $GOPATH, a variable set on your machine.

To set the GOPATH variable on Linux or Mac, use an export command. I set these to go into a root folder for my user called go. If this is a new folder, make sure to build it (using mkdir or finder) to avoid any unexpected issues.

export GOPATH=/Users/kyler/go

To build the binary run make build from the provider’s folder. Don’t worry if this takes a while to build — on my relatively beefy 2017 MacBook pro this takes 8–10 minutes.

Once built, let’s confirm our binary was put in the right place and exists. Within our $GOPATH (in my example, /Users/kyler/go), there’s a folder called bin, then we see that the azurerm provider file was created, woot!

Using the file command we can confirm this is a binary file created for macOS. By default, Go will compile terraform for the platform you’re on.

If you want to cross-compile it for another platform, the Terraform README.md referenced earlier has pretty good instructions:

GOOS=windows GOARCH=amd64 make build

Local Terraform Provider Paths

Now we’ve compiled the provider, but when we run terraform later on it won’t intuitively know to find the provider here. Rather, it’ll download the last binary release of the provider from Hashi’s provider repository, which defeats the whole point here!

Terraform looks for local providers in a few locations on the system it’s run on before it looks to download it from a public repo. I think of this like a hosts file for DNS — it’s consulted before a DNS request is ever sent to a server. The wiki page from Hashi has more details if you want to dive in.

In short, the path on Windows is %APPDATA%\terraform.d\plugins and on linux-like system (like macOS), it’s ~/.terraform.d/plugins.

So we need to copy our binary there! But there’s a small complexity. Terraform doesn’t look exactly in that path for a binary, it looks in child folders that are constructed based on the platform you’re running on. This part isn’t documented well by Hashi yet, and took some experimenting by me to get to. Hashi has a separate wiki that has a partial list of these platforms here.

You’ll end up playing some mad-libs here. The path mad-lib is:

(path on system)/(provider reference name)/(provider name)/(provider version)/(platform reference in Go)

Let’s go through each separately to better understand what’s happening here:

• Path on system: This is the file system path we talked about earlier. Since I’m doing this on a mac, I’ll use ~/.terraform.d/plugins.

• Provider reference name: This is an optional folder path that must align between the binary folder path and the main.tf. I like to use local here as a reminder that we’re using a local binary.

• Provider name: This must align between your main.tf and the folder path. I kept things simple and left the same name as the real azurerm provider uses, which is unsurprisingly azurerm.

• Provider version: This must align between the file path where the binary is stored and the version set in your terraform main.tf config. To my knowledge, there’s nothing in the provider source code that has to align with how it’s called. In this case, I’m testing version 2.40.0.

• Platform reference in Go: Terraform uses the Go language to check the kernel platform it’s running on. In Mac’s case, it’s darwin_amd64.

The real path MacOS will look in is: ~/.terraform.d/plugins/terraform.example.com/local/azurerm/2.40.0/darwin_amd64.

This part was by far the hardest part to figure out. Hashi calls provider development an advanced task, and I suppose that’s why the documentation is a bit lacking.

So to satisfy our requirements, let’s copy the binary to the proper path. First, we create the directory path, then we copy the file to it.

mkdir -p ~/.terraform.d/plugins/terraform.example.com/local/azurerm/2.40.0/darwin_amd64
cp ~/go/bin/terraform-provider-azurerm ~/.terraform.d/plugins/terraform.example.com/local/azurerm/2.40.0/darwin_amd64

Terraform main.tf Provider Block Updates

Now that we have the provider staged on our system, let’s update our main.tf with the corresponding values so it knows to use the local provider. In the main.tf of your Terraform config, set this configuration for the provider block:

terraform {
  required_version = ">=0.13.0"

required_providers {
    azurerm = {
      version = "~> 2.40.0"
      source  = "terraform.example.com/local/azurerm"
    }
  }
}

Note the version limiter matches the file path we set, as well as the source.

Terraform’s logic here is to look first in the local file path and if that fails, to then look literally at terraform.example.com, which of course doesn’t work, because it isn’t a real website.

Terraform Init

Now we’re ready to initialize terraform. Navigate to where your main.tf is in a command line and run terraform init. You’ll see right away if terraform succeeds at finding the provider — it’ll be listed as installed and “unauthenticated”, like this:

If you see any error message about failing to find the provider, you’ll need to dig deeper. A failure might look like this:

This likely means the provider isn’t named correctly or isn’t in the proper path. But that’s hard to troubleshoot by guessing. A better way to test is to turn up our logging, by running this command: TF_LOG=TRACE terraform init.

You’ll see a lot going on here, but look at the paths that terraform is searching for, and compare that with where you copied the binary provider to (you did remember to copy it, right?). Read these over carefully and you’ll be able to sort this out.

Authenticate to Azure

Let’s prove this stuff is working. In order for this provider to talk to anything we’ll need to authenticate to Azure, so first run:

az login

If your system doesn’t have this command, use the following to install it (provided you have brew installed): brew update && brew install azure-cli

A browser will pop open and you can authenticate and complete MFA requirements if your environment has it. If you have multiple accounts in your tenant, which is very common in an enterprise env, you’ll need to select which one you want to use. First, list all the subscriptions:

az account list

Then set which subscription you want to use with the az account set command (obviously replacing the subscription ID with the one you want to use):

az account set —-subscription="12341234–12345–1234-1234–123412341234"

Build Stuff!

Now you have the whole picture of how to copy down the terraform provider source code, build the binary from the source, stage the file, and tell terraform to use it.

You’re now empowered to test provider branches and be an early validator for changes. Just remember to take a backup of your state file, these early versions are often broken or could cause issues.

Good luck out there. 
kyler