🔥Let's Do DevOps: Ansible AWX/Tower Host Authentication 101

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I’ve recently been spending a lot of time figuring out the idiosyncratic world of Ansible AWX, the open-source leading edge of RedHat’s Tower product. AWX is a platform for running Ansible code, and it supports all sorts of great features:

• Dynamic Inventories — It can talk to cloud providers directly and download, filter, and build targetable inventory groups of hosts that exist in those clouds.

• Job templates and credential injection — Jobs can be created to use root-level credentials for certain tasks that lower-credentialed folks might need to do, allowing us to create a job center of common tasks.

• Job scheduling — Once these jobs are defined, if they need to be a regular task AWX supports a cron-based scheduler that permits running jobs from the AWX runner server on a schedule.

If you want more information on Ansible I wrote a deeper-dive into why AWX is better than local Ansible on your computer:

Running Ansible from your Computer: Why AWX/Tower is Better

Credential Injection

I want to spend this blog focusing on the second bullet I called out above — credential management and injection.

Credential management is the most unintuitive piece of the unintuitive AWX platform. At first blush, the AWX system makes it seem like only a single credential can be used for any job template, which is not true. This only appears to be true due to some bad UX design.

You can use any number of credentials on a job template that you’d like.

I want to walk through how we do it, but before we get there we need to understand the AWX pieces in play and how variables (including credentials) flow through the different AWX components. I’ll deep-dive into these shortly, but let’s do a 10,000-foot view of the components to get us started:

• Credential Types: Credential Types are a template that defines which values can be set in a credential. Due to some quirkiness here, you’ll end up with many of the exact same Credential Types with slightly different names. Think of Credential Types as defining the structure, and Credentials (described next) defines the values in that structure. These define what variables are passed forward by a Credential to a job template or inventory to consume them.

• Credentials: Credentials are usernames, passwords, SSH keys, and whatever else is needed in your env to authenticate to a service. An example would be a username of “admin” and a password of “password”.

• Inventories: My favorite way to manage host inventories is to use a Data Source, which means a connection to the metadata on a cloud platform. This has the big advantage of updating at job runtime and being “always correct” since it loads real data. Inventories consume the injected environmental variables to authenticate to the target hosts.

I want to deep dive into each of these and walk through what happens at each layer. Let’s start with Credential Types.

Credential Types

Credential Types are a template that AWX uses to help build Credentials. It basically defines the fields and rules around those fields for Credentials built off that template.

Credential types are global to the entire server, irrespective of the Organization the credentials are created within. The other items we’ll walk through here (Credentials, Job Templates, etc.) all live within an Organization and are available only to folks with access to that Org.

Side note: I have no idea why Credentials are global — based on how they’re used and accessed I’d put them into an Org if I was able to redesign AWX from the ground up.

Credential Types have two major fields, an Input Configuration (what values should be provided into the credential), and an Injector Configuration (how should those values be transformed and provided to jobs or inventories).

Input Configurations for a user/password combo look like this:

fields:
  - id: username
    type: string
    label: Username
  - id: password
    type: string
    label: Password
    secret: true
required:
  - username
  - password

And the Injector Config for the same username/password looks like this:

extra_vars:
  my_ssh_key_file: 'null'
  userlogin: '{{username}}'
  userpass: '{{password}}'

Note the extra “my_ssh_key_file” being set to null. I do this so all my Credential Types use the same value-set. That allows us to standardize what values the inventories consume. If we didn’t do this, our inventories would have to use only Credentials that are the same type, for example, username/password vs username/SSH key.

This might be over-engineering, but it feels like a standard, easy way for the consumers of automation in my environment to use credentials. They’re always the same, and any unused values are set to ‘null’. I recommend this for easy credential management in AWX.

Username/SSH key examples look very similar. Here’s an Input Configuration:

fields:
  - id: username
    type: string
    label: Username
  - id: my_ssh_key
    type: string
    label: Private Key
    format: ssh_private_key
    secret: true
    multiline: true
required:
  - username
  - my_ssh_key

And here’s the Injector Configuration for the same:

extra_vars:
  my_ssh_key_file: '{{tower.filename}}'
  userlogin: '{{username}}'
  userpass: 'null'
file:
  template: '{{my_ssh_key}}'

Note that here we’re authenticating using a username and SSH key, and have no use for a password, so we set that value to null. If this seems crazy or doesn’t make sense, read on to the Inventory section. It’ll make more sense there.

Remember, these Credential Types have three major quirks to be aware of:

• They are global to the server, and exposed to every Org (just the credential template, not the Credential values)

• Only one Credential per Credential Template can be used on a job. Meaning if you have 5 username/password combos to use in a job template you need to make sure each Credential is based on a different Credential Template (even if those Credential Templates are exactly identical)

• They can’t be modified once a Credential is built based on them, so make sure to research all value settings for Credential Types and standardize before letting users go on them.

My best practice to stay sane here is to name your Credential types something like “user: steve/pass data center A” or something really specific since you’ll end up with a lot of them that need to be unique.

Credentials

Credentials are what you think of what you imagine authenticating to cloud platforms or hosts. They are based on a Credential Template and provide the real and sometimes sensitive values.

Here’s what they look like. Note that this Credential exists within an Org, and the values can be hidden once created — see the “Private Key” value here. I can over-write it, but even as a full systems admin I can’t “read” the Private Key. If you look above at the Username/SSH key Credential Type this is based on, you’ll see the Input Configuration settings for Private Key, where “secret: true” that leads to this “ENCRYPTED” block.

If you’re working in a multi-Org environment each different Org can create their own copy of the Credential even based on the same Credential Template.

Note also that the Injector Config portion is hidden from regular users. They know they’re putting usernames and SSH keys into this Credential object, but they’re not sure how we’re translating or forwarding the values.

This is another reason to standardize the Inventory variables. Let’s talk about that next.

Inventories and Credential Variables

Inventories collect hosts together and are used to target job templates. Job Templates can target either the entire inventory (with no limit), only specific hosts by targeting the host’s hostname, or a group of hosts.

For instructions on how to build these dynamic inventories for AWS or Azure, check out my other posts. Here’s what an inventory looks like, and shows the Variables can set for the Inventory:

The Variables we set are these:

ansible_private_key_file: "{{ my_ssh_key_file }}"
ansible_user: "{{ userlogin }}"
ansible_password: "{{ userpass }}"

Note that these Variables are exactly alike if this inventory contains all Linux hosts (username/SSH key) or windows hosts (username/password), at least in terms of authentication. This is possible due to our hacky flexible solution earlier with the Injector Configuration on each, where if a value isn’t needed (for instance an SSH key when authenticating to windows with username/password), we set that value to “null”.

A quirk to be aware of at this layer is:

• Variables cascade down from the top-level (Inventory-level) to all hosts and Groups within that inventory. The top-level variables can be over-ridden at the lower-levels (specific Hosts or Groups).

Something else to be aware of here is that Variables are also used to define how AWX connects to hosts — what port and protocol is used. AWX defaults for historical reasons to SSH protocol and TCP/22. That works great for Linux, but what about Windows? You can tell AWX to connect using WinRM by setting additional variables like this:

ansible_connection: winrm
ansible_port: 5985
ansible_winrm_transport: ntlm

Remember variables waterfall down, so don’t set this as a global variable if your inventory contains Linux hosts — AWX will try to connect to Linux with Windows Remote Management protocol and it won’t work.

Summary: Tying It All Together

When you build a job template, you’ll select the inventory and all the credentials you want to use. Remember that you can absolutely add lots of different Credentials, even those with different attributes like username/password vs username/SSH key to the same job. They’ll be tried in sequence as AWX attempts to connect to hosts.

However, those Credentials must be based on different Credential Types or AWX won’t let you select them.

The Credential Type sets the fields that Credential fills out, and transforms them as part of the Injector configuration. The Inventory receives the “injected” variables and sets them as environmental variables as Ansible runs on each of these hosts.

This way, we’ve achieved a standard method of getting multiple authentications to a single job and flowing them down to any type of host that requires them.

As I continue working with AWX and making this all more dynamic I’ll write up my findings. If you have any tips and tricks to make this work better please let me know!

Good luck out there. 
kyler