🔥Let's Do DevOps: Managing GitHub Repo Backdoors - Repo Deploy Keys🔥

aka, simple bash scripts rule

Nov 12, 2024

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

Like most GitHub Enterprise and Org admins, I feel like I’ve got a pretty good handle on the security of the Org. For instance, we use SSO (Single Sign-On) for our users, so if anyone leaves the Org, we disable their access to the Org code, as well as their SSH keys. This is all automated and pretty well instant, and doesn’t keep me up at night.

However, there is one feature of GitHub that *does* keep me up at night - Repository Deploy Keys. Repo Deploy Keys are a really flexible feature of GitHub, which allows an SSH key to be created inside a Repo, and grant direct access to the code.

That sounds great, but note a very important distinction here - the Repo Deploy Key isn’t under any user or other identity - it’s a direct key to the Repo. If someone has a copy of that key, and leaves your Org, does the key stop working? Nope, they continue to have consistent access to your code, and if the little box is checked, push access too.

Surely since these keys are such a backdoor to your Org and short-cut security so well, GitHub must make them easy to find and inventory, right? Nope.

There’s no Org or Enterprise level view of Repo Deploy Keys.

An up until a few days ago (Oct 23, 2024, in this blog announcement), you couldn’t disable them at the Org level. And it’s a super good idea to turn them off, but what will break when you do? Well, good question.

Let’s go find out.

Repo Deploy Keys - In a Repo

Let’s start from a Repo - where can I see if Repo Deploy Keys are in use?

Go into the Repo, then Settings. Then on the left column, find the Security header, and Deploy Keys.

If you see something like this, there is a Repo Deploy Key created. If we look closely we can learn a few things:

This key was created on Nov 2, 2024 by user KyMidd (oh hey, that’s me!). You can see if this user is still employed at your company.
“Never used”. That’s great news, this Repo key hasn’t been used. If you see a date, it has been used, potentially a lot, and most recently on that date.
“Read/write” - This one is less good. These Repo keys can be Read-Only (the default) or “Read/write” which permits pushing updates to the codebase.

If the key has never been used, you can delete it by clicking “Delete” over on the right.

If you see this screen, you don’t have any deploy keys in this particular repo. Good for you!

If we don’t see any Deploy Keys in this Repo, that’s awesome! Now just time to, um, check every other repo. Which might be totally reasonable - maybe you’re a little tiny business that has like 20 Repos. If so, go to town, dude.

More likely, you’re an enterprise and you have thousands of repos. You’ll be clicking for weeks, and by the time you’re done, you’ll have to start over to check again. It’s just not reasonable.

We need a better way. Let’s find one.

Find all the Repo Keys in Your Org

At date of publication, GitHub doesn’t yet let you find all the Repo Keys in use across your Organization or Enterprise. Maybe one day (I hope!), but for now we’ll need to do the work ourselves.

Given that GitHub isn’t helping us here, how are we going to do this ourselves? Well, GitHub has an excellent CLI, as well as an excellent API.

We could directly call the API endpoint, but the GitHub CLI does it for us, so let’s just use that instead.

The default command “gh repo deploy-key list” lists all the Repo keys in a single Repo that you’ve already cloned. And like, we could certainly clone thousands of repos in sequence to scan their deploy keys, but I really don’t want to do that.

Thankfully there is a flag we can use here, “-R” to specify a repo.

gh repo -R MyOrg/MyRepoName deploy-key list

Here’s an example showing a positive match. If you get an error message when you run your own command, try “gh auth login” to get permissions. You might have to select the Org you’re going to scan to make sure your terminal can see all the private repos, too.

> gh repo -R KyMidd/AwsCodePipelineDemo deploy-key list 
ID         TITLE                             TYPE        KEY                                                                               CREATED AT          
111202165  Totally Not a Security Issue Key  read-write  ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII+fgxUrNo9o0e8U9fmXFSj4U5KtKDFMf/NRgVIz4z9O  about 12 minutes ago

And a negative match:

> gh repo -R KyMidd/TerraformingAllResources deploy-key list
no deploy keys found in KyMidd/TerraformingAllResources

And that’s all well and good, but do I really have to go export a list of all my repos and then run this command thousands of times? Well, no, we can be a bit sneaky. The GH CLI can also print out a list of all repositories, and we can pipe that right into this command.

for repo in $(gh repo list MyOrg -L 10000 | awk '{print $1}'); do
  echo "Looking at: $repo"
  gh repo -R $repo deploy-key list
  echo ""
done

Here’s an example of it running on my own Repos, and showing both a negative and positive result. So basically, it’ll find all your repos (up to 10k in this example, but could probably go higher) and iterate through all of them, printing out where Repo Keys are found for you to go investigate.

> for repo in $(gh repo list KyMidd -L 10000 | awk '{print $1}'); do
  echo "Looking at: $repo"
  gh repo -R $repo deploy-key list
  echo ""
done
Looking at: KyMidd/ActionOrgWideLicenseInventory
no deploy keys found in KyMidd/ActionOrgWideLicenseInventory
...
Looking at: KyMidd/AwsCodePipelineDemo
ID         TITLE                             TYPE        KEY                                                                               CREATED AT          
111202165  Totally Not a Security Issue Key  read-write  ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII+fgxUrNo9o0e8U9fmXFSj4U5KtKDFMf/NRgVIz4z9O  about 15 minutes ago
...

Never Let This Evil In My House Again

Remediating them all is a matter of homework for you. Once you’ve locked them all down, and want to make sure no one in your Org can create them again, well, that’s easy.

You’ll require Org Admin permissions, and the setting is currently per Org (there is also an Enterprise setting once you’ve removed all keys from all your Orgs!).

Head up to your Org landing page, then Settings. Find the Security banner in the left column, and then click through to Deploy Keys.

Change the “by default, Enabled” setting to Disabled, and save.

Now, if a Repo Admin attempts to create a Deploy Key in a Repo in your Org, they’ll see this error message, and be unable to create it. Rad.

Summary

In this article we talked about what GitHub Deploy Keys are, and how they can compromise your security in a pretty sneaky way. We also talked about the options for finding them, which could include clicking for weeks, or a simple little bash script to find them all pretty quickly using the powerful GitHub CLI and APIs.

Simple bash scripts rule.

Good luck out there.
kyler

Let's Do DevOps