🔥Let’s Do DevOps: K8s, Fetching AWS Secrets Manager Secrets On Pod Launch (Securely)
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can…
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
Hey all!
A recent project of mine has been to help a team transition their Kubernetes (often stylized as k8s) secrets from a config-map to the AWS secrets manager, and have the pods dynamically fetch secrets on launch using the CSI secrets manager and AWS CSI driver.
And I had only one question — what do any of those words mean?
If you’d prefer to skip all the talk and go right to the code, scroll to the bottom of this article for the GitHub link of k8s, terraform, and docker code I used to get all this working.
Let’s go over some terms first and then dig into this project.
Terms
K8s itself is a dozen or so technologies in a trench-coat, so lets start there:
K8s/Kubernetes — a declarative tool for orchestrating containers, as well as managing Services, Deployments, and many other API objects to support the orchestration goal. A group of co-managed K8s servers operating together is called a Cluster
Container — A lightweight packaging of tools built and stored as a container image. To learn more about Containers, see Containers 101, a tech talk from myself and Sai Gunaranjan
Pod — The name K8s uses for a grouping of individual containers. This can be 1 container or many
Service — Since containers will come and go, services provide an intrepid network mapping from the broader network to the correct pods
Deployment — A description of Pods, Services, and their configurations to deploy to a Cluster
Service Account — An authorization object in K8s, can be used for intra-cluster authn, and for some external auth (like to IAM, in this walk-through!)
Container Storage Interface (CSI) — CSI is an open-source standard written to support k8s and other tools and permit modular Drivers for specific i/o requirements. The CSI deployment puts pods and other configuration into your cluster. More info on the CSI here
CSI Driver — Drivers for the CSI permit specific use cases, like “AWS Secrets” or other particular use cases. These drivers deploy pods and other configuration into your cluster
WHEW. That’s already a lot, right? K8s is a lot, and there are some great PluralSight courses from Nigel Poulton:
Because we’re integrating K8s with the AWS secrets manager using IAM, we’ll also need to know some of those terminologies. Let’s go over those next:
AWS Secrets Manager — The standard AWS store for “secret” or sensitive data. Can be used to store string, json, or binary data
Identity and Access Manager — The AWS service which controls who can do what, where
IAM Role — An assumable object for users and machines which, when used, grants the permissions assigned by the linked IAM policy objects
IAM Policy — A specific list of actions which can be taken by whom, on specific resources
OpenID Connect (OID) Provider — Permits trusts from objects “outside” AWS, which, since these k8s pods exist outside AWS, they qualify for. We’ll configure this provider to trust our K8s cluster
I think that’s it for terminology. If you need to take a stretch break or get some water, I understand. And we’re just getting started.
Terraform AWS Deployment, Prep Cloud
K8s was totally new to me when I started this project, so I whipped out some lovely terraform and got started. First, we deploy an EKS cluster. The EKS cluster is the management layer of K8s. It doesn’t provide the compute for any jobs, but it orchestrates and monitors the jobs.
resource "aws_eks_cluster" "aws_eks_cluster" { | |
name = "aws_eks_cluster" | |
role_arn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/EksClusterRole" | |
enabled_cluster_log_types = [ | |
"api", | |
"audit", | |
"authenticator", | |
"controllerManager", | |
"scheduler" | |
] | |
vpc_config { | |
subnet_ids = [ | |
"subnet-111", #Public Subnet 1 ID | |
"subnet-222" #Public Subnet 2 ID | |
] | |
} | |
timeouts { | |
delete = "30m" | |
} | |
} |
We also build a node-group to run the jobs assigned to EKS. The EKS cluster itself is the management part, the node-groups are the worker bees that provide compute for jobs.
resource "aws_eks_node_group" "eks-node-group" { | |
cluster_name = aws_eks_cluster.aws_eks_cluster.name | |
node_group_name = "eks-node-group" | |
node_role_arn = aws_iam_role.eks_node_role.arn | |
subnet_ids = [ | |
"subnet-111", #Private Subnet 1 ID | |
"subnet-222" #Private Subnet 2 ID | |
] | |
scaling_config { | |
desired_size = 1 | |
max_size = 1 | |
min_size = 1 | |
} |
Then we build our AWS IAM OID provider and tell it to trust the CA cert of our EKS cluster.
data "tls_certificate" "aws_eks_cluster_cert" { | |
url = aws_eks_cluster.aws_eks_cluster.identity[0].oidc[0].issuer | |
} | |
resource "aws_iam_openid_connect_provider" "oid_provider" { | |
client_id_list = ["sts.amazonaws.com"] | |
thumbprint_list = [data.tls_certificate.aws_eks_cluster_cert.certificates[0].sha1_fingerprint] | |
url = aws_eks_cluster.aws_eks_cluster.identity[0].oidc[0].issuer | |
} |
Next we create our secret in Secrets Manager. I’m creating a json-encoded cert so we only have to map a single secret onto our pod, but that’s totally up to you. You can map as many secrets as you want. The only real gotcha here is if your pod is configured to map a secret that no longer exists, the pod will fail to launch. That might be catastrophic depending on your config, so be aware.
Also, note my passwords below are totally in clear-text. That defeats the whole point, right? Normally these would be populated by hand or with some other not-cleartext-in-terraform-duh method, but for our lab this is fine.
resource "aws_secretsmanager_secret" "app1-super-secret-json" { | |
name = "app1-super-secret-json" | |
} | |
resource "aws_secretsmanager_secret_version" "app1-super-secret-json" { | |
secret_id = aws_secretsmanager_secret.app1-super-secret-json.id | |
secret_string = jsonencode({ | |
app_password = "Hunter12" | |
smtp_password = "HiMom" | |
other_password = "TheBestPasswordEvah!@" | |
}) | |
} |
Now we need an IAM role for our Eks pod to link to. It’ll utilize this role’s permissions to grab the secret in secrets manager. The way it gets to this IAM Role is to use the STS service to “assume” it. The policy on the role that permits this is called the “assume role policy”.
The Principal, or AWS identity, which is permitted to connect to this role, is the OID Provider we created above. The conditions limit the connection to only come from the OID Provider (that makes sense) using STS and also only from the K8s service account (SA) we’ll put on our pod.
This condition makes we security configuration really tight! Not just any pod can grab this role and use it — only the pod with that particular SA assigned to it.
data "aws_iam_policy_document" "app1_eks_assume_role_policy" { | |
statement { | |
actions = ["sts:AssumeRoleWithWebIdentity"] | |
effect = "Allow" | |
condition { | |
test = "StringEquals" | |
variable = "${replace(aws_iam_openid_connect_provider.oid_provider.url, "https://", "")}:aud" | |
values = ["sts.amazonaws.com"] | |
} | |
# Condition to limit this role to be utilized by only the service account specified | |
condition { | |
test = "StringEquals" | |
variable = "${replace(aws_iam_openid_connect_provider.oid_provider.url, "https://", "")}:sub" | |
values = ["system:serviceaccount:default:our-eks-sa-name"] | |
} | |
principals { | |
identifiers = [aws_iam_openid_connect_provider.oid_provider.arn] | |
type = "Federated" | |
} | |
} | |
} |
Then we build the IAM role, policy, and link the two. Notice the policy permits us to grab any secret in secrets manager with a name of “app1*”, with the “*” character being a wildcard. That ensures if we want to create more secrets in the future, if we use the same naming scheme, the secrets will be reachable by this identity.
resource "aws_iam_role" "app1_eks_role" { | |
name = "app1_eks_role" | |
assume_role_policy = data.aws_iam_policy_document.app1_eks_assume_role_policy.json | |
} | |
resource "aws_iam_policy" "retrieve_secret" { | |
name = "app1_retrieve_secret" | |
path = "/" | |
description = "IAM secret EKS" | |
policy = jsonencode( | |
{ | |
Version = "2012-10-17" | |
Statement = [ | |
{ | |
"Effect" : "Allow", | |
"Action" : [ | |
"secretsmanager:GetSecretValue", | |
"secretsmanager:DescribeSecret" | |
], | |
"Resource" : [ | |
"arn:aws:secretsmanager:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:secret:app1*" | |
] | |
} | |
] | |
} | |
) | |
} | |
resource "aws_iam_role_policy_attachment" "app1_eks_secret_retireve_attach" { | |
role = aws_iam_role.app1_eks_role.name | |
policy_arn = aws_iam_policy.retrieve_secret.arn | |
} |
Next let’s talk K8s!
K8s: CSI Secrets Driver
Now that our AWS cloud provider is ready, and K8s is created and reachable, let’s start deploying some functionality to it.
First up, we need the CSI Secrets Driver. This is an open source project that helps map data in and out of K8s. Think of it like our plumbing.
We could deploy this entirely with helm
, with these two commands:
> helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
> helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system
However, that doesn’t permit very easy code tracking for our Infra As Code purposes. It’d be great to have the pure K8s config, right? Well, we can export it!
helm template csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true --namespace kube-system
This outputs the K8s config to a (huge) file. We can check that file into our git repo and update it every once in a while to make sure of the exact changes we’re deploying. Deploy this file to your K8s:
kubectl apply -f ./k8s/csi_secrets_driver/csi_secrets_driver.yml
K8s: AWS Secrets Manager CSI Driver
It confused me at first that these two components have such similar names. The main difference is the “CSI Driver” is platform agnostic — it’s pure K8s functionality. The specific drivers that plug into it, like the AWS Secrets Manager CSI Driver, is specifically for AWS and Secrets Manager, even.
This file is also installed with helm, but the source is stored on GitHub and shared here:
https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
You can download the file and apply it to your K8s cluster like this:
kubectl apply -f ./k8s/aws_secrets_provider_package/aws-secrets-store-provider.yml
Now our cluster supports leveraging IAM roles for our pods! Let’s build a pod to confirm.
Building our Pod
The first step of deploying our pod and leveraging all this awesome infrastructure is to build the service account. It exists stand-alone, before it’s assigned to any pods.
Note the name that has to match what we told our IAM role to expect, as well as the annotation which has to match our IAM role’s ARN exactly.
apiVersion: v1 | |
kind: ServiceAccount | |
metadata: | |
name: our-eks-sa-name | |
annotations: | |
eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/app1_eks_role |
Next up, we build a SecretProviderClass. This identifies the exact secrets we want our pod to reach out and grab — the objects at the bottom. Note the objectName should match the secret name exactly in secrets manager.
Also, you are able to map n
secrets here — just create another ObjectName/ObjectType group within “objects”.
apiVersion: secrets-store.csi.x-k8s.io/v1 | |
kind: SecretProviderClass | |
metadata: | |
name: secrets-provider-name | |
spec: | |
provider: aws | |
parameters: | |
objects: | | |
- objectName: "app1-super-secret-json" | |
objectType: "secretsmanager" |
Next up we build our deployment. This is a standard K8s deployment except for the volume — see that the secretProviderClass matches the name of the SecretProvider we created above. I also set readOnly here, which will control the permissions in the pod to write to the pod.
The Service Account name must match exactly — it’s leveraged to connect to the IAM role we created.
Also, the container’s volumeMounts are how the secrets are injected into the container. They are mounted in the /mnt/secrets
location in the host.
apiVersion: apps/v1 | |
kind: Deployment | |
metadata: | |
name: secret-fetcher | |
labels: | |
app: secret-fetcher | |
spec: | |
replicas: 1 | |
selector: | |
matchLabels: | |
app: secret-fetcher | |
template: | |
metadata: | |
name: secret-fetcher | |
labels: | |
app: secret-fetcher | |
spec: | |
serviceAccountName: our-eks-sa-name | |
volumes: | |
- name: secrets-store-inline | |
csi: | |
driver: secrets-store.csi.k8s.io | |
readOnly: true | |
volumeAttributes: | |
secretProviderClass: "secrets-provider-name" | |
containers: | |
- name: secret-fetcher | |
image: your-docker-username/your-docker-container-repo-name:latest | |
volumeMounts: | |
- name: secrets-store-inline | |
mountPath: "/mnt/secrets-store" | |
readOnly: true |
Run it!
That is a LOT of configuration. Let’s turn it on and see if it works!
# cd /mnt/secrets-store/
# ls -l
-rw-r--r-- 1 root root 76 Jul 14 16:02 app1-super-secret-json
# cat app1-super-secret-json | jq
{
"app_password": "Hunter12",
"other_password": "HiMom"
"smtp_password": "TheBestPasswordEvah!@"
}
BAM! There are our secrets, dynamically fetched from the AWS Secrets Manager on each relaunch. Each time the pod is deleted and relaunched by our Deployment, secrets refreshed. Also when we do a rolling deployment of a new version of our pod, the secrets are refreshed.
No more static passwords for us!
References
These references are invaluable if you want to dig into any of these concepts or troubleshoot any of this not working properly.
https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver_tutorial.html
https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html#installation
https://secrets-store-csi-driver.sigs.k8s.io/getting-started/usage.html
https://secrets-store-csi-driver.sigs.k8s.io/troubleshooting.html
Summary
In this tutorial, we build a K8s cluster and all supporting resources, as well as an OID provider and IAM role to be consumed by a pod in K8s. Then we deployed the CSI Driver main package as well as the AWS Secrets Manager CSI Driver to our cluster. Then we deployed our own pod to utilize all the stuff we built, and watched it pull secrets directly from the Secret Manager, and then we echoed those passwords out within the pod host.
All code is stored here:
GitHub - KyMidd/K8s-CSI-SecretsManager
Deployable docker, k8s, and terraform to deploy a pod to fetch secrets dynamically from AWS Secrets Manager on pod…github.com
Good luck out there!
kyler