I find myself mentioning the term Principle Of Least Privilege often, so I thought, “Let’s write a practical blog post of how to implement this principle in the CI/CD realm”.
In this blog post, I’ll describe the process of granting the least privileges required to execute aws s3 ls
and terraform apply
by a CI/CD runner.
In case you’re from health tech, this process might help you with qualifying some of the HIPAA compliance requirements, see HIPAA’s Minimum Necessary Requirement.
Imagine this, you’ve created an IAM user cicd-user
, with AdministratorAcess and generated the access keys AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
for that user. You’ve created that user so that your CI/CD service, whatever it is, GitHub Actions, drone.io, Jenkins, etc., will be able to apply changes in your AWS account.
This is a common scenario that usually happens in small startup companies, where the product and sales are far more important than meeting regulations and securing the product. But why do people do that? Because every time a CI/CD attempts to do something, figuring out which policies are required for the job is a nightmare.
“Let’s provide the CI/CD service an admin permission, we’ll deal with that later, we must focus on the product.”
TIP: Listen to Avenged Sevenfold - Nightmare while reading this blog-post.
A typical “Nightmare Situation” where you need to create an IAM policy for a CI/CD service; here goes.
aws
or terraform
command.aws
or terraform
…NOTE: Sometimes, you’ll get Forbidden status code: 403
, which is quite useless for finding out the required IAM permissions.
Here comes iamlive by Ian Mckay.
Generate an IAM policy from AWS calls using client-side monitoring (CSM) or embedded proxy
I’ll focus on the Proxy Mode, which supports running iamlive
as a Docker container. No worries, we’ll get to that.
The way iamlive
works is pretty simple. iamlive
runs in the background in proxy mode and serves 0.0.0.0:10080
, which allows access from “any IP”, but only we have access to this process, so we’re good. In a separated terminal, you set HTTP_PROXY
, HTTPS_PROXY
, and AWS_CA_BUNDLE
according to iamlive
.
UPDATE 17 Apr, 2022: Previously, I provided a way to pull iamlive source code and build a Docker image locally. Since then, I’ve created unfor19/iamlive-docker which automatically builds and pushes iamlive to my DockerHub registry.
The Docker image that we’ll be running - unfor19/iamlive-docker
We need to proxy all of aws
and terraform
requests via iamlive-test
, and then iamlive
will be able to generate the relevant IAM permissions according to the invoked request. Pretty awesome, right?
Zooming in on some of the arguments
-p 80:10080
and -p 443:10080
- Maps the ports 80 and 443 in the Host to the Container port 10080--bind-addr 0.0.0.0:10080
- iamlive
listens on port 10080, from any IP address--force-wildcard-resource
- Makes it easier to iterate over missing permissions--output-file "/app/iamlive.log"
- Save the generated permissions to a file upon kill -HUP 1.First Terminal - iamlive-test
docker run \
-p 80:10080 \
-p 443:10080 \
--name iamlive-test \
-it unfor19/iamlive-docker \
--mode proxy \
--bind-addr 0.0.0.0:10080 \
--force-wildcard-resource \
--output-file "/app/iamlive.log"
# Runs in the background ...
# Average Memory Usage: 88MB
First, I recommend that you create a fresh new IAM user with no permissions at all, let’s name that user dummy-user
. Doing so will ease getting the minimum required permissions (all of them).
The fact that the iamlive-test
container is running means nothing to aws
and terraform
. To configure both CLIs to use this proxy server, open a new terminal window and execute the below commands.
Second Terminal - set environment variables
export AWS_ACCESS_KEY_ID="AKIA_DUMMY_USER_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="DUMMY_USER_SECRET_ACCESS_KEY"
export HTTP_PROXY=http://127.0.0.1:80 \
HTTPS_PROXY=http://127.0.0.1:443 \
AWS_CA_BUNDLE="${HOME}/.iamlive/ca.pem"
Say what? From where did this AWS_CA_BUNDLE come from? Well, this environment variable instructs tools that use the AWS SDK to trust the provided Certificate Authority Certificate, in our case, it’s ca.pem
.
The ca.pem
file is generated by iamlive
for each execution of the iamlive-test
container. We need to copy the ca.pem
file from the container to our machine (Host).
Second Terminal - copy ca.pem
docker cp iamlive-test:/home/appuser/.iamlive/ ~/
The environment variables HTTP_PROXY and HTTPS_PROXY are telling AWS’s SDK to forward traffic via a proxy server (iamlive-test
container).
So far, we’ve got two terminal windows. In the first terminal, we have the iamlive-test
container running, and it probably looks like it’s doing nothing, but it’s running, trust me. In the second terminal, we exported the relevant environment variables and copied ca.pem
from the iamlive-test
docker container to our machine (Host).
In the second terminal, we’ll execute commands with aws
and terraform
. After the command execution is completed, we’ll inspect the logs of the first terminal, which runs the iamlive-test
container.
Second Terminal - aws
aws s3 ls
# Output
# An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
Really? All I need is the ListBuckets
permission? The real permission is called s3:ListAllMyBuckets
, and I know that because the logs of iamlive-test
look like this.
First Terminal - iamlive-test
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": "*"
}
]
}
Before we proceed, it’s important to mention that terraform init
cannot be proxied via iamlive-test
since it attempts to access registry.terraform.io, and it’s not covered by iamlive
. So first, unset the proxy settings, and then execute terraform init
.
This is what it looks like when you attempt to execute terraform init
with the proxy settings (environment variables) on.
Second Terminal - terraform init with iamlive proxy
terraform init
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider
hashicorp/aws: could not connect to registry.terraform.io: Failed to
request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": x509:
certificate signed by unknown authority
For testing purposes, create a new directory and add the following main.tf
file.
main.tf
variable "region" {
type = string
default = "eu-west-1"
}
provider "aws" {
region = var.region
}
resource "aws_s3_bucket" "app" {}
Unset proxy related environment variables and then execute terraform init
Second Terminal - terraform init
unset HTTP_PROXY HTTPS_PROXY AWS_CA_BUNDLE
mkdir terraform-iamlive
cd terraform-iamlive
vim main.tf # copy-paste the above main.tf file
terraform init
# Terraform has been successfully initialized!
Finally, we can execute terraform apply
and see which permissions are required for the task.
Second Terminal - terraform apply
export AWS_ACCESS_KEY_ID="AKIA_DUMMY_USER_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="DUMMY_USER_SECRET_ACCESS_KEY"
export HTTP_PROXY=http://127.0.0.1:80 \
HTTPS_PROXY=http://127.0.0.1:443 \
AWS_CA_BUNDLE="${HOME}/.iamlive/ca.pem"
# In terraform-iamlive dir
terraform apply
# Output
# Error: error reading S3 Bucket (terraform-20210422212704452600000001): Forbidden: Forbidden
# │ status code: 403, request id: A25SVTBABN0B3DSH, host id: /c1b5TsnsBE23AaDDHJQ34yLAYdrR7y3kvu2lqEX7VvstffawROKWwcPYfxNjleeluZPg9nucKY=
No idea what that means; let’s check iamlive-test
container logs to see if iamlive
knows which permissions are required.
First Terminal - iamlive-test
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:GetCallerIdentity",
"ec2:DescribeAccountAttributes",
"s3:ListBucket"
],
"Resource": "*"
}
]
}
Sometimes, you won’t get the full list of required permissions. To overcome that, add the given IAM policy and invoke terraform apply
again to see which permissions are missing.
Also, you might want to limit "*"
to specific resources or patterns, but still, it’s better than the current nightmare.
Here’s the output after adding the above IAM policy to my “dummy-user”.
First Terminal - iamlive-test
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:GetCallerIdentity",
"ec2:DescribeAccountAttributes",
"s3:ListBucket",
"s3:GetBucketAcl"
],
"Resource": "*"
}
]
}
A new permission was added - s3GetBucketAcl
. We need to iterate this process a few times, but each time it’s a simple copy-paste of the generated permissions to the existing IAM policy in AWS Console.
This is the final result, after eight (8) iterations. If you’re about to deploy a large stack with multiple resources, you’ll have to iterate more than a few times.
First Terminal - iamlive-test
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:GetCallerIdentity",
"ec2:DescribeAccountAttributes",
"s3:ListBucket",
"s3:GetBucketAcl",
"s3:GetBucketCORS",
"s3:GetBucketWebsite",
"s3:GetBucketVersioning",
"s3:GetAccelerateConfiguration",
"s3:GetBucketRequestPayment",
"s3:GetBucketLogging",
"s3:GetLifecycleConfiguration",
"s3:GetReplicationConfiguration",
"s3:GetEncryptionConfiguration",
"s3:GetBucketObjectLockConfiguration",
"s3:GetBucketTagging",
"s3:CreateBucket"
],
"Resource": "*"
}
]
}
IMPORTANT: Remember, limit "*"
to specific resources or patterns.
NOTE: After adding the policy to an IAM user, it took a few retries to get the updated IAM policy from iamlive-test
. Just keep on adding permissions until a successful attempt.
The beauty of omitting the flag --rm
in the docker run
command is that iamlive-test
will not be removed when it stops. This is important! We want to keep the same ca.pem
in the next execution of iamlive-test
instead of re-copying ca.pem
from iamlive-test
to our machine (Host).
First Terminal - iamlive-test
docker stop iamlive-test
docker start -i iamlive-test
# Keep it running in the background
We can send a SIGHUP signal to the iamlive-test
container, which instructs iamlive
to dump its latest output to the file iamlive.log
. I piped the output through jq to beautify it.
Second Terminal - send SIGHUP to iamlive-test
docker exec iamlive-test kill -HUP 1 && \
docker exec iamlive-test cat /app/iamlive.log | jq
To avoid using a proxy server for AWS SDK operations, unset the relevant environment variables, or restart your terminal window.
unset HTTP_PROXY HTTPS_PROXY AWS_CA_BUNDLE
I’ve tried getting the required permissions by investigating AWS CloudTrail logs, and again, it was a nightmare. I just wanted a simple way, with the least overhead, to generate permissions easily for my CI/CD services.
In case you don’t know, AWS provides the free service IAM Policy Simulator, which is great for testing and debugging IAM policies in your AWS account. Then again, different tools for different purposes.
Fresh from the oven, AWS extended the capabilities of IAM Access Analyzer, posted on Apr 19, 2021. Though its capabilities are still limited and do not cover all AWS services. It’s still nice to see that AWS is improving the capabilities to ease writing least privilege IAM policies.
I’ll probably write some Bash script that invokes terraform apply
and when the error code equals 403
, adds the output of iamlive.log
to the relevant IAM policy in AWS. That might make it friendlier than it is now; it’s annoying to copy-paste.
Got a better way to achieve the same thing? Have some thoughts about this process? Let’s discuss it; feel free to comment below!