This blog post is Part 2 of our series on Cloud Custodian for implementing Governance as Code in Cloud.
The previous blog post covered a basic understanding of Cloud Custodian, sort of getting familiar conceptually. It did not cover an actual business case but in this blog post we’ll walk through how to set up automatic EC2 instance shutdowns using Cloud Custodian, and integrate it seamlessly into your Jenkins CI/CD pipeline.
Managing costs in your AWS (Amazon Web Services) environment is crucial, and one effective way to achieve cost savings is by automatically shutting down EC2 instances during non-business hours or when they are not in use.
To achieve this, our Jenkins CI/CD pipeline is configured to trigger the Makefile, which compiles and prepares the necessary code for executing our Cloud Custodian policies. These policies, driven by Jinja templates, are then deployed to AWS environment, where they autonomously manage EC2 instances, ensuring cost-efficiency and resource optimization.
Here is what our directory structure looks like –
cloud-custodian/
│
├── policies/
│ │
│ # Cloud custodian policy file dynamically getting created here
│ │
├── templates/
│ │
│ └── cloud-custodian/
│ │
│ # Jinja templates for Cloud Custodian policies
│ │
│ └── instance-auto-shutdown.yaml.j2
│
└── tools/
│
│
└── mugc.py
│
JenkinsFile
Makefile
Targeting Makefile from JenkinsFile :
Our Jenkinsfile will basically trigger a target in Makefile within which we will build our policy file using a Jinja template.
To target a Makefile from within a Jenkinsfile, you can use a sh (shell) step in your Jenkins Pipeline. You’ll execute the make command with the desired Makefile target as an argument.
pipeline {
agent any
stages {
stage('Connect to AWS') {
steps {
script {
// Generate AWS CLI profile for assuming IAM role
}
}
}
stage('Build and Target Make') {
steps {
// Navigate to the directory containing your Makefile
dir('path/to/your/makefile/directory') {
// Execute the Makefile target
sh "set +x;make ${make_action} TERRAFORM_FOLDER=${terraform_folder_path} \
AWS_ACCOUNT_ID=${pass var for your AWS account} \
AWS_REGION=${region} \
LAYER=cloud-custodian "
}
}
}
}
// Add post-build actions or notifications as necessary
}
Makefile :
There are some good to know points which i think are important before you get your hands dirty with the Makefile that i am using.
- Phony targets are targets that do not represent actual files, but rather they are used to specify a sequence of tasks or dependencies that need to be executed when the target is invoked.
- Yasha is a Python library and command-line tool that provides functionality for working with Jinja2 templates. Specifically, it’s designed to render or generate text files based on Jinja2 templates and input data.
- ?= is useful in Makefiles when you want to provide default values for variables but also allow users to override those defaults by setting the variables externally or within the Makefile itself.
# Declare the "all" target as a phony target.
# Phony targets are not associated with files and always execute their commands.
.PHONY: all
# The default target, "all," depends on the "install-dependencies" target.
all: install-dependencies
# Variable assignments with conditional defaults.
TERRAFORM_FOLDER ?= ""
LAYER ?= ""
LAYER_FOLDER ?= "cloud-custodian"
AWS_REGION ?= "eu-west-1"
AWS_ACCOUNT_ID ?= ""
AWS_ACCOUNT_AUTO_SHUTDOWN ?= "<Target AWS Account ID for this policy>"
# Target to install dependencies.
install-dependencies:
pip3 install yasha
aws --version
# Target to render Cloud Custodian policies.
cloud-custodian-render-policies: install-dependencies
@echo -e "\nRendering Cloud Custodian Policies\n" && \
cd ${LAYER_FOLDER} && \
yasha --aws_shared_services_account_id=${AWS_ACCOUNT_ID} --aws_account_auto_shutdown_id=${AWS_ACCOUNT_AUTO_SHUTDOWN} -o policies/instance-auto-shutdown.yaml templates/cloud-custodian/instance-auto-shutdown.yaml.j2 && \
cat policies/instance-auto-shutdown.yaml
# Target to perform a dry run of Cloud Custodian policies.
cloud-custodian-plan: cloud-custodian-render-policies
cd ${LAYER_FOLDER} && \
custodian run --dryrun --region eu-west-1 --region me-south-1 --profile ${AWS_ACCOUNT_ID}-profile policies/instance-auto-shutdown.yaml -s tools/output
# Target to apply Cloud Custodian policies.
cloud-custodian-apply: cloud-custodian-plan
cd ${LAYER_FOLDER} && \
custodian run --region eu-west-1 --region me-south-1 --profile ${AWS_ACCOUNT_ID}-profile policies/instance-auto-shutdown.yaml -s tools/output
- The
alltarget is declared as phony because it doesn’t correspond to an actual file, and it depends on theinstall-dependenciestarget. - Variable assignments with conditional defaults are used to define variables that can be overridden by users when running the Makefile. If a variable is not already defined or is empty, it is assigned the specified default value.
- The
install-dependenciestarget installs the necessary dependencies,yashaand the AWS CLI tool. - The
cloud-custodian-render-policiestarget depends oninstall-dependencies. It renders Cloud Custodian policies using theyashatool and specifies the required parameters. It also displays the rendered policy for inspection. - The
cloud-custodian-plantarget depends oncloud-custodian-render-policies. It performs a dry run of the Cloud Custodian policies using thecustodian runcommand, specifying the AWS region and profile. - The
cloud-custodian-applytarget depends oncloud-custodian-plan. It applies the Cloud Custodian policies to the specified AWS regions and profile.
Jinja Template :
policies:
- name: auto-shutdown-ec2-{{ aws_account_auto_shutdown_id }}
mode:
type: periodic
function-prefix: lz-cloud-custodian-
schedule: "rate(5 minutes)"
role: arn:aws:iam::{{ aws_shared_services_account_id }}:role/CloudCustodianRole
execution-options:
assume_role: arn:aws:iam::{{ aws_account_auto_shutdown_id }}:role/CloudCustodianAssumeRole
metrics: aws
resource: ec2
filters:
- type: offhour
tag: CUSTODIANOFF
default_tz: Asia/Dubai
offhour: 15
actions:
- stop
- type: tag
tags:
StoppedByCloudCustodian: Instance stopped by auto-shutdown-ec2-{{ aws_account_auto_shutdown_id }}.
- The Jinja2 template is used to generate Cloud Custodian policy definitions for EC2 instances.
{{ aws_account_auto_shutdown_id }}and{{ aws_shared_services_account_id }}are Jinja2 placeholders, which will be replaced with actual values when rendering the template. We are passing both these values via Makefile.- The policy has the following components:
name: Specifies the name of the policy, including the AWS account ID for auto-shutdown.mode: Defines the execution mode of the policy. It’s set to periodic, running every 5 minutes. It also specifies the IAM role to assume (role) and additional execution options.resource: Specifies the AWS resource type that this policy targets, which is EC2 instances in this case.filters: Contains filter rules to select the instances to which the policy will be applied. In this case, it uses a filter of typeoffhourto target instances tagged withCUSTODIANOFF. It sets the default time zone to “Asia/Dubai” and the off-hour time to 15 (3:00 PM).actions: Lists the actions to take on the selected instances. In this policy, it specifies actions tostopthe instances based on filter and tag them with a message indicating that they were stopped by Cloud Custodian.- In short this Cloud Custodian policy will be deployed as a Lambda function in the target AWS account which will run every 5 minutes, filter the ec2 instances that are running and have a tag as CUSTODIANOFF and then if the current time is ahead of off-hour time i.e 03:00 PM then turn off that ec2 instance and also configure a tag on that ec2 instance with a message indicating that it was shutdown by Cloud Custodian.
Execution and verifying the results :
Jenkins console output –
If you remember in our Makefile we had a CAT cmd to display the content of the policy file that is being generated by Jinja template.
Here is an output of the CAT cmd showing up in the console output of my Jenkins run –
Lambda function in AWS –
Lets navigate to our AWS account and check if the Lambda function is deployed and if deployed check the logs to see if the cloud custodian has detected any ec2 instance that has tag set as CUSTODIANOFF which it needs to turn off.
As you can see the logs of the Lambda function clearly show that the Cloud Custodian policy was able to filter 1 ec2 instance that had CLOUDCUSTODIANOFF tag and which was in running state.
Cloud Custodian then as per the ACTIONS configured on filter went ahead and stopped the ec2 instance while also tagging that ec2 instance with a message that the instance was stopped by Cloud Custodian.
In this blog post, we’ve explored a powerful and efficient solution for managing your AWS EC2 instances: Cloud Custodian combined with Jenkins automation. By leveraging the capabilities of Cloud Custodian and Jenkins, you can ensure that your EC2 instances are stopped at specific times or under specific conditions, helping you optimize costs, enhance security, and streamline resource management.
Here are the key takeaways :
- Cost Optimization: Automatically stopping EC2 instances during off-hours or when they are not in use can significantly reduce your AWS costs. Cloud Custodian makes it easy to define and enforce such policies.
- Enhanced Security: Stopping instances that are not actively needed can improve your AWS environment’s security posture by reducing the attack surface.
- Jenkins Integration: Jenkins acts as the orchestrator, allowing you to schedule Cloud Custodian policy executions at specific times or in response to events.
- Flexibility: Cloud Custodian policies are highly customizable, enabling you to tailor them to your organization’s specific needs and compliance requirements.
- Resource Optimization: By stopping instances when they are not required, you free up resources for other workloads, making better use of your AWS infrastructure.
- Continuous Improvement: Use Jenkins pipelines to continuously update and refine your Cloud Custodian policies as your infrastructure evolves.
Implementing this solution can lead to cost savings, improved security, and better resource management in your AWS environment. Whether you’re managing a small development environment or a large-scale production system, Cloud Custodian and Jenkins offer a flexible and scalable approach to EC2 instance management.
Don’t hesitate to start implementing these practices in your AWS environment. If you have questions or need further assistance, please feel free to reach out. Thank you for reading, and happy cloud management with Cloud Custodian and Jenkins!
Yash Kamdar 