Post updated: 2019-08-21
Version2 of the lambda function available.
Git repo: https://github.com/smitphilip/Ec2-Auto-StartStop
It’s been an interesting week… I’ve been getting my hands dirty with a number of new AWS tasks, as I’m part of a task-force type team that is busy developing a new product where components of it will live in AWS.
This is part of a larger internal proof of concept, and in the light of cost-saving, while building our MVP, the team decided to shut down the EC2 instances while we’re not working on them.
So, just one thing before we carry on; I was not aware of AWS Instance Scheduler (thanks jamson920 for pointing me this direction), hence I developed my own version. But since sharing this post with a few others, I also learned that Instance Scheduler is not available in all AWS regions, so I still see a need for this functionality.
Anyway, I got it working relatively quickly though, and found the Lamdba (serverless) functionality very intriguing. This specific function is rather powerful, especially in a development environment that lives in the public cloud, in cases where not all the instances need to run 24/7.
I do believe that the flexibility of the platform and interoperability of all the components makes this a must-have skill in anyone’s cloud toolbox. I’m being thrown into the deep end on a number of these automation tasks, so keep an eye out for a few of these types of posts.
Before we get started; let’s cover the AWS services that we’ll be using to achieve what we need.
- EC2 - This is, of course, the compute instance, for which we’ll write functions to automatically start and stop.
- Cloudwatch - Cloudwatch is the monitoring service. Amongst other features, it is used to collect and track metrics, collects and monitors log files, and automatically react to changes in your AWS resources. Very cool. So we’ll need this to know the state of our EC2 instance.
- Lambda - AWS’s serverless platform. Basically, it’s a computing service for which you don’t have to manage the operating system. You give it your code and tell it when to run it, and you pay only for the time it takes to run your code.
- IAM - For those that don’t know, this stands for Identity and Access Management. We’re going to create a user policy for the Lambda function to use, and this will allow it to access (and change the state) of the EC2 instance.
With that covered, let’s get going…
I’ve created meaningful tags for my EC2 instances. This is so that I can target a collection of EC2s and we don’t have to maintain a list of instance IDs in my Lambda functions. But I will share a simplified method if you just want to list all the instances that the Lambda function needs to target. Note: You’ll have to update both functions if IDs are changes or need to be added. But we’ll get there…
The tag I’ll be working with is called ‘DeviceCategory’.
Next, let’s create an IAM policy for the Lambda function to use. As explained this allows the Lambda function to start and stop the EC2s. The Describe action is also added to allow us to get the tag values.
Navigate to the IAM service, on the left pane, select Policies, and at the top of the page that opens, select the Create Policy button. You can create the policy with the visual editor, which is rather straight forward. Or, alternatively, select the JSON tab, and paste the policy below into the policy editor text box.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:Describe*",
"ec2:Start*",
"ec2:Stop*"
],
"Resource": "*"
}
]
}
Click the Review Policy button, give your policy a name (mine is ec2-lambda-access), a description, review the summary of the policy, and click the Create policy button. While we’re here, let’s take care of the Role. In the left pane of the IAM service console, select Roles, and click Create Role. (you can also create the policy via the Role feature). On the Create Role window, under the Type of Trusted Entity, select AWS Service, and under the section Choose the service that will use this role, select Lambda, and then click the Next button. On this page, we will attach the policy to the role. So either search for the policy we created in the previous step, or click the Create policy and follow the steps outlined for this requirement. Once selected, or created, click the Next button. If you’d like to attach tags, feel free, after this we’re moving to the last page. Give your Role a name, a description, and double-check the Policies option is correct. Once you’re happy with that, click the Create Role button.
Next, we’ll move onto the fun stuff. Lambda!
Navigate to the Lambda service in the AWS console. Click the Functions option in the left pane, and click the Create Function button, on the right. On the next page that opens, select Author from Scratch. In the Basic Information section at the bottom of the page, enter a name, I called mine ‘AutoStartEC2’, and select Python2.7 in the Runtime dropdown. Under the Permissions option, select Use an Existing Role and in the dropdown that appears, select the Role that we created in the previous step. Cool, once you’re happy with that, click the Create Function button.
As explained; these functions below are to automatically start and stop instances based on specific EC2 tags. I’ve tagged mine with a ‘DeviceCategory’ tag and would like to apply this to all devices with the ‘Sandvine-SPB’ tag.
import boto3
region = 'eu-west-1'
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
deviceFilters=[
{
'Name': 'tag:DeviceCategory',
'Values': [
'Cust1-SPB',
]
},
]
instance_ids = []
ec2Reservations = ec2.describe_instances(Filters=deviceFilters)['Reservations']
for reservation in ec2Reservations:
ec2Instances = reservation['Instances']
for instance in ec2Instances:
instance_ids.append(instance['InstanceId'])
print("Starting instance: {}".format(','.join(instance_ids)))
ec2.start_instances(InstanceIds=instance_ids)
The above is only the AutoStartEC2 function. You’ll have to create a second function and make two minor changes.
print("Stopping instance: {}".format(','.join(instance_ids)))
ec2.stop_instances(InstanceIds=instance_ids)
Alternatively, we could also check for the current status of the instance and perform the opposite action. i.e. If the function is called and the instances are started, then stop them. I think I’ll build this into my version 2.0.
If you’d like to simplify the above function and only list the instances you need to start and stop every time, you can replace the functions with something like this. (NOTE: every time you have new instances that this needs to be applied to, the Instance-ID needs to be added. And removed if they no longer exist.)
import boto3
region = 'eu-west-1'
instances = ['i-12341234123412345' 'i-12341234123412346']
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
ec2.start_instances(InstanceIds=instances)
print 'started your instances: ' + str(instances)
This Lambda stuff can get quite intense, but with that comes flexibility, and with flexibility comes endless possibilities!!
Now, the final step is to schedule these functions to be triggered at specific times of the day. From the AWS console, navigate to CloudWatch, and in the left pane, under Events, select Rules. Click the Create Rule button. I’ve created two, one for every function.
Under the Event Source, select the Schedule radio button. I needed to trigger my ‘AutoStartEC2’ function every weekday morning at 8am. Under the Cron Expression field, I entered the following: 0 6 ? * MON-FRI *
To elaborate on what these values mean:
- 0 = Minutes
- 6 = Hours
- ? = Day of Month
- * = Month
- Mon-Fri = Day of week
- * = Year
The Offical documentation on creating a rule expression can be found here
Two things on the values that I selected; the Hours. AWS schedules these rules at UTC time and I needed it to be run at 8am GMT+2, hence the 6am. My value for Day of Month (?). The definition for the question mark wildcard is as follows: “The ? (question mark) wildcard specifies one or another. In the Day-of-month field, you could enter 7 and if you didn’t care what day of the week the 7th was, you could enter ? in the Day-of-week field."
But also take note of the limits: “You can’t specify the Day-of-month and Day-of-week fields in the same cron expression. If you specify a value (or a * ) in one of the fields, you must use a ‘?’ (question mark) in the other."
So my requirement is specifically a weekday, hence the day of the month had to be a question mark.
When playing around with the cron expression trying to find the right values; AWS helps you out by displaying the next 10 trigger dates as soon as it finds a valid expression. It’s quite handy when you’re mixing and matching values.
Duplicate this Event for the AutoStop function and you’re good to go.
I’m working on a few more serverless automation tasks. I’ll share them once I’m happy with the results…
Version 2
As promised, I’ve updated the lambda function to a version 2. We now check the current state of the EC2’s with the resource tags that we defined. If these are started, we stop them. This means we can use one script that’s scheduled for the morning and the evening.
In the morning, the function will start our scripts and in the evening our function will stop them again.
import boto3
# Definitions
region = 'eu-west-1'
mytag = 'Cust1-SPB'
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
deviceFilters=[
{
'Name': 'tag:DeviceCategory',
'Values': [
mytag,
]
},
]
instance_ids = []
ec2Reservations = ec2.describe_instances(Filters=deviceFilters)['Reservations']
for reservation in ec2Reservations:
ec2Instances = reservation['Instances']
for instance in ec2Instances:
instance_ids.append(instance['InstanceId'])
if instance['State']['Name'] == 'stopped':
print('starting instances')
ec2.start_instances(InstanceIds=instance_ids)
else:
print('stopping instances')
ec2.stop_instances(InstanceIds=instance_ids)