Automate Marketing Initiatives with Salesforce Marketing Cloud Learn More

How to Automate Instance Snapshot Management in AWS EC2

To create a backup of your EC2 instances EBS snapshots play an important role. In this article, we are going to automate EBS volume snapshots, using Lambda and CloudWatch Events function for a daily backup and delete the snapshots after a particular limit of time.

 

Prerequisites

  • AWS account
  • IAM user in AWS account
  • IAM user must have permission to access services for creating Lambda function.
  • We need an IAM role to define what permissions our lambda function has.
  • CloudWatch permission to create Rules.

To Create IAM Role

Log in to the AWS console with root account or with IAM user who has permission to create Roles.

 

  • Click on Services and select IAM services.
  • IAM services page will open from the left panel click on Create Role.
  • Create Role page will open select lambda service(Because we need permission to access lambda function).
  • On selecting lambda function it will invite you to attach permission and policies to the role. but we will create our customer policy.

 

We want our function able to perform the following:

 

  • Read all type of information from EC2 (we will give it fully describe and read access)
  • Create and Delete Snapshots
  • Create and Delete Snapshots
  • Create and access Cloudwatch Event.


To create the above policy using a visual editor or by providing a JSON. You can use the below JSON policy.

 

{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "logs:*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": "ec2:Describe*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot",
                    "ec2:CreateTags",
                    "ec2:DeleteTags",
                    "ec2:ModifySnapshotAttribute"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "logs:*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": "ec2:Describe*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot",
                    "ec2:CreateTags",
                    "ec2:DeleteTags",
                    "ec2:ModifySnapshotAttribute"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "logs:*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": "ec2:Describe*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot",
                    "ec2:CreateTags",
                    "ec2:DeleteTags",
                    "ec2:ModifySnapshotAttribute"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }

 

Tagging

Lambda function uses EC2 instance tags to identify for which instance the snapshot will create.

 

  • From EC2 Services select the EC2 instance for which we want to create a snapshot.
  • From the Tags section adds the below tag.
  • The name of the tag is “auto_snapshot” value for the tag is “true”.
  • Save the changes by clicking the Save button.

Lambda function to create a Snapshot

1

Navigate to the AWS Lambda Management Console from Services.

  • Click on Create function new page will open.
  • Select Author from Scratch.
  • Give a specific name for creating a lambda function in the Function name.
  • Choose Python 3.6 as a runtime.
  • For roles select Use an existing role and select the role we made earlier.

Code

I m using Boto3. Boto3 is SDK for python, which provides programmatic connectivity to create, configure, and manage AWS services with python.

 

import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           
import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           
import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           
import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           

Every time when the above function run, it will perform the following steps:

 

  • Create a snapshot for all EC2 instance which tagged with auto_snapshot: true
  • Delete old created snapshots which were made by these function
  • You can use a tag for retention days. Create a tag in EC2 instance with name “retention_days” with value “10” (you can use any number of days according to your requirements)
  • You can also add a value of retention days in lambda function retention_days = 10. If you have not made any changes in the above lambda code by default it takes 15 days for retention.
  • Lambda function for creating a snapshot creates a “Deletion” tag with “Date” Which calculated based on the retention days. Delete function deletes the snapshots older than the retention days.
  • We need to change Basic Settings for the lambda function, the default timeout for the lambda function is 3seconds, we need to increase it (I used 59 sec) otherwise, the create function will timeout before completing the snapshots correctly.
  • You can use a tag for retention days. Create a tag in EC2 instance with name “retention_days” with value “10” (you can use any number of days according to your requirements)
  • You can also add a value of retention days in lambda function retention_days = 10. If you have not made any changes in the above lambda code by default it takes 15 days for retention.
  • Lambda function for creating a snapshot creates a “Deletion” tag with “Date” Which calculated based on the retention days. Delete function deletes the snapshots older than the retention days.
  • We need to change Basic Settings for the lambda function, the default timeout for the lambda function is 3seconds, we need to increase it (I used 59 sec) otherwise, the create function will timeout before completing the snapshots correctly.

CloudWatch Rule to create Snapshot

The lambda function is created, and we use the CloudWatch rule to automate the lambda function.

 

  • Navigate to CloudWatch Management Console from Services.
  • From CloudWatch page move to Rules from left panel
  • Click on Create rule. A new page will open.
  • Use the below settings to create a rule.
2

The above settings create an EC2 snapshot every day at 8:00 am (timezone GMT). You can use cron expression according to your requirement.

 

Targets are used to select the Lambda function for which the rule is used.

 

You can also use a Fixed rate of option to create a snapshot if you want a snapshot after a fixed number of minutes Hours and Days.

 

  • Back to the Lambda Management Console Function.
  • Select recently created function. The created lambda function page will open.
  • Click on Add trigger button from a Designer section.
  • Select the CloudWatch Events to trigger.
  • Use recently created CloudWatch rule from existing rules and click on Add.
  • The lambda function to create a snapshot for Ec2 instance is completed. You can test it by clicking the Test button.

 

A place for big ideas.

Reimagine organizational performance while delivering a delightful experience through optimized operations.

Lambda function to Delete a Snapshot

It is very important to manage to create or delete function as we are creating a lot of snapshots(according to our requirements) and if we do not delete it the number of snapshots will be created and we need to delete it manually.

 

Please follow the below steps to automate the delete function.

 

  • Click on Create function new page will open.
  • Select Author from Scratch.
  • Give a specific name for creating a lambda function in the Function name.
  • Choose Python 3.6 as a runtime.
  • For roles select Use an existing role and select the role we made earlier.

Code

import boto3
import re
import datetime
from datetime import timedelta
 
# Set the global variables
globalVars  = {}
globalVars['RetentionDays'] = "15"

ec = boto3.client('ec2')
iam = boto3.client('iam')
 
def lambda_handler(event, context):
    account_ids = list()
    try:
        """
        You can replace this try/except by filling in `account_ids` yourself.
        Get your account ID with:
        > import boto3
        > iam = boto3.client('iam')
        > print iam.get_user()['User']['Arn'].split(':')[4]
        """
        iam.get_user()
    except Exception as e:
        # use the exception message to get the account ID the function executes under
        account_ids.append(re.search(r'(arn:aws:sts::)([0-9]+)', str(e)).groups()[1])
 
    retention_days = ( datetime.date.today() - datetime.timedelta(days= int(globalVars['RetentionDays'])) ).strftime('%Y-%m-%d')
    delete_on = datetime.date.today().strftime('%Y-%m-%d')
    filters = [
        {'Name': 'tag-key', 'Values': ['automatic-snapshot-delete-on']},
        {'Name': 'tag-value', 'Values': [delete_on]},
    ]
    snapshot_response = ec.describe_snapshots(OwnerIds=account_ids, Filters=filters)
    print("snapshot_response {0}".format(snapshot_response))
    
    all_snap = ec.describe_snapshots(OwnerIds=account_ids)
    for snap in all_snap['Snapshots']:
        snapshot_time = snap['StartTime'].strftime('%Y-%m-%d')
        if snapshot_time <= retention_days:
            snapshot_response['Snapshots'].append(snap)
 
    for snap in snapshot_response['Snapshots']:
        print("Deleting snapshot {0}".format(snap['SnapshotId']))
        ec.delete_snapshot(SnapshotId=snap['SnapshotId'])

 

Every time when the above function run, it will perform the following steps:

 

  • Delete the Snapshot according to the tag of EC2 instance. Otherwise, it takes the default retention_days value “15”. It deletes all the Snapshots older than 15days.
  • Also, change Basic Settings for the lambda function, the default timeout for the lambda function is 3seconds, we need to increase it (I used 59 sec) otherwise, the delete function will timeout before deleting the snapshots correctly.

CloudWatch Rule

Now the lambda function is created, and we can also use the CloudWatch rule to automate delete lambda function.

 

  • Navigate to CloudWatch Management Console from Services.
  • From CloudWatch page move to Rules from left panel
  • Click on Create rule. A new page will open.
  • Use the below settings to create a rule.
3

The above settings delete an EC2 snapshot which is older than 10 days.

 

  • Back to the Lambda Management Console Function.
  • Select recently created delete function. Delete the lambda function page will open.
  • Click on Add trigger button from a Designer section.
  • Select the CloudWatch Events to trigger.
  • Use recently created CloudWatch rule from existing rules and click on Add.


The lambda function to delete a snapshot for EC2 instance is completed.

Top Stories

Supercharge Your Business Integrating E commerce Business With Odoo For 20% Revenue Growth
Supercharge Your Business: Integrating E-commerce Business With Odoo For 20% Revenue Growth
In the dynamic world of digital commerce, businesses are constantly seeking innovative strategies to drive growth and stay ahead of the competition. One transformative approach is to integrate e-commerce business with Odoo. This integration promises not just streamlined operations but also a significant boost in revenue – with some businesses
Boost Revenue by 15% How Odoo and BI Tool Integration Can Transform Your Business Analytics
Boost Revenue by 15%: How Odoo and BI Tool Integration Can Transform Your Business Analytics
In today's fast-paced business landscape, the ability to effectively collect, analyze, and leverage data is crucial. This is where the Odoo and BI Tool Integration come into play as game changers. The collaboration between Odoo, a suite of business applications, and BI tools can transform data analytics and reporting in
How To Manage Users & Access Control in Odoo and Protect Your Data
How To Manage Users & Access Control in Odoo and Protect Your Data
In the enterprise resource planning (ERP) world, it is essential to manage users & access control in Odoo. This is not a feature. A necessity. Odoo provides a range of applications that offer a foundation for managing users and access control. This ensures both data security and operational efficiency. The
Revolutionizing Digital Transformation with Salesforce Experience Cloud
Revolutionizing Digital Transformation with Salesforce Experience Cloud
In the ever-evolving tech landscape, businesses must adapt, innovate, and thrive. Salesforce Experience Cloud (SEC for short) is at the heart of this revolution. SEC is reshaping enterprise engagement with customers and streamlining operations. But how do you win the race and stand out among numerous companies in terms of
How Salesforce Experience Cloud Integration Boosts Productivity in Business (1)
How Salesforce Experience Cloud Integration Boosts Productivity in Business?
More than 150,000 companies have grown their businesses by shaking hands with Salesforce. Developed and customer-preferred companies like IBM, Amazon Web Services, Toyota, T-Mobile and many more are now fans of Salesforce. So, if an organization wants to become the next big fish in its industry, incorporating Salesforce Experience Cloud
From Insight to Action The Essential Guide to Salesforce Experience Cloud Introduction
From Insight to Action: The Essential Guide to Salesforce Experience Cloud Introduction
In this fast-paced world, we live in today, it's astonishing to know that a staggering 84% of customers prioritise the experience they have with a company just as much as the actual products and services offered. This eye-opening statistic heralds a shift in the business landscape, underscoring the equal importance

          Success!!

          Keep an eye on your inbox for the PDF, it's on its way!

          If you don't see it in your inbox, don't forget to give your junk folder a quick peek. Just in case.



              You have successfully subscribed to the newsletter

              There was an error while trying to send your request. Please try again.

              Zehntech will use the information you provide on this form to be in touch with you and to provide updates and marketing.