Join us at GITEX 2024! Discover our solutions at Hall 4, Booth H-30 Schedule a Meeting Today.
Automate Marketing Initiatives with Salesforce Marketing Cloud Learn More
Join us at GITEX 2024! Discover our solutions at Hall 4, Booth H-30 Book your live demo today.

How to Automate Instance Snapshot Management in AWS EC2

To create a backup of your EC2 instances EBS snapshots play an important role. In this article, we are going to automate EBS volume snapshots, using Lambda and CloudWatch Events function for a daily backup and delete the snapshots after a particular limit of time.

 

Prerequisites

  • AWS account
  • IAM user in AWS account
  • IAM user must have permission to access services for creating Lambda function.
  • We need an IAM role to define what permissions our lambda function has.
  • CloudWatch permission to create Rules.

To Create IAM Role

Log in to the AWS console with root account or with IAM user who has permission to create Roles.

 

  • Click on Services and select IAM services.
  • IAM services page will open from the left panel click on Create Role.
  • Create Role page will open select lambda service(Because we need permission to access lambda function).
  • On selecting lambda function it will invite you to attach permission and policies to the role. but we will create our customer policy.

 

We want our function able to perform the following:

 

  • Read all type of information from EC2 (we will give it fully describe and read access)
  • Create and Delete Snapshots
  • Create and Delete Snapshots
  • Create and access Cloudwatch Event.


To create the above policy using a visual editor or by providing a JSON. You can use the below JSON policy.

 

{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "logs:*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": "ec2:Describe*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot",
                    "ec2:CreateTags",
                    "ec2:DeleteTags",
                    "ec2:ModifySnapshotAttribute"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "logs:*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": "ec2:Describe*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot",
                    "ec2:CreateTags",
                    "ec2:DeleteTags",
                    "ec2:ModifySnapshotAttribute"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "logs:*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": "ec2:Describe*",
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot",
                    "ec2:CreateTags",
                    "ec2:DeleteTags",
                    "ec2:ModifySnapshotAttribute"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }

 

Tagging

Lambda function uses EC2 instance tags to identify for which instance the snapshot will create.

 

  • From EC2 Services select the EC2 instance for which we want to create a snapshot.
  • From the Tags section adds the below tag.
  • The name of the tag is “auto_snapshot” value for the tag is “true”.
  • Save the changes by clicking the Save button.

Lambda function to create a Snapshot

1

Navigate to the AWS Lambda Management Console from Services.

  • Click on Create function new page will open.
  • Select Author from Scratch.
  • Give a specific name for creating a lambda function in the Function name.
  • Choose Python 3.6 as a runtime.
  • For roles select Use an existing role and select the role we made earlier.

Code

I m using Boto3. Boto3 is SDK for python, which provides programmatic connectivity to create, configure, and manage AWS services with python.

 

import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           
import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           
import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           
import boto3
import collections
import datetime
from pprint import pprint
 
ec = boto3.client('ec2')

 
def lambda_handler(event, context):
    reservations = ec.describe_instances(
        Filters=[
            {'Name': 'tag-key', 'Values': ['auto_snapshot', 'true']},
        ]
    ).get(
        'Reservations', []
    )
 
    instances = sum(
        [
            [i for i in r['Instances']]
            for r in reservations
        ], [])
 
    print("Found {0} instances that need backing up".format(len(instances)))
 
    to_tag = collections.defaultdict(list)
    for instance in instances:
        pprint(instance)
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'][0]
        except IndexError:
            retention_days = 15
 
        for dev in instance['BlockDeviceMappings']:
            if dev.get('Ebs', None) is None:
                continue
            vol_id = dev['Ebs']['VolumeId']
            print("Found EBS volume {0} on instance {1}".format(
                vol_id, instance['InstanceId']))
 
            snap = ec.create_snapshot(
                VolumeId=vol_id,
            )
            
            to_tag[retention_days].append(snap['SnapshotId'])
            
            for tags in instance['Tags']:
                if tags["Key"] == 'Name':
                    instancename = tags["Value"]
            
            print("Retaining snapshot {0} of volume {1} from instance {2} for {3} days for {4}".format(
                snap['SnapshotId'],
                vol_id,
                instance['InstanceId'],
                retention_days,
                instancename
            ))
            delete_date = datetime.date.today() + datetime.timedelta(days=retention_days)
            delete_fmt = delete_date.strftime('%Y-%m-%d')
            print("Will delete {0} snapshots on {1}".format(len(to_tag[retention_days]), delete_fmt))
            print("instance id now ")
            ec.create_tags( Resources=[snap['SnapshotId']],Tags=[
                            {'Key': 'automatic-snapshot-delete-on', 'Value': delete_fmt},
                            {'Key': 'Name', 'Value': instancename},
                            {'Key': 'Instance ID', 'Value': instance['InstanceId']}
            ])
           

Every time when the above function run, it will perform the following steps:

 

  • Create a snapshot for all EC2 instance which tagged with auto_snapshot: true
  • Delete old created snapshots which were made by these function
  • You can use a tag for retention days. Create a tag in EC2 instance with name “retention_days” with value “10” (you can use any number of days according to your requirements)
  • You can also add a value of retention days in lambda function retention_days = 10. If you have not made any changes in the above lambda code by default it takes 15 days for retention.
  • Lambda function for creating a snapshot creates a “Deletion” tag with “Date” Which calculated based on the retention days. Delete function deletes the snapshots older than the retention days.
  • We need to change Basic Settings for the lambda function, the default timeout for the lambda function is 3seconds, we need to increase it (I used 59 sec) otherwise, the create function will timeout before completing the snapshots correctly.
  • You can use a tag for retention days. Create a tag in EC2 instance with name “retention_days” with value “10” (you can use any number of days according to your requirements)
  • You can also add a value of retention days in lambda function retention_days = 10. If you have not made any changes in the above lambda code by default it takes 15 days for retention.
  • Lambda function for creating a snapshot creates a “Deletion” tag with “Date” Which calculated based on the retention days. Delete function deletes the snapshots older than the retention days.
  • We need to change Basic Settings for the lambda function, the default timeout for the lambda function is 3seconds, we need to increase it (I used 59 sec) otherwise, the create function will timeout before completing the snapshots correctly.

CloudWatch Rule to create Snapshot

The lambda function is created, and we use the CloudWatch rule to automate the lambda function.

 

  • Navigate to CloudWatch Management Console from Services.
  • From CloudWatch page move to Rules from left panel
  • Click on Create rule. A new page will open.
  • Use the below settings to create a rule.
2

The above settings create an EC2 snapshot every day at 8:00 am (timezone GMT). You can use cron expression according to your requirement.

 

Targets are used to select the Lambda function for which the rule is used.

 

You can also use a Fixed rate of option to create a snapshot if you want a snapshot after a fixed number of minutes Hours and Days.

 

  • Back to the Lambda Management Console Function.
  • Select recently created function. The created lambda function page will open.
  • Click on Add trigger button from a Designer section.
  • Select the CloudWatch Events to trigger.
  • Use recently created CloudWatch rule from existing rules and click on Add.
  • The lambda function to create a snapshot for Ec2 instance is completed. You can test it by clicking the Test button.

 

A place for big ideas.

Reimagine organizational performance while delivering a delightful experience through optimized operations.

Lambda function to Delete a Snapshot

It is very important to manage to create or delete function as we are creating a lot of snapshots(according to our requirements) and if we do not delete it the number of snapshots will be created and we need to delete it manually.

 

Please follow the below steps to automate the delete function.

 

  • Click on Create function new page will open.
  • Select Author from Scratch.
  • Give a specific name for creating a lambda function in the Function name.
  • Choose Python 3.6 as a runtime.
  • For roles select Use an existing role and select the role we made earlier.

Code

import boto3
import re
import datetime
from datetime import timedelta
 
# Set the global variables
globalVars  = {}
globalVars['RetentionDays'] = "15"

ec = boto3.client('ec2')
iam = boto3.client('iam')
 
def lambda_handler(event, context):
    account_ids = list()
    try:
        """
        You can replace this try/except by filling in `account_ids` yourself.
        Get your account ID with:
        > import boto3
        > iam = boto3.client('iam')
        > print iam.get_user()['User']['Arn'].split(':')[4]
        """
        iam.get_user()
    except Exception as e:
        # use the exception message to get the account ID the function executes under
        account_ids.append(re.search(r'(arn:aws:sts::)([0-9]+)', str(e)).groups()[1])
 
    retention_days = ( datetime.date.today() - datetime.timedelta(days= int(globalVars['RetentionDays'])) ).strftime('%Y-%m-%d')
    delete_on = datetime.date.today().strftime('%Y-%m-%d')
    filters = [
        {'Name': 'tag-key', 'Values': ['automatic-snapshot-delete-on']},
        {'Name': 'tag-value', 'Values': [delete_on]},
    ]
    snapshot_response = ec.describe_snapshots(OwnerIds=account_ids, Filters=filters)
    print("snapshot_response {0}".format(snapshot_response))
    
    all_snap = ec.describe_snapshots(OwnerIds=account_ids)
    for snap in all_snap['Snapshots']:
        snapshot_time = snap['StartTime'].strftime('%Y-%m-%d')
        if snapshot_time <= retention_days:
            snapshot_response['Snapshots'].append(snap)
 
    for snap in snapshot_response['Snapshots']:
        print("Deleting snapshot {0}".format(snap['SnapshotId']))
        ec.delete_snapshot(SnapshotId=snap['SnapshotId'])

 

Every time when the above function run, it will perform the following steps:

 

  • Delete the Snapshot according to the tag of EC2 instance. Otherwise, it takes the default retention_days value “15”. It deletes all the Snapshots older than 15days.
  • Also, change Basic Settings for the lambda function, the default timeout for the lambda function is 3seconds, we need to increase it (I used 59 sec) otherwise, the delete function will timeout before deleting the snapshots correctly.

CloudWatch Rule

Now the lambda function is created, and we can also use the CloudWatch rule to automate delete lambda function.

 

  • Navigate to CloudWatch Management Console from Services.
  • From CloudWatch page move to Rules from left panel
  • Click on Create rule. A new page will open.
  • Use the below settings to create a rule.
3

The above settings delete an EC2 snapshot which is older than 10 days.

 

  • Back to the Lambda Management Console Function.
  • Select recently created delete function. Delete the lambda function page will open.
  • Click on Add trigger button from a Designer section.
  • Select the CloudWatch Events to trigger.
  • Use recently created CloudWatch rule from existing rules and click on Add.


The lambda function to delete a snapshot for EC2 instance is completed.

Top Stories

Enhancing GraphQL with Roles and Permissions
Enhancing GraphQL with Roles and Permissions
GraphQL has gained popularity due to its flexibility and efficiency in fetching data from the server. However, with great power comes great responsibility, especially when it comes to managing access to sensitive data. In this article, we'll explore how to implement roles and permissions in GraphQL APIs to ensure that
Exploring GraphQL with FastAPI A Practical Guide to begin with
Exploring GraphQL with FastAPI: A Practical Guide to begin with
GraphQL serves as a language for asking questions to APIs and as a tool for getting answers from existing data. It's like a translator that helps your application talk to databases and other systems. When you use GraphQL, you're like a detective asking for specific clues – you only get
Train tensorflow object detection model with custom data
Train Tensorflow Object Detection Model With Custom Data
In this article, we'll show you how to make your own tool that can recognize things in pictures. It's called an object detection model, and we'll use TensorFlow to teach it. We'll explain each step clearly, from gathering pictures, preparing data to telling the model what to look for in
Software Development Team
How to deploy chat completion model over EC2?
The Chat Completion model revolutionizes conversational experiences by proficiently generating responses derived from given contexts and inquiries. This innovative system harnesses the power of the Mistral-7B-Instruct-v0.2 model, renowned for its sophisticated natural language processing capabilities. The model can be accessed via Hugging Face at – https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2.Operating on a dedicated GPU server g4dn.2xlarge,
How to deploy multilingual embedding model over EC2
How to deploy multilingual embedding model over EC2?
The multilingual embedding model represents a state-of-the-art solution designed to produce embeddings tailored explicitly for chat responses. By aligning paragraph embeddings, it ensures that the resulting replies are not only contextually relevant but also coherent. This is achieved through leveraging the advanced capabilities of the BAAI/bge-m3 model, widely recognized for
Tracking and Analyzing E commerce Performance with Odoo Analytics
Tracking and Analyzing E-commerce Performance with Odoo Analytics
Odoo is famous for its customizable nature. Businesses from around the world choose Odoo because of its scalability and modality. Regardless of the business size, Odoo can cater to the unique and diverse needs of any company. Odoo has proven its capacity and robust quality in terms of helping businesses

          Success!!

          Keep an eye on your inbox for the PDF, it's on its way!

          If you don't see it in your inbox, don't forget to give your junk folder a quick peek. Just in case.









              You have successfully subscribed to the newsletter

              There was an error while trying to send your request. Please try again.

              Zehntech will use the information you provide on this form to be in touch with you and to provide updates and marketing.