CloudifyOps Mini-blog series: Simple Scheduled Tasks for S3 using AWS Lambda Function and Amazon CloudWatch Event

Automating repeated tasks in a scheduled manner is also known for so-called cron jobs. It helps in reducing resource costs, maintaining system availability, and removing manual workloads efficiently over time.

In this article, we will walk you through the steps for creating a simple scheduled task using Amazon CloudWatch and AWS Lambda.

The AWS Resources

There are two AWS resources involved in building the scheduled task:

Amazon CloudWatch Event (Rules)
AWS Lambda (Lambda Function)

Use Case Scenario

Every morning, our cloud engineer sends a report to managers about the Performance Analysis. Technically, what he will do is call an API endpoint that will fetch data from a database and perform some data transformation to produce the reporting analysis. Once the process is done, he will send the generated report to the managers through email. He needs to do this every morning at 8:00 AM GMT+8.

We can see that sending email notifications to the managers in the morning can be automated by creating a scheduled task.

The diagram below shows how it can be done on AWS.

Let’s break down the above process flow:

The CloudWatch Rules will activate the Lambda function according to the scheduled time defined in the cron expression. In this case, the rule is configured to be triggered every day at 8:00 AM GMT+8.
The Lambda function will be activated once it is invoked by CloudWatch Events. It will call the Reporting Email API endpoint to perform the required tasks.
Then, the Reporting Email API will perform the analysis and generate the report. Once it is done, it will automatically send the email to managers.

Now, let’s get into setting up this scheduled task pipeline on AWS.

Create AWS Lambda Function

Go to AWS Management Console and log in using your AWS Account. Then, proceed to AWS Lambda in the console and click on the Create function.

Choose ’author from scratch’ and define your function name. In this tutorial, we will be creating a Python function. Thus, select Python 3.7 as runtime.

You will need to choose or create an execution role before creating the function. Once it is done, click on the Create function.

Once done, we can see our function created.

Lambda Code Editor:

In the Code tab below, we see the Code source where we can write our custom code. Any API calls can be done, including executing a complex and highly computational task in the Lambda function. For this example, we will write a code where this function will call an API endpoint.

import boto3
from datetime import datetime, timedelta
def lambda_handler(event, context):
    # Set your S3 bucket name and the number of days to retain logs
    bucket_name = "your-s3-bucket-name"
    retention_days = 30
    # Initialize the S3 client
    s3_client = boto3.client('s3')
    # Calculate the date before which logs should be deleted
    deletion_date = datetime.now() - timedelta(days=retention_days)
    # List objects in the S3 bucket
    objects = s3_client.list_objects_v2(Bucket=bucket_name)
    if 'Contents' in objects:
        for obj in objects['Contents']:
            # Get the object key (filename) and last modified date
            obj_key = obj['Key']
            last_modified = obj['LastModified']
            # Check if the object is a log and is older than the deletion date
            if obj_key.endswith('.log') and last_modified < deletion_date:
                # Delete the log object
                s3_client.delete_object(Bucket=bucket_name, Key=obj_key)
                print(f"Deleted: s3://{bucket_name}/{obj_key}")
    return {
        'statusCode': 200,
        'body': 'Log rotation and deletion complete'
    }

Click on Deploy once you have completed the code. It is done! Let’s create an Amazon CloudWatch Event to trigger this Lambda function on a timely basis.

Create Scheduled Event in the Amazon CloudWatch Console

Go to the AWS Management Console and log in using your AWS Account. Proceed to Amazon CloudWatch and click Rules under the event section in the left navigation menu. We will see the Events list and then click on Create Rule.

Event Source

In the ‘create rule’ page, the Event pattern is selected by default in the event source section. We wish to create scheduled events, therefore we need to select the Schedule option. When we select the Schedule Option, we see there are two kinds of event rules that we can define:

Fixed-rate schedule
Schedule based on cron expression

For a fixed rate, we need to define the interval and its unit. The units are minutes, hours, and days.

In our case, let’s define our schedule event rule using the cron expression. A cron expression is a string composed of 6 or 7 fields separated by white spaces. The cron expression field can consist of any of the allowed values, together with various combinations of the allowed special characters for that field.

We will define our expression that will trigger the event every day at 8:00 AM GMT+8 time zone. This is how the cron expression will look like. We can construct a very detailed schedule of events using the cron expression and depending on how intricate the use case that we are working on at the moment.

Add Targets

Click on the Add target and we will see that the Lambda function is selected by default. Now, select the function that will be our target and for this example, we will select the lambda-scheduled-task function we created earlier.

Click on Configure details once everything is done.

Configure Rule Details

We need to provide a name and description for our event rule. Make sure that Enabled is checked and click on Create Rule.

The event rule will be created and we will be redirected to the Rules list page. From here, we can see that our scheduled event is successfully created.

Now this event will be triggered every day at 8:00 AM in GMT+8 and it will also trigger our Lambda function accordingly.

In this article, we discussed the related components for scheduled tasks on AWS, how to create and configure the Lambda function, and how to create scheduled event rules using Amazon CloudWatch Event. We hope this article will help you build customized scheduled tasks on AWS.

To know more about how the CloudifyOps team can support you in your cloud journey, write to us at sales@cloudifyops.com.

CloudifyOps Mini-blog series: Simple Scheduled Tasks for S3 using AWS Lambda Function and Amazon CloudWatch Event

The AWS Resources

Use Case Scenario

Create AWS Lambda Function

Lambda Code Editor:

Create Scheduled Event in the Amazon CloudWatch Console

Event Source

Add Targets

Configure Rule Details

Services

Solutions

Recent Blogs

Our Services

Our Solutions

Offices

Contact us