Amazon DynamoDB table updates by an event driven AWS Lambda function in a multi-account environment

Introduction

This article shows how we can implement a solution in which we are able to update a AWS DynamoDB table with a AWS Lambda function which will be triggered by an AWS EventBridge rule.

In the fast-paced world of cloud computing, managing resources efficiently is crucial. Amazon Web Services (AWS) provides tools that streamline this process. Among them, Amazon DynamoDB is a powerful NoSQL database service that handles data with ease. AWS Lambda lets you run code without worrying about servers, and AWS EventBridge acts as a way to trigger actions in response to events.

One of the challenges that engineers often face when working with DynamoDB is managing table updates efficiently. Traditional methods of updating data in a DynamoDB table might involve manual interventions and complex code deployments. Fortunately, AWS presents an elegant solution through event-driven AWS Lambda functions orchestrated by AWS EventBridge rules. 

Understanding AWS Lambda and DynamoDB Integration

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. It automatically scales your application in response to incoming traffic, ensuring you pay only for the compute time you consume. Amazon DynamoDB, on the other hand, is a fully managed NoSQL database service that allows you to store and retrieve data with high performance and scalability. 

When you integrate AWS Lambda with DynamoDB, you enable seamless and efficient updates to DynamoDB tables through event-driven triggers. This means for instance that you can set up a Lambda function to automatically execute whenever specific changes occur in your DynamoDB table. These changes can include inserts, updates, or deletions of items in the table.

Use Case: Automating Instance License Management with AWS EventBridge and Lambda

Let's dive into a use case that demonstrates how these services can work together to automate resource management.

The Scenario

Imagine a scenario where a large international company's cloud infrastructure, with resources deployed all over the globe in different AWS regions and accounts, relies heavily on Linux instances in the cloud. Each of those instances is running a third-party application that requires a license. To efficiently manage these licenses and to not manually keep track of the table which can be very time-consuming and error-prone, an event-driven solution is being developed. Or to put it simply, each time a Linux instance has been started, a Lambda function needs to update the license inventory, respectively the DynamoDB table. 

To automate this use case, you can leverage the following AWS services:

Destination Account

Amazon DynamoDB: Create a central DynamoDB table (e.g. in the shared service account) to store information about each Linux instance in each AWS organization account. This includes the licenses, instance ARNs and IDs and any other relevant details such as types or launch times.

CloudFormationDynamoDB.yaml

AWS Lambda: Develop a Lambda function that adds new instance information to the DynamoDB table whenever a Linux instance is launched in one of the organization accounts.

LambdaUpdateDynamoDB.pyCloudFormationLambdaFunction.yaml

AWS EventBridge Rule: Set up an EventBridge rule in the target account respectively “shared service account” to detect the incoming event from the source account. This rule finally will act as a trigger for the Lambda function.

CloudFormationEventBridgeRuleTargetAccount.yaml

Source Account

AWS EventBridge Rule: Set up an EventBridge rule to detect when a Linux instance is started. This rule will forward the event to the destination account above.

CloudFormationEventBridgeRuleSourceAccount.yaml

Note: 

• All the source code is also available on Github: https://github.com/mulou/nc-blogpost-ddb-evb-lambda
• The CloudFormation code to set up the source event bridge rule has a dependency to the destination rule. Therefore, please follow the setup order mentioned above.

Architecture

The very high level architecture could look something like this:

How It Works

• (1) Instance Launch: Whenever a new Linux instance is launched in your AWS environment, AWS EventBridge detects this event.
• (2) EventBridge Rule: The EventBridge rule in the source account is designed to match this event and triggers the associated EventBridge rule in the target account.
• (3) Lambda Function: The Lambda function, which you've pre-configured, gets invoked by the EventBridge rule in the target account. This function extracts information about the launched instance, such as its ID, type, and launch time.
• (4) DynamoDB Interaction: The Lambda function then interacts with the DynamoDB table, gets a free license key for the instance and adds a new entry containing all the instance details. This effectively creates a centralized repository for all instance-related information.

You can test your setup by performing actions that trigger the EventBridge rule you have configured. Monitor the logs and CloudWatch metrics to ensure that your Lambda function is being invoked and the table has been updated as expected.

Note: The instance termination and therefore the licenses release procedure is out of scope for this post. Eventually I will explain this in more detail on the following blog.

Benefits of Automation:

• Efficiency: No more manual logging or data entry. The process of capturing instance details is automated and immediate.
• Accuracy: Human errors in recording instance details are eliminated, ensuring accurate and up-to-date information.
• Centralization: All instance details are stored in a single DynamoDB table, making it easy to monitor and manage instances at a glance.
• Timeliness: As soon as an instance is launched, its details are captured and stored. There's no delay in updating the information.
• Scalability: This setup scales effortlessly. Whether you're launching one instance or a hundred, the process remains consistent and automated.

Conclusion

This use case demonstrates the power of AWS services working in harmony. By combining Amazon DynamoDB, AWS Lambda and AWS EventBridge, you can automate resource management tasks, making your operations more efficient, accurate, and scalable. This approach showcases the practical magic of cloud computing, where technology takes care of the repetitive tasks, allowing you to focus on higher-value activities.

Reference

• Boto3 Doc DynamoDB
• CloudFormation AWS::DynamoDB::Table