Automate Inventory of AWS API Gateways and Lambda functions associations.

Introduction

In today's cloud-driven world, managing AWS resources efficiently is crucial for businesses. When deprecating AWS accounts, one of the challenges is identifying the connections between various services, especially API Gateways and Lambda functions. While the approach outlined in this post provides a solid foundation for gathering the required information, it's important to note that this is an example rather than a full end-to-end solution. A full scale deprecation strategy may involve additional dependencies and complexities beyond API Gateway and Lambda integrations.

This task also becomes essential to ensure that no critical service is accidentally disabled, leading to potential downtime or data loss. In this post, we'll walk through the process of creating a comprehensive inventory of these associations, providing you with the necessary insights to manage your AWS environment effectively.

Background

Our customer is preparing to deprecate several AWS accounts as part of their cloud resource and cost optimisation strategy. To do this smoothly, they need to map out which API Gateways are connected to which Lambda functions. This is essential not just for understanding the architecture, but also for ensuring that key services continue running smoothly during the transition. The following sections will explore the steps needed to create this inventory, with a focus on gathering data.

Understanding AWS API Gateway and Lambda integration

Before diving into the technical details, it's important to understand how AWS API Gateway and Lambda functions interact. API Gateway serves as a front door for applications to access data, business logic, or functionality from backend services, which are often implemented using Lambda functions. This integration allows for serverless architectures that are highly scalable and cost-effective.

However, in a real-world scenario, other services like DynamoDB, S3, or Step Functions might also be involved, and these should be considered when planning a full deprecation strategy.

Automating the inventory process using a Lambda Function

To make it easier to link AWS API Gateway resources with Lambda functions, we created a Lambda function that handles this process automatically. Below is a walkthrough of the core components of the function, explain its logic, and discuss any AWS limitations that influenced its design.

To make it easier to link AWS API Gateway resources with Lambda functions, we created a Lambda function that handles this process automatically. It's important to emphasise that this Lambda function operates within a single AWS account and region. For larger environments with multiple accounts or regions, this process would need to be replicated or enhanced to cover cross-account or cross-region use cases. Below is a walkthrough of the core components of the function, explaining its logic, and discussing any AWS limitations that influenced its design.

Core Components of the Lambda Function:

1. Initialisation:
   The function initialises the necessary AWS SDK (Boto3) clients for API Gateway and Lambda within a specified AWS region. This example focuses on a single region, but for a comprehensive solution, you'd need to handle multiple regions and accounts.
2. Fetching API Gateway data:
   The function retrieves all available API Gateways within the region using the get_rest_apis() method. Given the potential large number of APIs, a limit of 100 is set per request to adhere to AWS limits, but this can be adjusted based on your needs.
3. Processing API Gateways:
   For each API Gateway, the function:
   • Fetches its associated tags using get_tags().
   • Retrieves resource integrations to identify Lambda functions connected to the API Gateway using the get_resources() and get_integration() methods.
4. Extracting Lambda ARNs:
   The function parses the integration URIs to extract Lambda ARNs, handling various URI formats. This step is critical as Lambda ARNs can appear in different formats, making it necessary to accurately extract and standardise them.
5. Handling AWS limitations:
   • Rate Limits: The function implements logging to track progress and handles potential AWS API rate limits by structuring the API calls with error handling.
   • Resource Tagging: Fetching tags in AWS has certain constraints. The function addresses these by directly querying the tags for both API Gateway and Lambda functions.
6. Storing results in S3:
   The function compiles the gathered data into a JSON file and uploads it to an S3 bucket for easy access and further analysis. In our case, to make the data more readable and easier to handle, we imported the JSON file into Excel, allowing the project manager to review the data and make informed decisions based on the spreadsheet.

Code Explanation

import json
import boto3
import time 
import logging
import re

# Setup logger for debugging and monitoring
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def main(event, context):
    # Specify the AWS region
    region = 'us-east-1'
    api_client = boto3.client('apigateway', region_name=region)

    # Retrieve API Gateway information
    response = api_client.get_rest_apis(limit=100)
    apis = response['items']
    data = []

    # Process each API Gateway
    for api in apis:
        api_name = api['name']
        api_id = api['id']

        # Fetch tags and integrations
        api_tags = get_api_tags(api_client, api_id, region)
        api_integrations = get_resource_integrations(api_client, api_id, region)
        data.append({
            'API ID': api_id, 
            'API Name': api_name,
            'API Tags': api_tags,
            'API Integrations': api_integrations,
        })
    
    # Upload data to S3
    upload_s3(data, region)
    
    return {
        'statusCode': 200,
        'body': json.dumps('Successful')
    }

# Additional helper functions
# (Include a brief explanation for each of the helper functions as done previously)

# Example:
def get_resource_integrations(api_client, api_id, region):
    # This function retrieves the Lambda function integrations for each API resource
    # and extracts the relevant ARNs and tags.
    ...

Implementing the solution

• Gathering data: Describe the process of using AWS CLI or SDKs to extract relevant data. Include sample commands or code snippets. This gathering process is per account and per region, meaning that for organisations with multiple accounts or operating in multiple regions, the process would need to be repeated or automated across those environments.
• Organising the inventory: Discuss the best practices for organising this data, possibly in a CSV file or a database, and how to ensure it is easily accessible and understandable.
• Discuss the best practices for organising this data, possibly in a CSV file or a database, and how to ensure it is easily accessible and understandable. There are also commercial tools available that can help with inventory management, and AWS services like AWS Config might offer alternatives or complements to this approach.
• Automating the process: Explore options for automating this inventory process using scripts or AWS services like Lambda itself for continuous monitoring. While this post focuses on API Gateway and Lambda, a broader inventory might include other AWS resources that are critical to your infrastructure.

Preparing for deprecation

Once the inventory is ready, outline the steps to take before deprecating any AWS account:

• Verification: Double-check associations to ensure all critical services are accounted for. It's important to verify that no other critical dependencies are overlooked during this process.
• Migration planning: If needed, plan for migrating resources to another AWS account. Consider using automation tools to assist with this process to minimise downtime and errors.
• Communication: Make sure stakeholders are aware of the changes and have signed off on the deprecation plan. It's also crucial to have a rollback plan in place in case something doesn't go as expected.

Conclusion

By integrating automation into your AWS inventory process, just like the Lambda function we highlighted, you can significantly reduce manual work and the potential risk of human errors. This function not only helps in linking API Gateways to Lambda functions but also ensures you're all set for any AWS account changes by giving you a full picture of your infrastructure.

By integrating automation into your AWS inventory process, just like the Lambda function we highlighted, you can significantly reduce manual work and the potential risk of human errors. This function not only helps in linking API Gateways to Lambda functions but also ensures you're all set for any AWS account changes by giving you a full picture of your infrastructure. Remember, this example focuses on gathering information, but a complete solution would involve organising and possibly automating further aspects of your AWS environment, accounting for all potential dependencies.