Lambda layers for Python runtime

CATEGORIES

Tech

AWS Lambda

AWS Lambda is one of the most popular serverless compute services in the public cloud, released in November 2014. It runs your code in the response to events like DynamoDB, SNS or HTTP triggers without provisioning or managing any infrastructure. Lambda takes care of most of the things required to run your code and provides high availability. It allows you to execute even up to 1000 parallel functions at once! Using AWS lambda you can build applications like:

  • Web APIs
  • Data processing pipelines
  • IoT applications
  • Mobile backends
  • and many many more…

Creating AWS Lambda is super simple: you just need to create a zip file with your code, dependencies and upload it to S3 bucket. There are also frameworks like serverless or SAM that handles deploying AWS lambda for you, so you don’t have to manually create and upload the zip file.

There is, however, one problem.

You have created a simple function which depends on a large number of other packages. AWS lambda requires you to zip everything together. As a result, you have to upload a lot of code that never changes what increases your deployment time, takes space, and costs more.

AWS Lambda Layers

Fast forward 4 years later at 2018 re:Invent AWS Lambda Layers are released. This feature allows you to centrally store and manage data that is shared across different functions in the single or even multiple AWS accounts! It solves a certain number of issues like:

  • You do not have to upload dependencies on every change of your code. Just create an additional layer with all required packages.
  • You can create custom runtime that supports any programming language.
  • Adjust default runtime by adding data required by your employees. For example, there is a team of Cloud Architects that builds Cloud Formation templates using the troposphere library. However, they are no developers and do not know how to manage python dependencies… With AWS lambda layer you can create a custom environment with all required data so they could code in the AWS console.

But how does the layer work?

When you invoke your function, all the AWS Lambda layers are mounted to the /opt directory in the Lambda container. You can add up to 5 different layers. The order is really important because layers with the higher order can override files from the previously mounted layers. When using Python runtime you do not need to do any additional operations in your code, just import library in the standard way. But, how will my python code know where to find my data?

That’s super simple, /opt/bin is added to the $PATH environment variable. To check this let’s create a very simple Python function:


import os
def lambda_handler(event, context):
    path = os.popen("echo $PATH").read()
    return {'path': path}

The response is:

 
{
    "path": "/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin\n"
}

 

Existing pre-defined layers

AWS layers have been released together with a single, publicly accessible library for data processing containing 2 libraries: NumPyand SciPy. Once you have created your lambda you can click  `Add a layer` in the lambda configuration. You should be able to see and select the AWSLambda-Python36-SciPy1x layer. Once you have added your layer you can use these libraries in your code. Let’s do a simple test:


import numpy as np
import json


def lambda_handler(event, context):
    matrix = np.random.randint(6, size=(2, 2))
    
    return {
        'matrix': json.dumps(matrix.tolist())
    }

The function response is:

 >code>
{
  "matrix": "[[2, 1], [4, 2]]"
}

 

As you can see it works without any effort.

What’s inside?

Now let’s check what is in the pre-defined layer. To check the mounted layer content I prepared simple script:


import os
def lambda_handler(event, context):
    directories = os.popen("find /opt/* -type d -maxdepth 4").read().split("\n")
    return {
        'directories': directories
    }

In the function response you will receive the list of directories that exist in the /opt directory:


{
  "directories": [
    "/opt/python",
    "/opt/python/lib",
    "/opt/python/lib/python3.6",
    "/opt/python/lib/python3.6/site-packages",
    "/opt/python/lib/python3.6/site-packages/numpy",
    "/opt/python/lib/python3.6/site-packages/numpy-1.15.4.dist-info",
    "/opt/python/lib/python3.6/site-packages/scipy",
    "/opt/python/lib/python3.6/site-packages/scipy-1.1.0.dist-info"
  ]
}

Ok, so it contains python dependencies installed in the standard way and nothing else. Our custom layer should have a similar structure.

Create Your own layer!

Our use case is to create an environment for our Cloud Architects to easily build Cloud Formation templates using troposphere and awacs libraries. The steps comprise:
<h3″>Create virtual env and install dependencies

To manage the python dependencies we will use pipenv.

Let’s create a new virtual environment and install there all required libraries:


pipenv --python 3.6
pipenv shell
pipenv install troposphere
pipenv install awacs

It should result in the following Pipfile:


[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
troposphere = "*"
awacs = "*"
[dev-packages]
[requires]
python_version = "3.6"

Build a deployment package

All the dependent packages have been installed in the $VIRTUAL_ENV directory created by pipenv. You can check what is in this directory using ls command:

 
ls $VIRTUAL_ENV

Now let’s prepare a simple script that creates a zipped deployment package:


PY_DIR='build/python/lib/python3.6/site-packages'
mkdir -p $PY_DIR                                              #Create temporary build directory
pipenv lock -r > requirements.txt                             #Generate requirements file
pip install -r requirements.txt --no-deps -t $PY_DIR     #Install packages into the target directory
cd build
zip -r ../tropo_layer.zip .                                  #Zip files
cd ..
rm -r build                                                   #Remove temporary directory

When you execute this script it will create a zipped package that you can upload to AWS Layer.

 

Create a layer and a test AWS function

You can create a custom layer and AWS lambda by clicking in AWS console. However, real experts use CLI (AWS lambda is the new feature so you have to update your awscli to the latest version).

To publish new Lambda Layer you can use the following command (my zip file is named tropo_layer.zip):


aws lambda publish-layer-version --layer-name tropo_test --zip-file fileb://tropo_layer.zip

As the response, you should receive the layer arn and some other data:


{
    "Content": {
        "CodeSize": 14909144,
        "CodeSha256": "qUz...",
        "Location": "https://awslambda-eu-cent-1-layers.s3.eu-central-1.amazonaws.com/snapshots..."
    },
    "LayerVersionArn": "arn:aws:lambda:eu-central-1:xxxxxxxxxxxx:layer:tropo_test:1",
    "Version": 1,
    "Description": "",
    "CreatedDate": "2018-12-01T22:07:32.626+0000",
    "LayerArn": "arn:aws:lambda:eu-central-1:xxxxxxxxxxxx:layer:tropo_test"
}

The next step is to create AWS lambda. Yor lambda will be a very simple script that generates Cloud Formation template to create EC2 instance:

 
from troposphere import Ref, Template
import troposphere.ec2 as ec2
import json
def lambda_handler(event, context):
    t = Template()
    instance = ec2.Instance("myinstance")
    instance.ImageId = "ami-951945d0"
    instance.InstanceType = "t1.micro"
    t.add_resource(instance)
    return {"data": json.loads(t.to_json())}

Now we have to create a zipped package that contains only our function:


zip tropo_lambda.zip handler.py

And create new lambda using this file (I used an IAM role that already exists on my account. If you do not have any role that you can use you have to create one before creating AWS lambda):


aws lambda create-function --function-name tropo_function_test --runtime python3.6 
--handler handler.lambda_handler 
--role arn:aws:iam::xxxxxxxxxxxx:role/service-role/some-lambda-role 
--zip-file fileb://tropo_lambda.zip

In the response, you should get the newly created lambda details:


{
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "CodeSha256": "l...",
    "FunctionName": "tropo_function_test",
    "CodeSize": 356,
    "RevisionId": "...",
    "MemorySize": 128,
    "FunctionArn": "arn:aws:lambda:eu-central-1:xxxxxxxxxxxx:function:tropo_function_test",
    "Version": "$LATEST",
    "Role": "arn:aws:iam::xxxxxxxxx:role/service-role/some-lambda-role",
    "Timeout": 3,
    "LastModified": "2018-12-01T22:22:43.665+0000",
    "Handler": "handler.lambda_handler",
    "Runtime": "python3.6",
    "Description": ""
}

Now let’s try to invoke our function:


aws lambda invoke --function-name tropo_function_test --payload '{}' output
cat output
{"errorMessage": "Unable to import module 'handler'"}

Oh no… It doesn’t work. In the CloudWatch you can find detailed log message: `Unable to import module ‘handler’: No module named ‘troposphere’` This error is obvious. Default python3.6 runtime does not contain troposphere library. Now let’s add layer we created in the previous step to our function:


aws lambda update-function-configuration --function-name tropo_function_test --layers arn:aws:lambda:eu-central-1:xxxxxxxxxxxx:layer:tropo_test:1

When you invoke lambda again you should get the correct response:


{
  "data": {
    "Resources": {
      "myinstance": {
        "Properties": {
          "ImageId": "ami-951945d0",
          "InstanceType": "t1.micro"
        },
        "Type": "AWS::EC2::Instance"
      }
    }
  }
}

Add a local library to your layer

We already know how to create a custom layer with python dependencies, but what if we want to include our local code? The simplest solution is to manually copy your local files to the /python/lib/python3.6/site-packages directory.

First, let prepare the test module that will be pushed to the layer:


$ find local_module
local_module
local_module/__init__.py
local_module/echo.py
$ cat cat local_module/echo.py
def echo_hello():
    return "hello world!"

To manually copy your local module to the correct path you just need to add the following line to the previously used script (before zipping package):


cp -r local_module 'build/python/lib/python3.6/site-packages'

This works, however, we strongly advise transforming your local library into the pip module and installing it in the standard way.

Update Lambda layer

To update lambda layer you have to run the same code as before you used to create a new layer:


aws lambda publish-layer-version --layer-name tropo_test --zip-file fileb://tropo_layer.zip

The request should return LayerVersionArn with incremented version number (arn:aws:lambda:eu-central-1:xxxxxxxxxxxx:layer:tropo_test:2 in my case).

Now update lambda configuration with the new layer version:

 
aws lambda update-function-configuration --function-name tropo_function_test --layers arn:aws:lambda:eu-central-1:xxxxxxxxxxxx:layer:tropo_test:2

Now you should be able to import local_module in your code and use the echo_hello function.

 

Serverless framework Layers support

Serverless is a framework that helps you to build applications based on the AWS Lambda Service. It already supports deploying and using Lambda Layers. The configuration is really simple – in the serverless.yml file, you provid the path to the layer location on your disk (it has to path to the directory – you cannot use zipped package, it will be done automatically). You can either create a separate serverless.yml configuration for deploying Lambda Layer or deploy it together with your application.

We’ll show the second example. However, if you want to benefit from all the Lambda Layers advantages you should deploy it separately.


service: tropoLayer
package:
  individually: true
provider:
  name: aws
  runtime: python3.6
layers:
  tropoLayer:
    path: build             # Build directory contains all python dependencies
    compatibleRuntimes:     # supported runtime
      - python3.6
functions:
  tropo_test:
    handler: handler.lambda_handler
    package:
      exclude:
       - node_modules/**
       - build/**
    layers:
      - {Ref: TropoLayerLambdaLayer } # Ref to the created layer. You have to append 'LambdaLayer'
string to the end of layer name to make it working

I used the following script to create a build directory with all the python dependencies:


PY_DIR='build/python/lib/python3.6/site-packages'
mkdir -p $PY_DIR                                              #Create temporary build directory
pipenv lock -r > requirements.txt                             #Generate requirements file
pip install -r requirements.txt -t $PY_DIR                   #Install packages into the target direct

This example individually packs a Lambda Layer with dependencies and your lambda handler. The funny thing is that you have to convert your lambda layer name to be TitleCased and add the `LambdaLayer` suffix if you want to refer to that resource.

Deploy your lambda together with the layer, and test if it works:


sls deploy -v --region eu-central-1
sls invoke -f tropo_test --region eu-central-1

Summary

It was a lot of fun to test Lambda Layers and investigate how it technically works. We will surely use it in our projects.

In my opinion, AWS Lambda Layers is a really great feature that solves a lot of common issues in the serverless world. Of course, it is not suitable for all the use cases. If you have a simple app, that does not require a huge number of dependencies it’s easier for you to have everything in the single zip file because you do not need to manage additional layers.

Read more on AWS Lambda in our blog!

Notes from AWS re:Invent 2018 – Lambda@edge optimisation

Running AWS Lambda@Edge code in edge locations

Amazon SQS as a Lambda event source

Blog

Minimizing AWS Lambda deployment package size in TypeScript

Our Senior Developer Vitalii explains how to significantly reduce the deployment package size of AWS Lambda functions written in TypeScript...

Blog

Problems with DynamoDB Single Table Design

Single Table Design is a database design pattern for DynamoDB based applications. In this article we take a look at...

Blog

Modular GraphQL server

Read about Kari's experiences with GraphQL modules!

Get in Touch

Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.








How to accelerate digital transformation with culture, APIs, and cloud

CATEGORIES

Insights

Digital transformation is constantly changing the way businesses and consumers interact with each other. In the simplest scenario, digitalisation is implemented by mapping a “physical” information process to a digital counterpart, aiming to produce the service more quickly and at a lower cost to customers. However, the greatest opportunity for using digitalisation is in business innovations that could disrupt the market. This may lead to a significant competitive advantage and should be the main objective for all companies going digital.

We will briefly explain how to unleash the potential of digital transformation for your business, enabling you to cut costs and lead times for your future innovation projects.

 

Develop a culture of experimentation

To begin harnessing the benefits of digital business opportunities, your company will need to set up an innovation process including idea gathering, validation and creation of prototypes. The customer should be involved with the innovation process to enable constant feedback and validation. Doing so leads to continuous refinement of the product and a better match with customer expectations. Since the input from the market may be constantly shifting the final objective of the project, agile methodologies of software development should be chosen over waterfall models.

The ability to validate concepts, as well as dismiss those that do not resonate with your customers, will quickly maximize the number of ideas that can be experimented with, and therefore increase the probability of focusing on initiatives that are more likely to succeed. Using this method of working will put your business in a competitive position in relation to your competitors.

An example of this culture can be taken from the gaming industry, where Supercell dismissed fourteen potential projects in the process of creating its four blockbuster games. Only a small proportion of the projects started were ultimately developed into a finished product and launched to the public.

APIfy your business to accelerate innovation

To accelerate digital innovation, ensure that your internal business applications and processes are available externally via an easy to use but secure API. Loose coupling of your internal business data with external customer-facing applications reduces costs and accelerates innovation by preventing the need to repeatedly connect the source application to your external application to update recent changes.

Not only do APIs improve productivity for creating new applications, but they may also enable customer, community or partner-driven application development initiatives, allowing third parties to build innovative applications on top of your own data. These third-party applications can lead to improved sales of your core offering and better customer satisfaction with minimal investments. In addition to the improved sales of your core offering, APIs may open new business opportunities via monetization of the data provided by the APIs.

In the context of APIs, remember that developers are your customers. For successful API adoption, optimal developer experience, i.e. capability to find, understand and utilize the API, is crucial.

 

Focus on value-adding work with the cloud

Modern cloud platforms offer infrastructure (as a service) but also building blocks (Platform as a Service) that enable developers to focus on the development of the actual service, whilst leaving the infrastructure work to the cloud provider.

In addition to traditional IT services such as storage and databases, cloud platforms also provide a large portfolio of more sophisticated services, including Internet of Things, data processing, and analysis; all at the click of a button. Such platforms enable development of high-end data-intensive solutions, with minimal up-front investment or commitment, minimal development effort and minimal delays.

Cloud platforms are the perfect environment in which to develop and maintain applications from concept to production, as they offer both the flexibility required in the prototyping phase and the scalability and durability required by production systems.

Blog

Nordcloud positioned in Gartner’s Magic Quadrant for Public Cloud Infrastructure Professional and Managed Services, Worldwide

We are joining the group of few Google Cloud Premier Partners and MSPs.

Blog

How Do Medium-Sized Companies Adapt Amazon Web Services With Success?

How do companies successfully adapt Amazon Web Services? Which AWS customer did it how? How can Nordcloud help medium-sized companies in particular...

Blog

Top 5 Cloud Strategy Tips For the Year 2020

In this chapter of ‘The Role of Transformational Partners in Organization Change’, we introduce Nordcloud’s vision for Cloud Strategy and...

Get in Touch

Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.