Lambda Layers with AWS CDK in Python

In the world of serverless computing, the use of AWS Lambda has quickly become a fundamental part of building scalable and efficient applications. One of the challenges in maintaining these applications is the maintenance of dependencies in multiple functions. For these scenarios we can use Lambda layers, which are nothing more than a zip file containing code or supplementary data, and can contain package dependencies or configuration files.

Some of the advantages of using Lambda layers are:

Sharing dependencies between multiple functions. Maintains consistency and reduces redundancy.
Reduce the size of the deployed package. It allows deployed packages to be small and organized, keeping the code of the functions lean. This is also a benefit in deployment time and prevents reaching the Lambda size limit.

Implementing Lambda layers in the IaC workflow is fundamental to reduce deployment and maintenance times of the entire application. In this article, we dive into how to create and deploy Lambda layers using AWS Cloud Development Kit (CDK) and Python.

AWS CDK is an open source software development framework to model and define cloud infrastructure by using a supported programming language. Some of its benefits are developing and maintaining infrastructure with code (IaC), defining it with general purpose programming languages, developing faster with low-level constructs.

We explore from the setup of the CDK project, through the layer structure, and how to integrate it with the CDK stack. Specifically, we will see how to generate a Lambda layer with Python packages that we will install with pip from PyPI, as well as how to generate a layer with object-relational mapping (ORM) for database interaction.

Starting with CDK

The AWS team has a great guide to getting started with CDK, however, in this article we will look at the general steps to get started with the project.

The prerequisites to start working with CDK are:

npm
Python

For this guide, npm 10.8.2 and Python 3.12.6 were used.

Start by installing and configuring CDK.

# Install CDK
npm install -g aws-cdk

# Verify installation
cdk --version

Configure security credentials for the AWS CDK CLI.

To create your first AWS CDK app, follow these steps:

# Create and navigate to a new directory for your application
mkdir infrastructure && cd infrastructure

# Initialize a new CDK project using cdk init
cdk init app --language python

After initializing your app, create and activate a virtual environment and install the AWS CDK core dependencies.

source .venv/bin/activate
python -m pip install -r requirements.txt

Project structure and layers definition

We will create two layers:

External libraries and Lambdas dependencies.
ORM objects for the definition of a database schema.

For these layers we will create a directory where we can place these definitions. The final structure of the project will be as follows. Consider it during the entire course of this guide.

├── README.md
├── app.py
├── infrastructure
│   ├── utils
│   │   ├── __init__.py
│   │   └── layers.py
│   ├── __init__.py
│   ├── constants.py
│   └── infrastructure_stack.py
├── cdk.json
├── ...
└── service
    ├── __init__.py
    ├── handlers
    │   └── hello_world
    │       └── index.py
    └── layers
        ├── common
        │   └── requirements.txt
        └── models
            ├── __init__.py
            └── orm_objects.py

In common/requirements.txt, we will include the packages we want to install with pip so that the Lambdas functions have all necessary dependencies available.
In this example we will use the following file, where we add the packages aws-lambda-powertools and sqlmodel.

# common/requirements.txt
aws-lambda-powertools==3.6.0
sqlmodel==0.0.22

In models/orm_objects.py, we will have the definition of our relational database model. As an example, we will define an Enum and two SQLModel classes.

# service/layers/models/orm_objects.py
import uuid
from enum import Enum
from typing import Optional

from sqlmodel import Field, SQLModel


class ModelCategory(int, Enum):
    BASIC = 0
    SLEEVELESS = 1


class Model(SQLModel, table=True):
    id: uuid.UUID = Field(default_factory=uuid.uuid4, primary_key=True)
    name: str
    model: ModelCategory = Field(default=ModelCategory.BASIC)
    description: Optional[str] = None


class Product(SQLModel, table=True):
    id: uuid.UUID = Field(default_factory=uuid.uuid4, primary_key=True)
    model_id: uuid.UUID = Field(default=None, foreign_key="model.id")
    size: str = Field(default=None)

Packaging the Layers

This section outlines the key considerations to ensure that our layers can provide the dependencies required for the project. Here, we will examine the files located in the infrastructure directory, which was introduced earlier. It is worth noting that a Lambda layer is essentially a zip file containing the data necessary for its operation.

While there are articles that suggest creating layers solely through code, employing a packaging process with Docker, in my experience, this approach is not always reliable or consistent. For this reason, the following solution is the one I have found to be more effective and robust in practice.

Constants definition

First, we will create a constants file that we will use in our infrastructure stack. In it, we will indicate the paths of the build contents.

# infrastructure/constants.py
COMMON_LAYER_BUILD_DIR = ".build/common_layer"
COMMON_LAYER_REQUIREMENTS = "service/layers/common/requirements.txt"
COMMON_LAYER_ID = "CommonLayer"
ORM_LAYER_BUILD_DIR = ".build/orm"
ORM_LAYER_OBJECTS_DIR = "service/layers/models"
ORM_LAYER_ID = "OrmLayer"
ASSET_HELLO_LAMBDA = "service/handlers/hello_world/"

A basic Lambda function

We will create a function called hello_world. In its handler, we will not include major functionality, but it will allow us to demonstrate that the dependencies are integrated correctly.

# service/handlers/hello_world/index.py
from aws_lambda_powertools.utilities.typing.lambda_context import LambdaContext
from models.orm_objects import ModelCategory
from sqlmodel import SQLModel


def handler(event: dict, context: LambdaContext) -> dict:
    print("Hello World")
    return {"hello": "world"}

Utility functions to build layers

In our project file structure it can be seen that we will need a utility file. In it we will include specific functions to package the layers.

What is happening in these functions?

build_lambda_layer:
Used for the common layer.

We define the path in which we will include our zip. It is necessary to include all packages inside a python directory because AWS Lambda expects packages to follow a specific directory structure in order to properly integrate them into the runtime environment.
If the output directory and file for this function exist, then delete them and create the output directory from scratch. This step is optional if you are working locally, but if it is omitted, the packages will not be updated.
Install the packages in the output directory specified in the first step. For this, we use pip, specifying the path to the requirements file and the path to the output directory.

build_orm_lambda_layer:
Used for the models layer.

Same steps as above.
We copy the files containing the class definition of our model to the output directory, ensuring that they stay inside the python directory.

# infrastructure/utils/layers.py
import shutil
import subprocess
from pathlib import Path


def build_lambda_layer(build_directory: str, requirements: str) -> None:
    layer_directory = Path(build_directory) / "python"
    # Clean existing build files
    if layer_directory.exists():
        shutil.rmtree(layer_directory.parent)
    # Create layer build directory
    layer_directory.mkdir(parents=True)
    # Install dependencies
    subprocess.run(
        f"pip install -r {requirements} -t {layer_directory} --quiet".split(),
        check=True,
    )


def build_orm_lambda_layer(build_directory: str, models_directory: str) -> None:
    layer_directory = Path(build_directory) / "python"
    orm_objects_source = Path(models_directory)
    # Clean existing build files
    if layer_directory.exists():
        shutil.rmtree(layer_directory.parent)
    # Create layer build directory
    layer_directory.mkdir(parents=True)
    # Copy dependencies
    if orm_objects_source.exists():
        shutil.copytree(
            orm_objects_source, layer_directory / "models", dirs_exist_ok=True
        )

Using the layers in the CDK Stack

Now we need to use these functions in our CDK stack. In the infrastructure/infrastructure_stack.py file, which is where we define the resources, we will include both layers and the Lambda function.

In this Stack:

We define both layers with lambda.LayerVersion.
Before instantiating each layer, we call our functions build_lambda_layer and build_orm_lambda_layer to generate our package structure.
With the assets located in the .build directory (as indicated in our constants file), we call lambda.Code.from_asset to define the code for the layers.
We also define our Lambda function, with lambda.Function. We use one of the runtimes defined as compatible with the layer (PYTHON_3_12) and indicate that we have an index file with a handler function, which will be the entry point of our Lambda function. In addition, we indicate the layers just defined.

With these definitions, we can deploy the code and use our layers.

Note on native dependencies:
Some libraries, such as pydantic (used by sqlmodel), include compiled extensions that must match the Lambda environment (Amazon Linux 2).
If you encounter errors such as “No module named ‘pydantic_core._pydantic_core'”, consider using Docker with the amazon/aws-lambda-python image to package the layers and ensure compatibility.

# infrastructure/infrastructure_stack.py
from aws_cdk import (
    Stack,
    RemovalPolicy,
)
from aws_cdk import aws_lambda as _lambda
from constructs import Construct

import infrastructure.constants as constants
from infrastructure.utils.layer import build_lambda_layer, build_orm_lambda_layer


class InfrastructureStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)
        # Create layers
        common_layer = self._build_common_layer()
        orm_layer = self._build_orm_layer()
        self._build_hello_lambda(
            common_layer, orm_layer
        )


    def _build_common_layer(self) -> _lambda.LayerVersion:
        """
        Install packages and create layer from asset
        """
        build_lambda_layer(
            constants.COMMON_LAYER_BUILD_DIR,
            constants.COMMON_LAYER_REQUIREMENTS,
        )
        return _lambda.LayerVersion(
            scope=self,
            id=constants.COMMON_LAYER_ID,
            code=_lambda.Code.from_asset(constants.COMMON_LAYER_BUILD_DIR),
            compatible_runtimes=[_lambda.Runtime.PYTHON_3_12],
            removal_policy=RemovalPolicy.DESTROY,
            description="Layer with AWS SDK, boto3, and PostgreSQL package",
        )

    def _build_orm_layer(self) -> _lambda.LayerVersion:
        """
        Copy files and create layer from asset
        """
        build_orm_lambda_layer(
            constants.ORM_LAYER_BUILD_DIR,
            constants.ORM_LAYER_OBJECTS_DIR,
        )
        return _lambda.LayerVersion(
            scope=self,
            id=constants.ORM_LAYER_ID,
            code=_lambda.Code.from_asset(constants.ORM_LAYER_BUILD_DIR),
            compatible_runtimes=[_lambda.Runtime.PYTHON_3_12],
            removal_policy=RemovalPolicy.DESTROY,
            description="Layer with ORM models",
        )

    def _build_hello_lambda(
        self,
        common_layer: _lambda.LayerVersion,
        orm_layer: _lambda.LayerVersion,
    ):
        """
        Create Lambdas with layers
        """
        lambda_function = _lambda.Function(
            scope=self,
            id="Connection",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="index.handler",
            code=_lambda.Code.from_asset(constants.ASSET_HELLO_LAMBDA),
            layers=[common_layer, orm_layer],
        )
        return lambda_function

Deploying and testing the layers

In the first instance, we must prepare the AWS environment for deployment by bootstrapping it using the following CDK CLI command in the root of the project.

cdk bootstrap

We now synthesize a CloudFormation template. The synthesis command runs a basic validation of the CDK code and generates the stack template.
This step is optional, as it runs automatically when deploying. However, AWS recommends running this step before each deployment to check for possible synthesis errors.

cdk synth

After solving the problems that appear in the synthesis, we will continue with the deployment of the project. The command generates a CloudFormation template and deploys the stack resources. The command to execute at the root of the project is:

cdk deploy

When the deployment is over, the CLI will deliver the ARN of the stack. In addition, we will be able to confirm the deployment in the AWS console, in the CloudFormation service.

Our Lambda function will look like in the figure below. This indicates that there are two layers in use, corresponding to the CommonLayer and OrmLayer, as defined in our stack with the constants used in the project.

We can see that these are the layers of our function at the bottom of the function page. It should look as shown in the following figure.

Next, we show the result of the execution of our function. In this image we can see that there was no import failure, this means that our layers worked correctly.

Final thoughts

The use of Lambda Layers in AWS CDK with Python is a key strategy for optimizing dependency management in serverless applications. As we have seen in this article, leveraging layers to share libraries and ORM objects allows you to reduce redundancies, better organize your code and improve deployment times.

From the configuration of the CDK environment to the creation and deployment of layers, we have explored how this methodology not only optimizes the use of resources, but also simplifies development and long-term maintenance.

Ultimately, implementing Lambda Layers effectively in AWS CDK is a fundamental practice for developing more efficient, scalable, and easier-to-manage serverless solutions. By applying these strategies, teams can build cleaner, more modular architectures that are ready to grow with business needs.

Subscribe

Lambda Layers with AWS CDK in Python

Starting with CDK

The prerequisites to start working with CDK are:

Project structure and layers definition

Packaging the Layers

Constants definition

A basic Lambda function

Utility functions to build layers

What is happening in these functions?

Using the layers in the CDK Stack

Deploying and testing the layers

Final thoughts

Read Next

Lambda Layers with AWS CDK in Python

Designing the New Visual Identity of NeuralWorks

Exploring Data Science in Graphs and Graph Neural Networks (GNNs)