How to share code in serverless with Python properly?

Question

I'm trying to find the best approach for robust serverless development with Python. The project is divided into multiple services using the serverless framework and versioned in a monorepo. Here's the structure I'd like to have:

• service_1/
  • serverless.yml
  • handler.py
• service_2/
  • serverless.yml
  • handler.py
• shared
  • module_a.py
  • module_b.py

module_a and module_b comprise shared logic, which should be available for both services. So far I've found 2 approaches: wrap shared code in an installable package and inject it to services via pip or provide shared code as a layer. Both solutions have flaws, most importantly it's impossible to rapidly develop the app because any change requires pip. I've noticed this problem is solved in Node.js and there are many unanswered questions about Python.

There is the plugin - serverless-package-common, which seems to tackle this issue, however, it doesn't look like a go-first approach.

I appreciate any form of help.

Answer 1

Maybe you can consider to use Lambda layer. You can find a quick guide here: link.

Let me give you a simple example. This example has two serverless projects - library which has shared code and service which has service codes.

1. library - This project should have below file structure. You have serverless.yml in the root of this project and another folder which will be export as a module. python will be linked and let lambda find your module when you use lambda with python.

./
└ serverless.yml
└ common/
    └ python/
        └ Common.py

serverless.yml - As you can see, folder common is explicitly declared here to be exported. Beware that layer name Common is re-used in resources and serverless framework will automatically match this with resource reference.

service: library
  provider:
    name: aws
    runtime: python3.7
    stage: dev
    region: ap-northeast-1
  layers:
    Common:
      path: common
  resources:
    Outputs:
      CommonLayerExport:
        Value:
          Ref: CommonLambdaLayer
        Export:
          Name: CommonLambdaLayer

common/python/Common.py (printException is an example of shared function which will be used in another project)

import sys
import traceback
import os

def printException(exception, writer=None):
    if writer is None:
      writer = print
    top = traceback.extract_tb(sys.exc_info()[2])[-1]
    writer(", ".join([type(exception).__name__, str(sys.exc_info()[1]), os.path.basename(top[0]), str(top[1])]))

2. service serverless.yml - As you can see, layers should be included per each functions and you can refer to a specific layer with CloudFormation reference.

service: layer-test
  provider:
    name: aws
    region: ap-northeast-1
    runtime: python3.7
    stage: dev
  functions:
    exceptionExample:
      handler: handler.func
      layers:
        - ${cf:library-dev.CommonLayerExport}
      events:
        - http:
            path: exceptionExample
            method: get

handler.py - Now we can easily import shared module from shared layer.

import json
import Common

def func(event, context):
    try:
        1/0 # just to raise an exception and test the shared function
    except Exception as e:
        Common.printException(e)

    response = {
        "statusCode": 200,
    }
    return response

One thing you should be careful is that as lambda layer does not pack imported modules, you should import modules which is used by your layer in your services as well.