AWS Lambda Environment Variables and Dependency Injection

Question

Are there any best practices or documentation available for using Dependency Injection or mocking Environment Variables when using AWS Lambda with .NET Core v1.0 ?

As an example, below is an example Lambda function ProcessKinesisMessageById that accepts a KinesisEvent and does some sort of processing. Part of this processing involves accessing some sort of External Service (like AWS S3 or a database) that need access to Environment Variables for setup.

public class AWSLambdaFileProcessingService
{
    private IFileUploadService _fileUploadService;

    // No constructor in the Lambda Function

    [LambdaSerializer(typeof(JsonSerializer))]
    public void ProcessKinesisMessageById(KinesisEvent kinesisEvent, ILambdaContext context)
    {
        Console.WriteLine("Processing Kinesis Request");

        _fileUploadService = new AWSFileUploadService(); // Can this be injected? (Constructor shown below)

        // some sort of processing
        _fileUploadService.DoSomethingWithKinesisEvent(kinesisEvent);
    }
}

// Example of of a class that needs access to environment variables
// Can this class be injected into the AWS Lambda function?  
// Or the Environment Variables mocked?
public class AWSFileUploadService : IFileUploadService
{
    private readonly IAmazonS3 _amazonS3Client;
    private readonly TransferUtility _fileTransferUtility;


    public AWSFileUploadService()
    {
        _amazonS3Client = new AmazonS3Client(
            System.Environment.GetEnvironmentVariable("AWS_S3_KEY"),
            System.Environment.GetEnvironmentVariable("AWS_S3_SECRET_KEY")
            );

        _fileTransferUtility = new TransferUtility(_amazonS3Client);
    }

    public bool DoSomethingWithKinesisEvent(KinesisEvent kinesisEvent)
    {
        // ....
    }

```

The function works okay after publishing it with Environment variables, and it can be tested using the Lambda Function View test console (in Visual Studio 2017) after publishing it to AWS. However, I am having trouble creating unit or integration tests without being able to mock or set the environment variables for use in local testing.

Does anyone have any suggestions or practices for testing the Lambda function locally?

Answer 1

This answer is an attempt to implement the recommendations from @Nkosi's answer.

I am not familiar with how to override a Lazy factory and tried different methods, and the below is my attempt at an implementation method to accomplish this. The new abstraction for the Environment Variables is included below along with a new implementation of the ILambdaContext interface to accept dependencies created by the lazy factory. I post this answer in order to augment the original question and expand beyond a short comment to @Nkosi's very helpful answer.

// code start

This is the AWS Lambda function - refactored to only accept a request and pass into newly created service (where the processing logic lives)

public class AWSLambdaFileProcessingService
{
    [LambdaSerializer(typeof(JsonSerializer))]
    public void ProcessKinesisMessageById(KinesisEvent kinesisEvent, ILambdaContext context)
    {
        Console.WriteLine("Processing Kinesis Request");

        IKinesisEventProcessingService kinesisEventProcessingService = new KinesisEventProcessingService(context);
        kinesisEventProcessingService.ProcessKinesisEvent(kinesisEvent);
    }
}

This is a new service to encapsulate all services that act on input

public class KinesisEventProcessingService : IKinesisEventProcessingService
{
    private IFileUploadService _fileUploadService;

    // constructor to attach Lazy loaded IFileUploadService
    public KinesisEventProcessingService(ILambdaContext context)
    {
        AWSLambdaFileProcessingServiceContext AWSLambdaFileProcessingServiceContext =
            LambdaContextFactory.BuildLambdaContext(context);

        _fileUploadService = AWSLambdaFileProcessingServiceContext.FileUploadService;
    }

    public void ProcessKinesisEvent(KinesisEvent kinesisEvent)
    {

        _fileUploadService.DoSomethingWithKinesisEvent(kinesisEvent);
        // ....

    }
}

This is an implementation of ILambdaContext that can also be used for testing this context allows for override of attached services in testing

public class AWSLambdaFileProcessingServiceContext : ILambdaContext
{
    public AWSLambdaFileProcessingServiceContext()
    {
        FileUploadService = default(IFileUploadService);
    }

    public string AwsRequestId { get; }
    // ... ILambdaContext properties
    public TimeSpan RemainingTime { get; }

    // Dependencies
    public IFileUploadService FileUploadService { get; set; }

}

// static class for attaching dependencies to the context
public static class LambdaContextFactory
{
    public static AWSLambdaFileProcessingServiceContext BuildLambdaContext(ILambdaContext context)
    {
        // cast to implementation that has dependencies as properties of context
        AWSLambdaFileProcessingServiceContext serviceContext = default(AWSLambdaFileProcessingServiceContext);

        if (context.GetType().Equals(typeof(AWSLambdaFileProcessingServiceContext)))
        {
            serviceContext = (AWSLambdaFileProcessingServiceContext)context;
        }
        else
        {
            serviceContext = new AWSLambdaFileProcessingServiceContext();
        }

        // lazily inject dependencies
        if (serviceContext.FileUploadService == null)
        {
            serviceContext.FileUploadService = FileUploadService.Value;
        }

        return serviceContext;
    }

    public static Lazy<IFileUploadService> FileUploadService = new Lazy<IFileUploadService>(() =>
    {
        ISystemEnvironmentService env = new SystemEnvironmentService();
        IAmazonS3 s3 = new AmazonS3Client(
            env.GetEnvironmentVariable("AWS_S3_KEY"),
            env.GetEnvironmentVariable("AWS_S3_SECRET_KEY")
        );
        IFileUploadService service = new AWSFileUploadService(s3);
        return service;
    });

This is an example of a test for the Lambda function

    /// <summary>
    /// This tests asserts that the Lambda function handles the input and calls the mocked service
    /// </summary>
    [Fact()]
    public void TestKinesisMessage()
    {
        // arrange
        string testMessage = "59d6572f028c52057caf13ff";
        string testStream = "testStream";

        IFileUploadService FileUploadService = new AWSFileUploadService(new Mock<IAmazonS3>().Object);
        // create the custom context and attach above mocked FileUploadService from Lazy factory
        var context = new AWSLambdaFileProcessingServiceContext();
        context.FileUploadService = FileUploadService;

        var lambdaFunction = new AWSLambdaFileProcessingService();

        KinesisEvent kinesisEvent = BuildKinesisTestRequest(testMessage, testStream);

        // act & assert
        try
        {
            lambdaFunction.ProcessKinesisMessageById(kinesisEvent, context);
        }
        catch (Exception e)
        {
            // https://stackoverflow.com/questions/14631923/xunit-net-cannot-find-assert-fail-and-assert-pass-or-equivalent
            Assert.True(false, "Error processing Kinesis Message :" + e.StackTrace);
        }
    }

Answer 2

The fact that this is AWS Lambda Function is implementation concern and really shouldn't have much bearing on the fact that the code in its current state is difficult to test in isolation. This is a matter of design issues.

Consider refactoring the code to be a little more flexible/maintainable.

Concerning environment variables, consider encapsulating the static class behind an abstraction to allow for looser coupling and better mocking.

public interface ISystemEnvironment {
    string GetEnvironmentVariable(string variable);
}

public class SystemEnvironmentService : ISystemEnvironment {
    public string GetEnvironmentVariable(string variable) {
        return System.Environment.GetEnvironmentVariable(variable);
    }
}

The AWSFileUploadService is tightly coupling itself to implementation concerns when based on the example provided, abstractions exists that can be taken advantage of.

public class AWSFileUploadService : IFileUploadService {
    private readonly IAmazonS3 _amazonS3Client;
    private readonly TransferUtility _fileTransferUtility;

    public AWSFileUploadService(IAmazonS3 s3) {
        _amazonS3Client = s3;
        //Not sure about this next class but should consider abstracting it as well.
        _fileTransferUtility = new TransferUtility(_amazonS3Client);
    }

    public bool DoSomethingWithKinesisEvent(KinesisEvent kinesisEvent) {
        //code removed for brevity
        return true;
    }
}

With the above two suggestions the AWSLambdaFileProcessingService can now be refactored to

public class AWSLambdaFileProcessingService {
    private IFileUploadService _fileUploadService;

    [LambdaSerializer(typeof(JsonSerializer))]
    public void ProcessKinesisMessageById(KinesisEvent kinesisEvent, ILambdaContext context) {
        Console.WriteLine("Processing Kinesis Request");
        _fileUploadService = FileUploadService.Value;
        // some sort of processing
        _fileUploadService.DoSomethingWithKinesisEvent(kinesisEvent);
    }

    public static Lazy<IFileUploadService> FileUploadService = new Lazy<IFileUploadService>(() => {
        var env = new SystemEnvironmentService();
        var s3 = new AmazonS3Client(
            env.GetEnvironmentVariable("AWS_S3_KEY"),
            env.GetEnvironmentVariable("AWS_S3_SECRET_KEY")
        );
        var service = new AWSFileUploadService(s3);
        return service;
    });
}

The Lazy factory can be replaced as needed when testing as it exposes an abstraction that can be mocked when testing.

The following example uses Moq

[TestMethod]
public void TestKinesisMessage() {
    //Arrange
    var testMessage = "59d6572f028c52057caf13ff";
    var testStream = "testStream";
    var kinesisEvent = BuildKinesisTestRequest(testMessage, testStream);
    var lambdaServiceMock = new Mock<ILambdaContext>();
    var fileUploadServiceMock = new Mock<IFileUploadService>();            
    //Replace the  lazy initialization of the service
    AWSLambdaFileProcessingService.FileUploadService = 
        new Lazy<IFileUploadService>(() => fileUploadServiceMock.Object);
    var subject = new AWSLambdaFileProcessingService();

    //Act
    subject.ProcessKinesisMessageById(kinesisEvent, lambdaServiceMock.Object);

    //Assert
    fileUploadServiceMock.Verify(_ => _.DoSomethingWithKinesisEvent(kinesisEvent), Times.AtLeastOnce());
}

In fact, with this design the system environment abstraction could be removed altogether as it too can be considered an implementation concern based on where and how it is being used.