When to use terraform vs serverless framework to deploy AWS lambdas and surrounding resources? [closed]

Question

When you use serverless to deploy AWS lambdas, you can create AWS resources. However, now I am starting to use Terraform to develop resources, and I am not sure what resources should be defined via the Terraform files vs. serverless.

Answer 1

First, you could read terraform/serverless comparison in https://serverless.com/learn/comparisons/ page.

And, actually you can pick either one of them or use both of them together because they are technically not mutually exclusive, one focus on one thing, the other focus on solving the other thing in the similar problem space but not the same problem etc. Choosing which one depends on a lot of factors really.

Basic thinking may goes like this:

When you want to focus on serverless application related resources, you might think of using serverless framework (serverless.yml)

But, if you want to focus on defining Full-fledged infrastructure or more traditional style cloud infrastructure (i.e. defining networking, servers, storing, load balancer etc. by yourself), you might think of using Terraform.

The best way is to experiment, just pick either one at a time to experiment. Then, you will see what fits to your specific task and what makes your self easier.

Answer 2

Might be a bit late to answer but I recently had some experience that might be useful to share about this topic.

I had used Serverless Framework to develop applications for a customer. The customer already has a middleware team with their own way to manage the infrastructure-related AWS resources. I had a similar question: what are the AWS resources should I put in my serverless project?

For the first attempt, I used Serverless Framework to create VPC and networking stuff along with the lambda functions. The problem with this solution is that my project has a lot of conflicts with the way the middleware team works. They have a central document to keep track of subnets and IP addresses. They have a single Internet, NAT, and VPN gateway for the whole organization, to which other VPCs must connect via a transit gateway. They use a third party software to manage firewall rules. And etc. I had to rework many times to modify my code to be aligned with their networking prerequisites.

For the second attempt, I put only Lambda functions and a few stuff in my serverless project, and have other stuff such DynamoDB tables, SNS, SQS, S3, IAM role, Security Group, VPC, etc. pre-provisioned by the existing IT request process of the middleware team, and put the reference IDs in my configuration. A new problem occurred with this solution: as the application grows, new AWS resources, such as DynamoDB tables, are required all the time and the provisioning plan should be aligned with the development cycle. The IT request process usually is out of sync with the development cycle. The project was delayed due to the pending provisioning of the required AWS resources. Yet to mentioned that the resource specification usually changes as we discover more during development. Issue another IT request to change the specification makes the process incredibly terrible.

From this experience, I have learned the importance of separating the "application-concerned resources" from "infrastructure-concerned resources". Mixing one into another causes the problems as mentioned above.

What works for me is to not put networking concerns into a serverless project. Instead, have such resources pre-provisioned by other means such as using Terraform in a separated project or by manual provisioning, and put their reference IDs in the configuration of your serverless project.

And for what resources should be in a serverless project, the minimal list for me in general are Lambda functions, DynamoDB tables, SNS topics, SQS queues, and CloudWatch Events (EventBridge).

I start with these dummies in general and then start analyzing the requirements to see if my application will need some other kind of resources during its growth. Add such resources to the application-concerned list and discuss with the related party such as middleware, architect, and security team to determine whether all the resources in the list can be controlled in a Serverless Framework project.

Answer 3

I think it's a difficult question to answer, as it depends on many factors, like your company's internal structure.

As a rule of thumb, I'd say that every resource that is only used by your serverless service should be defined in your serverless.yml file, and the shared resources should be defined using a terraform (or another technology) project. I used this approximation in the past and it works fine.

Yan Cui has a nice article talking about sharing code and infrastructure (https://hackernoon.com/aws-lambda-how-best-to-manage-shared-code-and-shared-infrastructure-827bed395eb7).