Resolving cyclical dependencies between AWS CDK CloudFormation stacks

Question

Context, I have a CDK app with two stacks using the following setup:

Stack_A:
    StateMachine_A
    Lambda_A
    S3Bucket_A
    IAMRole_A

Stack_B:
    StateMachine_B
    SageMakerTrainJob_B
    IAMRole_B

StateMachine_A runs Lambda_A using execution role IAMRole_A. A separate step in StateMachine_A writes data to S3Bucket_A. StateMachine_B runs SageMakerTrainJob_B using execution role IAMRole_B. Lambda_A's purpose is to start execution of StateMachine_B, whose SageMakerTrainJob_B needs to read from S3Bucket_A. Therefore, we have to configure the following permissions:

• IAMRole_A needs startExecution permissions on StateMachine_B.
• IAMRole_B needs read permissions on S3Bucket_A.

We tried to model this in CDK by creating a direct dependency in Stack_B on Stack_A, using references to IAMRole_A and S3Bucket_A within Stack_B's definition to grant the needed permissions in code. However, this generated the following error:

Error: 'Stack_B' depends on 'Stack_A' (dependency added using stack.addDependency()). Adding this dependency (Stack_A -> Stack_B/IAMRole_B/Resource.Arn) would create a cyclic reference.

Likewise, trying to model the dependency in the other direction gave the same error:

Error: 'Stack_A' depends on 'Stack_B' (dependency added using stack.addDependency()). Adding this dependency (Stack_B -> Stack_A/S3Bucket_A/Resource.Arn) would create a cyclic reference.

Is there any way around this using code dependencies? Are there any recommended best practices for situations like this? Some options we've considered include:

• Using a third stack that depends on both to give Stack_A and Stack_B access to each other's resources.
• Creating additional access roles for the necessary resources within each stack and maintaining assumeRole permissions for the Lambda/SageMaker roles somewhere outside of CDK.
• Putting them all in one stack. Not great for organization and makes the resources really tightly coupled--we may not want StateMachine_A to be the only entry point to StateMachine_B in the future.

Also , I see there were similar-sounding issues during CDK development with CodeCommit/CodePipeline and APIGateway/Lambda. Is this a related bug, or are we just trying to do something that's not supported?

Answer 1

Circular references are always tricky. This isn't a problem that is unique to the CDK. As you explain the problem logically you see where things start to break down. CloudFormation must create any resources that another resource depends on before it can create the dependent resource. There isn't a one solution fits all approach to this, but I'll give some ideas that work.

1. Promote shared resources to another stack. In your case the S3 bucket needs to be used by both stacks, so if that is in a stack that runs before both you can create the S3 bucket, use an export/import in stack B to reference the S3 bucket, and use and export/import in stack A to reference both the S3 bucket and the state machine in B.
2. Use wildcards in permissions. Often you can know the name, or enough of the name, of a resource to use wildcards in your permissions. You want to keep the permissions tightly scoped, but quite often a partial name match is good enough. Use this option with caution, of course. Also keep in mind that many resources can be named by you. Many prefer not to do this ever, and some resources you shouldn't (like and S3 bucket), but I often find it's easier to name things.
3. Create a custom resource to tie things together. If you have a true circular dependency that cannot be resolved (even in the same stack) you may need to use a custom resource to do the work for you. A prime example of this is S3 bucket events.

Answer 2

In addition to the guidelines by Jason, I'd like to mention a fourth option:

You can also try to decouple the two stacks by moving more towards an event driven architecture. While you can't resolve the dependency of IAMRole_B needs read permissions on S3Bucket_A in this example, you can resolve the dependency IAMRole_A needs startExecution permissions on StateMachine_B. Stack A could introduce an EventBridge where StateMachine_A raises an event as soon as it is finished. StateMachine_B can subscribe to this event and start as soon as it has been risen. The stacks would look like the following:

Stack_A:
    StateMachine_A
    Event_A
    S3Bucket_A
    IAMRole_A

Stack_B:
    StateMachine_B
    SageMakerTrainJob_B
    IAMRole_B

You would still have two dependencies:

• IAMRole_B needs read permissions on Event_A.
• IAMRole_B needs read permissions on S3Bucket_A.