Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to run Parallel builds with AWS Codebuild?

I am trying to setup a codebuild where I have one module that would create the dependencies for other builds. And these other builds will use the artifacts generated by the first build and these build should be ran in parallel. I was looking at documentation of CodeBuild it seems there isn't information there. Sample of my buildspec.yml

version: 0.2

#env:
  #variables:
     # key: "value"
     # key: "value"
  #parameter-store:
     # key: "value"
     # key: "value"
  #git-credential-helper: yes

phases:
  #install:
    #If you use the Ubuntu standard image 2.0 or later, you must specify runtime-versions.
    #If you specify runtime-versions and use an image other than Ubuntu standard image 2.0, the build fails.
    #runtime-versions:
      # name: version
      # name: version
    #commands:
      # - command
      # - command
#  pre_build:
#    commands:
#       - mkdir artifacts
#       - pwd
#       - ls
  build:
    commands:
       - cd frontend
       - npm install
       - cd ..
       - cd othermodule
       - npm install
  #post_build:
    #commands:
     #  - cp -a $CODEBUILD_SRC_DIR/frontend/dist. 
      # - command
artifacts:
  files:
     - package-lock.json
     - node_modules/*
  base-directory: frontend
#cache:
  #paths:
    # - paths
like image 676
KyelJmD Avatar asked Sep 03 '25 01:09

KyelJmD


2 Answers

CodeBuild is used to automate build step, not to automate the whole CICD process. In CodeBuild, you specify buildspec.yml to automate a sequence of steps that need to be performed in that particular build.

If you need to automate sequence of builds then the easiest option that you have is to use CodePipeline where you can create stage for each step in your CICD process. In your case, this would mean that one step (stage) would be CodeBuild action that you have described in your post which would transition into another stage where you can specify another CodeBuild actions and these actions can be specified to take artifacts from the previous step as an input and you can run them in parallel.

So it would look like this

INPUT -> STAGE (perform initial build) -> STAGE (specify multiple build actions in parallel - in console, this can be done by placing them next to each other horizontally instead of vertically)

The other option, without using CodePipeline, would be to use Lambda function with CloudWatch events. CodeBuild publishes event once the build is done. You can subscribe Lambda function to that event and write a code that would execute following builds as needed.

like image 192
Matus Dubrava Avatar answered Sep 04 '25 15:09

Matus Dubrava


From today codebuild support parallel test and splitting of test

https://aws.amazon.com/about-aws/whats-new/2025/01/aws-codebuild-test-splitting-parallelism/

like image 41
Jatin Mehrotra Avatar answered Sep 04 '25 16:09

Jatin Mehrotra