Create 2 Pipelines for a Node Project in GitLab

Question

I'm trying to run 2 pipelines for a project in GitLab, but I can't find any way to do it.

Answer 1

In gitlab CI you can't create multiple pipelines for one project explicitly. There are cases where multiple pipelines will run simultaneously, such as when you have jobs that run only for merge requests and other jobs that do not run on merge requests.

That said, there are ways to obtain the effect of running multiple series of jobs independently from one another.

The hacky way, before gitlab-ce 12.2

If you want to start 2 pipelines for the same project you can use pipeline triggers. This method is limited to 2 pipelines and gitlab CI is not meant to be used this way. Usually, triggers are used to start pipelines on another project.

All in your .gitlab-ci.yml:

stages:
  - start
  - build

###################################################
#                First pipeline                   #
###################################################
start_other_pipeline:
  stage: start
  script:
  # Trigger a pipeline for current project on current branch
  - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
  except:
  - pipelines

build_first_pipeline:
  stage: build
  script:
  - echo "Building first pipeline"
  except:
  - pipelines


###################################################
#                Second pipeline                  #
###################################################

# Will run independently of the first pipeline.

build_second_pipeline:
  stage: build
  script:
  - echo "Building second pipeline"
  only:
  - pipelines

To clean up this mess of a .gitlab-ci.yml, you can use the include keyword:

# .gitlab-ci.yml

include: 
  - '/first-pipeline.yml'
  - '/second-pipeline.yml'

stages:
  - start
  - build

# This starts the second pipeline. The first pipeline is already running.
start_other_pipeline:
  stage: start
  script:
  # Trigger a pipeline for current project on current branch
  - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
  except:
  - pipelines

# first-pipeline.yml

build_first_pipeline:
  stage: build
  script:
  - echo "Building first pipeline"
  except:
  - pipelines

# second-pipeline.yml

build_second_pipeline:
  stage: build
  script:
  - echo "Building second pipeline"
  only:
  - pipelines

The reason this works is the use only and except in the jobs. The jobs marked with

  except:
  - pipelines

do not run when the pipeline has started because of a trigger coming from another pipeline, so they don't run in the second pipeline. On the other hand,

  only:
  - pipelines

does the exact opposite, therefore those jobs run only when the pipeline is triggered by another pipeline, so they only run in the second pipeline.

The probably right way, depending on your needs ;)

In gitlab CE 12.2, it is possible to define Directed Acyclic Graphs to specify the order that your jobs run. This way, a job can start as soon as the job it depends on (using needs) finishes.

Answer 2

As of GitLab 12.7 it is also possible to use parent-child pipelines for this:

# .gitlab-ci.yml
trigger_child:
  trigger:
    include: child.yml

do_some_stuff:
  script: echo "doing some stuff"

# child.yml
do_some_more_stuff:
  script: echo "doing even more stuff"

The trigger_child job completes successfully once the child pipeline has been created.