How to have a Gitlab CI Pipeline run some jobs always and other jobs only on merge-requests?

Question

TL/DR: My goal is to have a Gitlab (CE-12.4.2) pipeline that executes some jobs only on merge-requests and other jobs always (on merge-requests and on all normal pushes). How must the .gitlab-ci.yml look to do this?

My use-case: I have a big pipeline running a lot of jobs (tests, validation, dep's, build, doc's, ...). Now I have added a staging environment (using kubernetes) and have the pipeline build a new image and deploy that in the staging-environment. This allows me to instantly open up the changed (web-)application and see how the changes behave and look without having to check them out locally. Now building an image and deploying it to staging would be much too resource-heavy to do for every push, so I only want deployments to staging when someone creates a merge-request for me to review.

A very simplified example:

install:
  script: ...

test:
  script: ...

build-image:
  script: ...
  only: [merge_requests]

deploy-staging:
  script: ...
  only: [merge_requests]

For all normal pushes, the jobs install and test should be executed.

For merge-requests, the jobs install, test, build-image and deploy-staging should be executed.

What i have tried: Gitlab has this feature to define only: [merge_requests] on a job, this causes that job to only be executed when the pipeline is executed for a merge-request. Sounds like exactly what I am looking for, but there is a big catch. Once that attribute is applied to one job in a pipeline, all other jobs in that pipeline that do not have that attribute will be removed from the pipeline when executed inside merge-requests. That seemed like a bug to me at first, but is actually documented behaviour:

In the above example, the pipeline contains only a test job. Since the build and deploy jobs don’t have the only: [merge_requests] parameter, they will not run in the merge request.

In order to re-introduce all the other jobs to the pipeline for merge-requests, I have to apply only: [merge_requests] to all other jobs. The problem with that approach is that now these regular jobs do not get executed for normal git-pushes anymore. And I have no way to re-introduce these regular jobs to pipelines for normal pushes, because Gitlab has no support for only: [always] or anything like that.

Now I also have noticed that the only syntax is candidate for deprecation and one should prefer the rules syntax instead, so I took a look at that. There are multiple problems with that approach:

• The only way to detect with rules whether the pipeline is executed for a merge-request or not is to evaluate the variables related to merge-requests, like $CI_MERGE_REQUEST_ID. Unfortunately these variables only exist when only: [merge_requests] is used, which would re-introduce the above problems.
• Rules only allow the conditional application of other attributes, so I still would have to use only, except or when attributes to actually remove or add jobs from or to the pipeline. Unfortunately Gitlab does not support anything like only: [never] or when: never, so i have no way to actually remove or add the jobs.

I also tried to have the jobs depend on another using need or dependencies attributes, this seemed to have no effect on whether the job is included to the pipeline or not.

The last thing I desperately tried was including all jobs always and just mark them as when: manual to be triggered manually by pushing a button. This somewhat works, but is very tedious because the deployment to staging is a multi-job process with every job taking quite some time to finish. So I would see a merge-request, push the button for the first job, wait 5 minutes, press the next button, wait 5 minutes again, and only then am able to use the staging. For many small merge-requests, this would take up a lot of my time and would not be a efficient solution. I also cannot just mark the first of these jobs as manual because Gitlab will then just skip that job and execute that later ones out of order (And again, needs and dependencies seem to have no effect on this when dealing with manually triggered jobs).

What i am a bit baffled about is that after searching the net, I have found nobody having the same problem. Either I am the only Gitlab User that wants to execute some jobs only for merge-requests without excluding all other jobs (which seems highly unlikely) or I am missing something obvious (which seems more likely). Am I missing something or does Gitlab really not support this use-case?

Answer 1

From the previous answer:

But there is a problem. This example launches two pipelines for every push after MR is created. One for only: [branch], one for only: [merge_requests]

Check if the last GitLab 13.8 (January 2021) can help:

Use both branch and MR pipelines without duplication

Previously it was not possible to run pipelines for branches first, then switch to merge requests pipelines when the MR is created.

Consequently, with some configurations, a push to a branch could result in duplicate pipelines if a merge request was already open on the branch: one pipeline on the branch and another for the merge request.

Now you can use the new $CI_OPEN_MERGE_REQUESTS predefined environment variable in your CI configurations to switch from branch pipelines to MR pipelines at the right time, and prevent redundant pipelines.

See Documentation and Issue.

user2993689 points out in the comments to the documentation for a similar example:

Avoid duplicate pipeline