How to make my project deploy automatically from dev to staging and manually to production in Jenkins using only a master git branch

Question

I have around 30 Wordpress websites so the way Jenkins is configured is I have a Job for every website. The process of development is as follows (I don't know if it's optimal but this is how we have it):

1. Since we have outsourcing developers, they have their own repository hosted in their own repository hosting provider. Whenever the code is ready for QA, they commit all their changes to our repository in the master branch.
2. I then run the Jenkins job manually with the jenkinsfile like the one below.
3. The workflow must be: deploy to dev environment, if and only if the previous deployment is successful, deploy to stage. Here we have to stop and let the QA people review the website for any broken links, errors, etc.
4. If everything looks like we expected and no errors have been found, then deploy finally to production environment.

NOTE: Some people have suggested me to have only staging and production. The reason why we don't have that configuration is because the dev environment is not accessible online, and the reason for that is because I use this environment to test back-end configuration (e.g. apache conf, etc.).

Also, some other people have suggested to have a branch for each environment, which in theory makes sense but I think it will change the way that our outsourcing devs are committing code to the repository, I mean, they will always have to commit the code to the dev branch and then merge to the stage branch to get deployed to stage, which I don't think is pretty good.

Now, the steps 2-4 look like the following: In order to give you an example on how that process looks we are going to have an example website and job called "Bearitos":

Inside of that Job called "Bearitos" there is a project called "Bearitos to any"

which basically means inside of that project I have a pipeline configured with three stages: dev, staging and prod which are parameterized with the following parameters: DEPLOY_TO: Dev/staging/prod and DEPLOY_DB: Yes/No . So depending of what the user chooses, Jenkins will deploy to that specific environment which I don't think it's even necessary to have those options since the correct deployment flow should be dev -> staging -> prod, there shouldn't be a scenario where dev or staging would be skipped and then deploy right next to production, so in my opinion this should be updated better

Inside of the Jenkinsfile I have defined the three stages Dev, Staging or Prod and also the options if it was chosen to build a DB or not, following is the example of how my Jenkinsfile looks like:

// Deployment template for CMS-based websites (Drupal or Wordpress)
// 
//
pipeline {
    agent any

    parameters {
        choice choices: ['Dev', 'Staging', 'Production'], description: "Choose which environment to push changes to.", name: "DEPLOY_TO"
        booleanParam defaultValue: true, "Choose whether to deploy the database.", name: "DEPLOY_DB"
    }

    environment {
         SITEID = "lb"
        NOFLAGS = "0"
        DBNAME = "wpress_myproject"
        DBSERVER = "dbserver"
        DBUSER = "WordpressUser"
        DBPASS = "hiddenpassword"
        EXCLUDE = "domain_commentmeta,domain_comments"  // separate multiple tables with commas
        DEPLOY_TO = "${params.DEPLOY_TO}"
        DEPLOY_DB = "${params.DEPLOY_DB}"
    }

    stages {
        stage("deploy-db-dev") {
            when {
                allOf { 
                    environment ignoreCase: true, name: "DEPLOY_TO", value: "dev"; 
                    environment ignoreCase: true, name: "DEPLOY_DB", value: "true"; 
                }
            }
            steps {
                // this stage only required until we make our dev the master DB
                // copy full dev database from bolwebdev1
                // import latest database dump to dev server
                script {
                    FILENM = sh(script: 'ls -t myproject-s-dump* | head -1', returnStdout: true)
                }
                //Fixing the problem with the collation existing in the sql dump file, refer to: https://stackoverflow.com/questions/42385099/1273-unknown-collation-utf8mb4-unicode-520-ci 
                //apparently, this is due to a version of mysql issue. Once the problem is fixed from the server side we can then remove the following lines. 

                sh """sed -i s/utf8mb4_unicode_520_ci/utf8mb4_unicode_ci/g ${FILENM}
                # The following line was added because the site is pointing to a staging server which we don't have control over, again, once this is fixed we can delete the following line of code. 
                sed -i s/myproject.staging.websites.3pth.com/myproject.example.net/g ${FILENM}
                mysql -h devserver2 -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_dev < ${WORKSPACE}/${FILENM}
                rm -f ${WORKSPACE}/${FILENM}"""
        }
        }
        stage("deploy-dev") {
            when {
                environment ignoreCase: true, name: "DEPLOY_TO", value: "dev"
            }
            steps {
                // copy files to devserver2
                // NOTE: if we move the repo to SVN, we should change httpdocs/ to ${env.SITEID}docs/
                sh """sudo chown jenkins:jenkins *

                #Replace the wp-config.php file with our domain file with our information. 
        /bin/cp httpdocs/wp-config-domain.php httpdocs/wp-config.php

                # prepare the dev server to receive files by changing the owner
                ssh webadmin@devserver2 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/'
                # copy files from control server to dev
                rsync --exclude=Jenkinsfile -rav -e ssh --delete ${WORKSPACE}/httpdocs/ webadmin@devserver2:/var/opt/httpd/${env.SITEID}docs/
                # fix the owner/permissions on the dev server
        ssh webadmin@devserver2 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/ && sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/ && sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'"""
            }
        }
        stage("deploy-db-staging") {
            when {
                allOf { 
                    environment ignoreCase: true, name: "DEPLOY_TO", value: "staging"; 
                    environment ignoreCase: true, name: "DEPLOY_DB", value: "true"; 
                }
            }
            steps {
                script {
                    def myexcludes = env.EXCLUDE.split(',').toList()
                    MYFLAGS = "-Q -K -c -e --default-character-set=utf8 "
                    if (env.NOFLAGS == "0") {
                        myexcludes.each {
                            MYFLAGS = "${MYFLAGS} --ignore-table=${env.DBNAME}_dev.${it}"
                        }
                    }
                }
                // pull a backup of the current dev database (may exclude some tables)
                sh """mysqldump -h devserver2 -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_dev ${MYFLAGS} > ${env.DBNAME}_dev.sql
        #Searching and replace for the URL to change from the dev sever to the staging server
                sed -i s/myproject.example.net/stage-myproject.example.net/g ${env.DBNAME}_dev.sql

        # create a backup copy of the current staging database (full backup)
                mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage > ${env.DBNAME}_stage_bak.sql
                # upload the dev database dump to the staging database
                mysql -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage < ${WORKSPACE}/${env.DBNAME}_dev.sql
                rm -f ${WORKSPACE}/${env.DBNAME}_dev.sql"""
       }
        }
        stage("deploy-staging") {
            when {
                environment ignoreCase: true, name: "DEPLOY_TO", value: "staging"
            }
            steps {
                // copy files from dev to control server
                sh """rsync --exclude=.svn --exclude=.git -rav -e ssh webadmin@devserver2:/var/opt/httpd/${env.SITEID}docs/ /tmp/${env.SITEID}docs/

                #Replace the wp-config.php file with our domain file with our information. 
            /bin/cp httpdocs/wp-config-domain.php httpdocs/wp-config.php

                #prepare the staging server to receive files by changing the owner
                ssh webadmin@stageserver 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/'
                # copy files from control server to staging
                rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@stageserver:/var/opt/httpd/${env.SITEID}docs/
                # fix the owner/permissions on the staging server
                ssh webadmin@stageserver 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/ && sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/ && sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'

                #delete the temporary files on the control server
                rm -Rf /tmp/${env.SITEID}docs/
                # clear the Incapsula caches
                if [[ \$( curl -sS -X POST \"http://www.example.net/incapcache.php?api_key=asdaswwGR)feasdsdda&site_id=stage&resource_url=stage-myproject.example.net\" | jq -r .debug_info.id_info) != \"incapsula cache cleared successfuly\" ]]; then exit 255; fi"""
            }
        }
        stage("deploy-db-production") {
            when {
                allOf { 
                    environment ignoreCase: true, name: "DEPLOY_TO", value: "production"; 
                    environment ignoreCase: true, name: "DEPLOY_DB", value: "true"; 
                }
            }
            steps {
                script {
                    def myexcludes = env.EXCLUDE.split(',').toList()
                    MYFLAGS = "-Q -K -c -e --default-character-set=utf8 "
                    if (env.NOFLAGS == "0") {
                        myexcludes.each {
                            MYFLAGS = "${MYFLAGS} --ignore-table=${env.DBNAME}_stage.${it}"
                        }
                    }
                }
                sh """cd ${WORKSPACE}
                # pull a backup of the current staging database (may exclude some tables)
                mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage ${MYFLAGS} > ${env.DBNAME}_stage.sql
        #Searching and replace for the URL to change from the stage sever to the prod server
                sed -i s/stage-myproject.example.net/www.myproject.com/g ${env.DBNAME}_stage.sql

                # create a backup copy of the current production database (full backup)
                mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_prod > ${env.DBNAME}_prod_bak.sql
                # upload the staging database dump to the production database
                mysql -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_prod < ${WORKSPACE}/${env.DBNAME}_stage.sql
                rm -f ${WORKSPACE}/${env.DBNAME}_stage.sql"""
        }
        }
        stage("deploy-production") {
            when {
                environment ignoreCase: true, name: "DEPLOY_TO", value: "production"
            }
            steps {
                // copy files from staging to control server
                sh """rsync --exclude=.svn --exclude=.git -rav -e ssh webadmin@stageserver:/var/opt/httpd/${env.SITEID}docs/ /tmp/${env.SITEID}docs/

                # prepare the production server to receive files by changing the owner
                ssh webadmin@prodserver1 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs'
                ssh webadmin@prodserver2 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs'
                # copy files from control server to production
                rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@prodserver1:/var/opt/httpd/${env.SITEID}docs/
                rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@prodserver2:/var/opt/httpd/${env.SITEID}docs/
                # fix the owner/permissions on the production server
                ssh webadmin@prodserver1 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/'
                ssh webadmin@prodserver2 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/'
                ssh webadmin@prodserver1 'sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/'
                ssh webadmin@prodserver2 'sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/'
                ssh webadmin@prodserver1 'sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'
                ssh webadmin@prodserver2 'sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'

                # delete the temporary files on the control server
                rm -Rf /tmp/${env.SITEID}docs/
                # clear the Incapsula caches
                if [[ \$( curl -sS -X POST \"http://www.example.net/incapcache.php?api_key=asdaswwGR)feasdsdda&site_id=088&resource_url=www.myproject.com\" | jq -r .debug_info.id_info) != \"incapsula cache cleared successfuly\" ]]; then exit 255; fi"""
            }
        }
    }
}

The problems that I'm currently facing with this approach are:

1. I can't figure out how make deployments automated since it's a parameterized pipeline so I'm not sure how to make it automated. The desired process would be to make the deployment automated once Jenkins polling every X amount of minutes on the git repository, deploy to Dev > Stage (only if Dev deployment was successful) automatically and then stop there until we manually deploy to Prod after we do QA on Staging.

2. The current Git configuration has configured only one branch (master) which is where the developers push the changes once they want to make a deployment to Dev -> Stage -> Prod. But I think the ideal scenario would have a dev branch for the dev deployments then stage branch for deploying to Stage environment and then master for deployment once we merge those dev and staging branches to the master branch. I'm not sure if this would be optimal, so I would appreciate any suggestions or ideas on this.

The desired approach will be to have mentioned problems resolved and also have an automated way to deploy and notify once the dev -> staging deployment was successful. As well as having an option to do the mentioned workflow manually, like we are doing right now (this is not that important, but would be a nice to have feature).

Thank you in advance for your help!

Answer 1

1. Get rid of your parameters, they are not needed (would stop you from automation).
2. Have a manual input in the deploy to prod stage

pipeline {
    agent any
    stages {
        stage('Deploy to prod') {
            input {
                message "Should we continue?"
                ok "Yes, we should."
            }
            steps {
                echo "Deploying."
            }
        }
    }
}

3. This should be multibranch pipeline project in Jenkins (since you want to it on all branches)
4. if you want to use different stages for different branches do use when

pipeline {
    agent any
    stages {
        stage('Example Build') {
            steps {
                echo 'Hello World'
            }
        }
        stage('Example Deploy') {
            when {
                branch 'production'
            }
            steps {
                echo 'Deploying'
            }
        }
    }
}

As for suggestions - I would say you need to match your git flow with your CI/CD flow. What's the lifecycle of given git branch type? What's the result of the given stage? Do you want to execute the stages for all the branches and deploy to prod just for one branch?

Answer 2

What abut implementing a separate deploy pipeline which is capable of deploying to all environments and is parametrized and implement another pipeline which is sheduled and triggers the pipeline to deploy to dev (stage dev) and when this job is succesfull then triggers the pipeline again to deploy to stage (stage qa). The deployment to prod can then be done manually.

https://jenkins.io/doc/pipeline/steps/pipeline-build-step/#-build-%20build%20a%20job