Testing with multiple upstream repositories

Question

I have a continuous build and test scenario that I'm trying to figure out how to implement in Jenkins, and I am hoping that someone can lead me in the right direction.

I have a project, foo, that depends on a couple of other upstream projects, bar and baz. All three projects are under continuous development, with API changes, random failures, etc. We want to use recent versions of the upstream projects, since they often make changes at our behest.

Given that there are specific known revisions of bar and baz that work with foo such that foo's tests pass, I would like the following:

Whenever bar or baz change, I would like to build them and test foo building against them. If these tests pass, I would like the new version of bar/baz to be the new "known good" version.

Whenever foo changes, I want to compile against the latest versions of bar and baz. If tests pass, the latest versions of bar and baz become the new "known good" versions. If tests fail, I want to compile against the last "known good" version of bar/baz.

The idea is to quickly learn when upstream changes cause problems, and to continue to check changes in our project against known-good upstream versions. When upstream fails against our project due to upstream bugs, we want to continue using the last known good version for testing. When upstream fails against our project due to API changes or behavioral improvements, we want to update our code to deal with those, and then switch to these new versions for future builds/testing.

I've glanced at Jenkins Pipeline, and it looks at first glance like it might be able to handle something like this, but I really don't want to go down the rabbit hole of learning a new DSL (and Groovy) if this is not a good fit for our problem.

Answer 1

What you describe is a combination of two tasks -- it will be good to clearly separate those:

1. Integration tests of the bar and baz components using the test framework foo
2. Updates of the test framework foo

Integration Tests

The first point can be implemented quite easily: basically, whenver bar or baz successfully pass their "component" CI jobs (say, bar-test or baz-test, respectively), trigger foo-test. In foo-test, pick up the versions of bar and baz and test them. If successful, record those version. Recording could be in several ways, e.g.,

• store in build data of foo (console output, build artifact)
• create some tag in your SCM system
• use the promoted builds plugin to mark builds of bar and baz

Updating the Test Framework

Point "2." is a bit more difficult in this context, and I think it is unwise to use one and the same foo-test job for the purposes of "1." and "2.": the objective here is completely different from integrating bar and baz, and especially the handling from SCM side is very different. It will be good to create a dedicated job -- say, foo-update -- for updating the version of foo that must be used for integrating bar and baz.

Such a job foo-update could work as follows:

• Trigger execution whenever foo changes
• When running, use the last known good versions of bar and baz, as determined in task "1."
• Perform foo integration tests of bar and baz. You know that they did pass successfully before, so...
  • If tests do not pass any longer, then it is due to the changes in foo. Manual action is then needed to handle this in foo, bar or baz.
  • If tests do pass, the test framework is compatible. Record the new version of foo as the version to be used in integration tests "1."

Recording the proper revision of foo can be done again on SCM-level or by recording it in Jenkins itself. Of course, your integration tests in foo-test need to pick up that version prior to execution.

Answer 2

I guess it depends if you really want to have all the combinations of bar and baz versions tested. This might be not trivial. If it's not a problem to test just the latest ones, then assuming you create version of you artifacts for bar and baz as a derivative of last successful build number, those could look like:

pipeline {
    agent any
    stages {
        stage('Example') {
            steps {
                echo 'Hello World'
            }
        }
    }
    post { 
        success { 
            build(job: 'test-job', wait: false)
        }
    }
}

Now the tricky part is the test-job, it should be triggered by either bar or baz, their versions however might not entirely be known if you pass just one of them a parameter. What I would propose instead would be to query the Jenkins API for latest successful build number of bar or baz jobs and derive the latest artifact version you would like to base your tests on in test-job. From your test-job in order to obtain the last successful build number you would do

curl --user <userName:password> https://<url>/job/<job-Name>/api/json?xpath=/*/lastStableBuild/number

Finally after successful test you would be to label both in your artifact solution as latest verified or something like this (this information would have to be stored in some external system).

This brings us to your foo job - it doesn't have a single responsibility, which I find confusing. I would rather say that tests of bar and baz should be a special job (test and label responsibility). Then foo should focus on using either latest verified or just latest artifacts for bar and baz. (which keeps me wondering, how do you manage versions of bar and baz in your foo project's configuration?)