"jx boot" fails in "openshift-3.11" provider with "tekton pipeline controller" pod into "crashloopbackoff" state

Question

Summary:

I already have a setup of "static jenkins server" type jenkins-x running in openshift 3.11 provider. The cluster was crashed and I want to reinstall jenkins-x in my cluster but there is no support for "static jenkins server" now.

So I am trying to install "jenkins-x" via "jx boot" but the installation fails with "tekton pipeline controller" pod into "crashloopbackoff" state.

Steps to reproduce the behavior:

jx-requirements.yml:

autoUpdate:
      enabled: false
      schedule: ""
    bootConfigURL: https://github.com/jenkins-x/jenkins-x-boot-config.git
    cluster:
      clusterName: cic-60
      devEnvApprovers:
      - automation
      environmentGitOwner: cic-60
      gitKind: bitbucketserver
      gitName: bs
      gitServer: http://rtx-swtl-git.fnc.net.local
      namespace: jx
      provider: openshift
      registry: docker-registry.default.svc:5000
    environments:
    - ingress:
        domain: 172.29.35.81.nip.io
        externalDNS: false
        namespaceSubDomain: -jx.
        tls:
          email: ""
          enabled: false
          production: false
      key: dev
      repository: environment-cic-60-dev
    - ingress:
        domain: ""
        externalDNS: false
        namespaceSubDomain: ""
        tls:
          email: ""
          enabled: false
          production: false
      key: staging
      repository: environment-cic-60-staging
    - ingress:
        domain: ""
        externalDNS: false
        namespaceSubDomain: ""
        tls:
          email: ""
          enabled: false
          production: false
      key: production
      repository: environment-cic-60-production
    gitops: true
    ingress:
      domain: 172.29.35.81.nip.io
      externalDNS: false
      namespaceSubDomain: -jx.
      tls:
        email: ""
        enabled: false
        production: false
    kaniko: true
    repository: nexus
    secretStorage: local
    storage:
      backup:
        enabled: false
        url: ""
      logs:
        enabled: false
        url: ""
      reports:
        enabled: false
        url: ""
      repository:
        enabled: false
        url: ""
    vault: {}
    velero:
      schedule: ""
      ttl: ""
    versionStream:
      ref: v1.0.562
      url: https://github.com/jenkins-x/jenkins-x-versions.git
    webhook: lighthouse

Expected behavior:

All the pods under jx namespace should be up & running and jenkins-x should be installed properly

Actual behavior:

Tekton pipeline controller pod is into "CrashLoopBackOff" state with error:

Pods with status in "jx" namespace:

NAME                                           READY     STATUS             RESTARTS   AGE
jenkins-x-chartmuseum-5687695d57-pp994         1/1       Running            0          1d
jenkins-x-controllerbuild-78b4b56695-mg2vs     1/1       Running            0          1d
jenkins-x-controllerrole-765cf99bdb-swshp      1/1       Running            0          1d
jenkins-x-docker-registry-5bcd587565-rhd7q     1/1       Running            0          1d
jenkins-x-gcactivities-1598421600-jtgm6        0/1       Completed          0          1h
jenkins-x-gcactivities-1598423400-4rd76        0/1       Completed          0          43m
jenkins-x-gcactivities-1598425200-sd7xm        0/1       Completed          0          13m
jenkins-x-gcpods-1598421600-z7s4w              0/1       Completed          0          1h
jenkins-x-gcpods-1598423400-vzb6p              0/1       Completed          0          43m
jenkins-x-gcpods-1598425200-56zdp              0/1       Completed          0          13m
jenkins-x-gcpreviews-1598421600-5k4vf          0/1       Completed          0          1h
jenkins-x-nexus-c7dcb47c7-fh7kx                1/1       Running            0          1d
lighthouse-foghorn-654c868bc8-d5w57            1/1       Running            0          1d
lighthouse-gc-jobs-1598421600-bmsq8            0/1       Completed          0          1h
lighthouse-gc-jobs-1598423400-zskt5            0/1       Completed          0          43m
lighthouse-gc-jobs-1598425200-m9gtd            0/1       Completed          0          13m
lighthouse-jx-controller-6c9b8994bd-qt6tc      1/1       Running            0          1d
lighthouse-keeper-7c6fd9466f-gdjjt             1/1       Running            0          1d
lighthouse-webhooks-56668dc58b-4c52j           1/1       Running            0          1d
lighthouse-webhooks-56668dc58b-8dh27           1/1       Running            0          1d
tekton-pipelines-controller-76c8c8dd78-llj4c   0/1       CrashLoopBackOff   436        1d
tiller-7ddfd45c57-rwtt9                        1/1       Running            0          1d

Error log:

2020/08/24 18:38:00 Registering 4 clients
2020/08/24 18:38:00 Registering 3 informer factories
2020/08/24 18:38:00 Registering 8 informers
2020/08/24 18:38:00 Registering 2 controllers
{"level":"info","caller":"logging/config.go:108","msg":"Successfully created the logger."}
{"level":"info","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"fatal","logger":"tekton","caller":"sharedmain/main.go:149","msg":"Version check failed","commit":"821ac4d","error":"kubernetes version \"v1.11.0\" is not compatible, need at least \"v1.14.0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"github.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain.MainWithConfig\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain/main.go:149\ngithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain.MainWithContext\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain/main.go:114\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/controller/main.go:72\nruntime.main\n\truntime/proc.go:203"}
After downgrading the tekton image from "0.11.0" to "0.9.0" the tekton pipeline controller pod is into running state. And a new tekton pipeline webhook pod got created and it is into "Crashloopbackoff"

Jx version:

Version        2.1.127
Commit         4bc05a9
Build date     2020-08-05T20:34:57Z
Go version     1.13.8
Git tree state clean

Diagnostic information:

The output of jx diagnose version is:

Running in namespace: jx
Version        2.1.127
Commit         4bc05a9
Build date     2020-08-05T20:34:57Z
Go version     1.13.8
Git tree state clean
NAME                          VERSION
Kubernetes cluster            v1.11.0+d4cacc0
kubectl (installed in JX_BIN) v1.16.6-beta.0
helm client                   2.16.9
git                           2.24.1
Operating System              "CentOS Linux release 7.8.2003 (Core)"

Please visit https://jenkins-x.io/faq/issues/ for any known issues.

Finished printing diagnostic information

Kubernetes cluster: openshift - 3.11

Kubectl version:

Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2018-10-15T09:45:30Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Operating system / Environment:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

I need to install "jenkins-x" via "jx boot" in "openshift-3.11" which uses default kubernetes version - 1.11.0 but "jx boot" requires atleast 1.14.0. Please suggest if there is any work around to get jenkins-x on openshift-3.11

Answer 1

As the error message shows (in the crashloop), kubernetes version "v1.11.0" is not compatible, need at least "v1.14.0", which make it not installable on OpenShift 3 (as it ships with Kubernetes 1.11.0). It seems jenkins-X comes with Tetkon Pipelines v0.14.2 which requires at least Kubernetes 1.14.0 (and later releases like Tekton Pipelines v0.15.0 requires Kubernetes 1.16.0).

{"level":"fatal","logger":"tekton","caller":"sharedmain/main.go:149","msg":"Version check failed","commit":"821ac4d","error":"kubernetes version \"v1.11.0\" is not compatible, need at least \"v1.14.0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"github.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain.MainWithConfig\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain/main.go:149\ngithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain.MainWithContext\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain/main.go:114\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/controller/main.go:72\nruntime.main\n\truntime/proc.go:203"}

Theorically, setting KUBERNETES_MIN_VERSION in the controller deployment might make it work but it is not being tested and most likely the controller won't behave correctly as it's using feature that are not available in 1.11.0. Other than this, there is no workaround that I know of.