kubectl pull image from gitlab unauthorized: HTTP Basic: Access denied

Question

I am trying to configure gitlab ci to deploy app to google compute engine. I have succesfully pushed image to gitlab repository but after applying kubernetes deployment config i see following error in kubectl describe pods:

Failed to pull image "registry.gitlab.com/proj/subproj/api:v1": rpc error: code = 2 
desc = Error response from daemon: {"message":"Get https://registry.gitlab.com/v2/proj/subproj/api/manifests/v1: unauthorized: HTTP Basic: Access denied"}

Here is my deployment gitlab-ci job:

docker:
  stage: docker_images
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker build -t registry.gitlab.com/proj/subproj/api:v1 -f Dockerfile .
    - docker push registry.gitlab.com/proj/subproj/api:v1
  only:
    - master
  dependencies:
  - build_java

k8s-deploy:
  image: google/cloud-sdk
  stage: deploy
  script:
    - echo "$GOOGLE_KEY" > key.json # Google Cloud service account key
    - gcloud auth activate-service-account --key-file key.json
    - gcloud config set compute/zone us-central1-c
    - gcloud config set project proj
    - gcloud config set container/use_client_certificate True
    - gcloud container clusters get-credentials proj-cluster
    - kubectl delete secret registry.gitlab.com  --ignore-not-found
    - kubectl create secret docker-registry registry.gitlab.com --docker-server=https://registry.gitlab.com/v1/ --docker-username="$CI_REGISTRY_USER" --docker-password="$CI_REGISTRY_PASSWORD" [email protected]
    - kubectl apply -f  cloud-kubernetes.yml

and here is cloud-kubernetes.yml:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
  name: proj
  labels:
    app: proj
spec:
  type: LoadBalancer 
  ports:
  - port: 8082
    name: proj
    targetPort: 8082
    nodePort: 32756
  selector:
    app: proj
---    
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: projdeployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: proj
    spec:
      containers:
      - name: projcontainer
        image: registry.gitlab.com/proj/subproj/api:v1
        imagePullPolicy: Always
        env:
          - name: SPRING_PROFILES_ACTIVE
            value: "cloud"
        ports:
        - containerPort: 8082
      imagePullSecrets:
        - name: registry.gitlab.com

I have followed this article

Answer 1

There is workaround, image could be pushed to google container registry, and then pulled from gcr without security. We can push image to gcr without gcloud cli using json token file. So .gitlab-ci.yaml could look like:

docker:
  stage: docker_images
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker build -t registry.gitlab.com/proj/subproj/api:v1 -f Dockerfile .
    - docker push registry.gitlab.com/proj/subproj/api:v1
    - docker tag registry.gitlab.com/proj/subproj/api:v1 gcr.io/proj/api:v1
    - docker login -u _json_key -p "$GOOGLE_KEY" https://gcr.io
    - docker push gcr.io/proj/api:v1
  only:
    - master
  dependencies:
  - build_java

k8s-deploy:
  image: google/cloud-sdk
  stage: deploy
  script:
    - echo "$GOOGLE_KEY" > key.json # Google Cloud service account key
    - gcloud auth activate-service-account --key-file key.json
    - gcloud config set compute/zone us-central1-c
    - gcloud config set project proj
    - gcloud config set container/use_client_certificate True
    - gcloud container clusters get-credentials proj-cluster
    - kubectl apply -f cloud-kubernetes.yml

And image in cloud-kubernetes.yaml should be:

gcr.io/proj/api:v1

Answer 2

You must use --docker-server=CI_REGISTRY. The same as you sue for docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY.

Also note that your docker secrets must be in the same namespace with Deployment/ReplicaSet/DaemonSet/StatefullSet/Job.