docker push fails : manifest invalid

Question

When trying to do docker push <IMAGE> to a private docker image registry, getting an error saying manifest invalid

along with the error, it's giving some [DEPRECATION NOTICE], which might be the cause of the failure as well.

Note: After getting this error and failing the docker push command. (in ci/cd it's failing the job, I'm guessing its throwing exit 1 ), when I check back in the registry, the image is present.

In the same docker registry, there is other images present as well. and they also are going through the same CI/CD pipeline. but for them, there is no issue detected. is there any possibility that the issue is originating from the Dockerfile? (Although it used to work previously, with the same Dockerfile)

I have doubled check that, the image version (e.g: <IMAGE_NAME>:0.0.1-5e90df92ed140-development) I'm pushing is not present in the registry.

I have also referred to multiple questions and issues regarding this and crossed checked the

• The user has permission to push & pull images
• manifests having the same numbers of fsLayers as history

Note: We are using JFrog Artifactory version to 6.12.2

Docker Version (In my local machine, where it fails as well. with the same message):

Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Error:

596280599f68: Layer already exists
5d6cbe0dbcf9: Pushed 
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the xxxxxxx.docker.repositories.xxxxxx.com registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
manifest invalid: manifest invalid

Answer 1

Ran into the same problem. The issue was that repo does not let you over-write images tags. I changed the tag to a new major version.

Answer 2

Ran into the samish problem with 6.16. The issue was authorization related. I had to add the Delete / Override Repository Action as well to the build agent user.

The build agent only had write permissions, and the second time it built the image with the same tag, it could not overwrite the manifest! This made the client to roll back to schema1 for some reason.

Figured it out through the System Logs of Artifactory.

Answer 3

This can happen after your system (e.g. Ubuntu) upgrades docker-ce-cli to a newer version e.g. from 5:19.03.15~3-0~ubuntu-bionic to 5:20.10.3~3-0~ubuntu-bionic

If upgrading your docker-registry to a newer version is not an option at the moment, then you might try to downgrade your docker-ce-cli to some previous version. To see the list of available versions:

sudo apt-cache policy docker-ce

sudo apt-cache policy docker-ce-cli

To downgrade to previous version run:

sudo apt install docker-ce=5:19.03.15~3-0~ubuntu-bionic docker-ce-cli=5:19.03.15~3-0~ubuntu-bionic

Same method can be applied to downgrade docker-ce package.

Check version:

docker --version

Then try to push again:

docker push your-docker-registry.com:5000/myprojects/my-postgresql-image:latest

Please note that keeping your software up-to-date might improve security of your system(s)!

Answer 4

Publishing my artifacts into artifactory failed with the same reason manifest invalid: manifest invalid due to the previous docker image existed. Removing the docker image resolved the issue.

Error

The push refers to repository [xxxxx.com/foo]
83850d400bdd: Pushed
309acb279757: Pushed
ebf469e844a6: Layer already exists
94c94d3b2e6b: Layer already exists
manifest invalid: manifest invalid
🚨 Error: The command exited with status 1

To remove the docker image

docker image rm <name>