GitLab CE: How to restore or repair repos with issues / merge requests that are suddenly missing?

Question

I started running GitLab CE inside of an x86 Debian VM locally about two years ago, and last year I decided to migrate the GitLab CE instance to a dedicated Intel NUC server. Everything appeared to go well with no issues, and my GitLab CE instance is up-to-date as of today (running 13.4.2).

I discovered recently though, that some repos that were moved give a "NO REPOSITORY!" error when visiting their project pages, and if they had any issue boards, merge requests, etc, that these were also gone. But you wouldn't suspect it since the broken repos appear in the repo lists along with working repos that I use all the time.

If I had to reason about these broken repos, it would be that they had their last activity over a year ago, with either no pushes ever made to them other than an initial push, or if changes were made, issues created, or merge requests created, it was literally over a year ago.

Some of these broken repos are rather large with a lot of history, whereas others are super tiny (literally just tracking changes to a shell script), so I don't think repo size itself has anything to do with it.

If I run the GitLab diagnostic check sudo gitlab-rake gitlab:check, everything looks good except for "hashed storage":

All projects are in hashed storage? ... no
  Try fixing it:
  Please migrate all projects to hashed storage

But then running sudo gitlab-rake gitlab:storage:migrate_to_hashed doesn't appear to complete (with something like six failed jobs in the dashboard), and running the "gitlab:check" again still indicates this "hashed storage" problem. I've also tried running sudo gitlab-rake gitlab:git:fsck and sudo gitlab-rake cache:clear but these commands don't seem to make a difference.

Luckily I have the latest versions of all the missing repos on my machine, and in fact, I still have the original VM running GitLab CE 12.8.5 (with slightly out of date copies of the repos.)

So my questions are:

1. Is it possible to "repair" the broken repos on my current instance? I suspect I could just "re-push" my local copies of these repos back up to my server, but I really don't want to lose any metadata like issues / merge requests and such.
2. Is there any way to resolve the "not all projects are in hashed storage" issue? (Again the migrate_to_hashed task fails to complete.)
3. Would I be able to do something like "backup", "inspect / tweak backup", "restore backup" kind of thing to fix the broken repos, or at least the metadata?

Thanks in advance.

Answer 1

Okay, so I think I figured out what happened.

I found this thread on the GitLab User Forums.

Apparently the scenario here is:

1. Have a GitLab instance that has repos not in "hashed storage"
2. Backup your repo
3. Restore your repo (either to the same server or migrating to another server)
4. Either automatically or manually, attempt to update your repos to "hashed storage"
5. You'll now find that any repo with a "ci runner" (continuous integration runner) will now be listed as "NO REPOSITORY!" and be completely unavailable, since the "hashed storage" migration process will fail

The fix is to:

1. Reset runner registration tokens as listed in this article in the GitLab documentation
2. Re-run the sudo gitlab-rake gitlab:storage:migrate_to_hashed process
3. Once the background jobs are completed, run sudo gitlab-rake gitlab:check to ensure the output contains the message:

All projects are in hashed storage? ... yes

If successful, the projects that stated "NO REPOSITORY!" should now be fully restored.

A key to know if you need to run this process is if you:

1. Log in to your GitLab CE instance as an admin
2. Go to the Admin Area
3. Look under Monitoring->Background Jobs->Dead
4. and see a job with the name

hashed_storage:hashed_storage_project_migrate

with the error

OpenSSL::Cipher::CipherError: