How to fix k8s namespace permissions in gitlab ci

Question

As I'm playing around with K8s deployment and Gitlab CI my deployment got stuck with the state ContainerStarting.

To reset that, I deleted the K8s namespace using kubectl delete namespaces my-namespace.

Now my Gitlab runner shows me

$ ensure_namespace
Checking namespace [MASKED]-docker-3
error: the server doesn't have a resource type "namespace"
error: You must be logged in to the server (Unauthorized)

I think that has something to do with RBAC and most likely Gitlab created that namespace with some arguments and permissions (but I don't know exactly when and how that happens), which are missing now because of my deletion.

Anybody got an idea on how to fix this issue?

Answer 1

In my case I had to delete the namespace in Gitlab database, so gitlab would readd service account and namespace:

On the gitlab machine or task runner enter the PostgreSQL console:

gitlab-rails dbconsole -p

Then select the database:

\c gitlabhq_production

Next step is to find the namespace that was deleted:

SELECT id, namespace FROM clusters_kubernetes_namespaces;

Take the id of the namespace to delete it:

DELETE FROM clusters_kubernetes_namespaces WHERE id IN (6,7);

Now you can restart the pipeline and the namespace and service account will be readded.