Renovate with Google Cloud, Github, Python project, and pyproject.toml

Question

High Level

I am trying to allow Renovate to manage Python dependency updates, including updates dependent upon our own internal registry, which lives in Google's Artifact Registry. We use GitHub to manage our repositories, and Github's provision of Renovate.

Details

To do this, I need to update the renovate.json5 file to allow access to the repo. There is scant information in the documentation, and I have looked in the following places:

• Private package support
• Python package manager support
• Renovate Git Issues filtered for pyproject.toml
• Scant documentation on GCP

and as far as I can tell, I am supposed to add something like this to get access to my private repo:

{
  "hostRules": [
    {
      "matchHost": "https://registry.company.com/pypi-simple/",
      "username": "engineering",
      "password": "abc123"
    }
  ]
}

It seems that Renovate can only use username + password for Python packages. However, I am using a service account, for which Google uses tokens (and other passwordless authentication means). For example, to access the repo, I add the following to pip.conf (or to [[tool.pdm.source]] in my pyproject.toml since I am using PDM to build):

[global]
extra-index-url = https://_json_key_base64:<long_string_for_private_info>@us-west1-python.pkg.dev/my-project/my-repo/simple/

In the case of Google / GCP, that <long_string_for_private_info> is actually a base 64 encoded JSON blob with a bunch of information:

> echo <long_string_for_private_info> | base64 --decode
{
  "type": "service_account",
  "project_id": "my-project",
  "private_key_id": "<filtered out for security>",
  "private_key": "-----BEGIN PRIVATE KEY-----\n<Filtered out for security>\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "<filtered out for security>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-project.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

The filtered out private key is also base64 encoded (so that there is a base64 encoded object that after decoding, it has another base64 encoded object in it).

Attempts so far

I have tried the following:

Using a token:

{
  "matchHost": "us-west1-python.pkg.dev",
  "hostType": "pypi",
  "encrypted": {
    "token": <hidden for security>
  }
}

Using a username and password:

{
  "matchHost": "us-west1-python.pkg.dev",
  "hostType": "pypi",
  "username": "_json_key_base64"
  "encrypted": {
    "password": <hidden for security>
  }
}

In each of the cases above (token or username/password), I have:

• Always encoded using Renovate's encryption page
• Tried the base64 encoded version of just the "private_key" mentioned above
• Tried the decoded version of just the "private_key" mentioned above
• Tried the base64 encoded entire "long_string_for_private_info" object mentioned in the extra-index-url in pip.conf.
  • Remember that I am able to download (and upload) Python packages to the registry, so I know that pip.conf file is correct.
• Tried the decoded version of the entire "long_string_for_private_info".

At this point, I have run out of ideas, and the documentation does not really explain how to use tokens with Python packages at all, let alone with Google's cloud infrastructure.

Error Reported

No matter how the secret is seen, the error is always the same, a variant of:

{
  "err": {
    "validationError": "Encrypted secret is scoped to a different repository: \"US-WEST1-PYTHON.PKG.DEV/MY-PROJ/MY-REPO\".",
    "message": "config-validation",
    "stack": "Error: config-validation\n    at tryDecrypt (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:124:31)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:186:30)\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:238:13)\n    at mergeRenovateConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/merge.ts:252:27)\n    at getRepoConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/config.ts:12:12)\n    at initRepo (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/index.ts:44:12)\n    at Object.renovateRepository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/index.ts:55:14)\n    at attributes.repository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:184:11)\n    at start (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:169:7)\n    at /opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/renovate.ts:18:22"

.
.
.

{
  "error": {
    "validationError": "Failed to decrypt field token. Please re-encrypt and try again.",
    "message": "config-validation",
    "stack": "Error: config-validation\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:192:27)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:238:13)\n    at mergeRenovateConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/merge.ts:252:27)\n    at getRepoConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/config.ts:12:12)\n    at initRepo (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/index.ts:44:12)\n    at Object.renovateRepository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/index.ts:55:14)\n    at attributes.repository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:184:11)\n    at start (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:169:7)\n    at /opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/renovate.ts:18:22"
  }
}

I have even inspected those two functions on Renovate's github repo, but they are very "introspective" and javascript is not my best language, so I could not get too far.

Answer 1

I actually did get this to work eventually, as I describe in Renovate's GitHub.

But I was only able to get things to work if I did not encrypt the password.

The following worked in the renovate.json5 file:

"hostRules": [
    {
       "matchHost": "us-west1-python.pkg.dev",
      "hostType": "pypi",
      "username": "_json_key_base64",
      "password": "<filtered for security>"
    }
]

And the following did not work:

"hostRules": [
    {
        "matchHost": "us-west1-python.pkg.dev",
        "hostType": "pypi",
        "username": "_json_key_base64",
        "encrypted": {
            "password": <the encrypted value spit out by using mend renovate>
        }
    }
]

The reference to Mend Renovate is to this site.

---

At this point, I think it might be a bug. But I'm still not convinced. Regardless, I can get this to work with the unencrypted base64 "password", but I cannot get it to work after encryption.

I suspect this is because Google's Artifact Registry does not just use a simple password, but it sends a relatively complex object. But I am not confident of this, and still looking for some help if anyone knows better, or if anyone can test it out and get things to work.