Azure DevOps release pipeline AttributeError: type object 'Retry' has no attribute 'DEFAULT_METHOD_WHITELIST'

Question

I have a release pipeline that has been working fine until 5/4/2023, when it started throwing this error and getting hung up in a retry loop upon trying to start a Databricks cluster. The log looks like this, and it does not exit until a user manually cancels it:

2023-05-04T15:31:48.9504235Z ##[section]Starting: Start cluster
2023-05-04T15:31:48.9507476Z ==============================================================================
2023-05-04T15:31:48.9507600Z Task         : Start a Databricks Cluster
2023-05-04T15:31:48.9507679Z Description  : Make sure a Databricks Cluster is started
2023-05-04T15:31:48.9507786Z Version      : 0.5.6
2023-05-04T15:31:48.9507851Z Author       : Microsoft DevLabs
2023-05-04T15:31:48.9507957Z Help         : 
2023-05-04T15:31:48.9508027Z ==============================================================================
2023-05-04T15:31:49.5839599Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:31:49.6142380Z Cluster *** not running, turning on...
2023-05-04T15:31:49.7846916Z Error: AttributeError: type object 'Retry' has no attribute 'DEFAULT_METHOD_WHITELIST'
2023-05-04T15:31:49.9842231Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:32:20.0262096Z Starting...
2023-05-04T15:32:20.2317560Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:32:50.2656819Z Starting...
2023-05-04T15:32:50.4489482Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:33:20.4791525Z Starting...
2023-05-04T15:33:20.6468387Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:33:50.6758000Z Starting...
2023-05-04T15:33:50.9257125Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:34:20.9617006Z Starting...
2023-05-04T15:34:21.1491705Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:34:51.1782850Z Starting...
2023-05-04T15:34:51.3835540Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:35:21.4213881Z Starting...
2023-05-04T15:35:21.6628904Z parse error: Invalid numeric literal at line 1, column 6
2023-05-04T15:35:51.6999474Z Starting...
2023-05-04T15:35:51.8763360Z parse error: Invalid numeric literal at line 1, column 6

This is happening for release pipelines to several different environments.

I tried restarting the Databricks cluster, but the same thing happens once the cluster starts again.

If the Start Cluster step is removed, the same happens in the next step, where it tries to deploy notebooks to a workspace.

Answer 1

The library was updated on 5/3/23: https://urllib3.readthedocs.io/en/2.0.2/changelog.html#id1

The more recent versions of the library (2.0.0, 2.0.1, 2.0.2) do not have the DEFAULT_METHOD_WHITELIST attribute for the retry class. Earlier versions like 1.24.3 do have it.

It appears that the DevOps framework has not been updated to be compatible with the newer versions.

The changelog shows that the 2.0.0 update happened on 4/26/23, but the issue did not show up in DevOps then, even though 2.0.0 also doesn't have that attribute. It's difficult to know if the changelogs are 100% accurate, what really went into 2.0.2, and if the error is reflective of what's really going on.

We found a workaround by doing a

pip install urllib3==1.24.3

before the Databricks cluster start (or whatever the first step is that accesses Databricks), and that solved the problem.

Update: there was a more recent update in DevOps to use the newer version of the library, so if you implemented the pip install fix before, you may need to remove it for the release pipeline to work again.