"Could not retrieve credential from local cache for service principal" when using Azure CLI 2.30.0 credentials in Python SDK on Azure Devops MS agent

Question

I an Azure Pipeline on a self-hosted agent I use this task

      - task: AzureCLI@2
        displayName: Azure CLI task with Python SDK
        inputs:
          azureSubscription: 'SUBSCRIPTION-SERVICE-CONNECTION'
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            python ./magic-script.py

with that I am able to use the credentials to authenticate Azure Python SDK:

client = get_client_from_cli_profile(GraphRbacManagementClient)

When I transfer this process to a MS hosted agent I get this error:

  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/common/client_factory.py", line 85, in get_client_from_cli_profile
    with_tenant=True,
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/common/credentials.py", line 98, in get_azure_cli_credentials
    cred, subscription_id, tenant_id = profile.get_login_credentials(resource=resource)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/_profile.py", line 335, in get_login_credentials
    credential = self._create_credential(account, client_id=client_id)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/_profile.py", line 592, in _create_credential
    return identity.get_service_principal_credential(username_or_sp_id)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/auth/identity.py", line 185, in get_service_principal_credential
    entry = self._msal_secret_store.load_entry(client_id, self.tenant_id)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/auth/identity.py", line 270, in load_entry
    .format(sp_id))
knack.util.CLIError: Could not retrieve credential from local cache for service principal ***. Run `az login` for this service principal.

---

Based on this migration guide I also tried AzureCliCredential like

credential = AzureCliCredential()
client = GraphRbacManagementClient(credential, os.environ["subscriptionId"])   

which get's my script signed in - but when using GraphRbacManagementClient I get this error locally on my dev box and on the agent:

    root_group = [g for g in graph_client.groups.list(
  File "C:\Python38\lib\site-packages\msrest\paging.py", line 143, in __next__
    self.advance_page()
  File "C:\Python38\lib\site-packages\msrest\paging.py", line 129, in advance_page
    self._response = self._get_next(self.next_link)
  File "C:\Python38\lib\site-packages\azure\graphrbac\operations\groups_operations.py", line 336, in internal_paging
    response = self._client.send(request, stream=False, **operation_config)
  File "C:\Python38\lib\site-packages\msrest\service_client.py", line 336, in send
    pipeline_response = self.config.pipeline.run(request, **kwargs)
  File "C:\Python38\lib\site-packages\msrest\pipeline\__init__.py", line 197, in run
    return first_node.send(pipeline_request, **kwargs)  # type: ignore
  File "C:\Python38\lib\site-packages\msrest\pipeline\__init__.py", line 150, in send
    response = self.next.send(request, **kwargs)
  File "C:\Python38\lib\site-packages\msrest\pipeline\requests.py", line 65, in send
    self._creds.signed_session(session)
AttributeError: 'AzureCliCredential' object has no attribute 'signed_session'

For StorageManagementClient this works. Seems to be depending on the SDK client used.

Answer 1

This seems to be due to an update to Azure-CLI between 2.29.0 and 2.30. In our pipeline, the az login task was running in the agent (MS-hosted) directly, so was using that version. Then our interaction with the API was in a conda environment with a pinned, older version of azure-cli. By doing the az login within the same environment as the subsequent calls (bringing the Azure CLI versions in line), that resolved it.

It is detailed in this github issue: https://github.com/Azure/azure-cli/issues/20153#issuecomment-958684723