job failing with ERROR: gcloud crashed (AttributeError): 'bool' object has no attribute 'lower' [closed]

Question

We noticed our jobs are failing with the below error on the dataproc cluster.

ERROR: gcloud crashed (AttributeError): 'bool' object has no attribute 'lower'

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics

Running diagnostics does not produce valuable insights:

$ gcloud info --run-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

Property diagnostic detects issues that may be caused by properties.
Checking hidden properties...done.
Hidden Property Check passed.
Property diagnostic passed (1/1 checks passed).

Answer 1

If your local gcloud is still broken, force a re-fetch of the feature flags configuration as Google has turned the problematic flag off, then re-run your gcloud invocation:

rm ~/.config/gcloud/.feature_flags_config.yaml

---

This appears to be a bad feature flag value that was recently released, which explains why prior versions of the gcloud SDK are impacted (at least the last few, going back to e.g. 412.0.0 works fine).

If you run gcloud with --verbosity=debug you'll see a stack ending in:

 4281, in _GetBoolProperty
    return property_value.value.lower() in ['1', 'true', 'on', 'yes', 'y']
AttributeError: 'bool' object has no attribute 'lower'

Which occurs when parsing the default value of the auth/service_account_use_self_signed_jwt flag.

Two viable workarounds until Google fixes their flag:

# Not a great idea for CI/CD use cases; this will fail as soon
# as the feature flag is removed.
gcloud config set 'auth/service_account_use_self_signed_jwt' false

# Likely not a great idea either, but it does work via a sledgehammer
# approach by preventing feature flags from being evaluated.
gcloud config set 'core/enable_feature_flags' false

As an update: As of around 2023-05-03T00:00:00Z the new flag disappeared from https://www.gstatic.com/cloudsdk/feature_flag_config_file.yaml so impact should be mitigated outside of people's laptops, which presumably only fetch feature flags periodically.

Our own CI systems have recovered.

It was rolled out as a 10% feature flag (no clue how they bucket, though) which is presumably why impact was relatively small:

# Feature Flag Config File
test/feature_flag:
- value: true
  weight: 0
- value: false
  weight: 100
auth/service_account_use_self_signed_jwt:
- value: true
  weight: 10
- value: false
  weight: 90