google.cloud namespace import error in __init__.py

Question

I have read through at least a dozen different stackoverflow questions that all present the same basic problem and have the same basic answer: either the module isn't installed correctly or the OP is doing the import wrong.

In this case, I am trying to do from google.cloud import secretmanager_v1beta1.

It works in my airflow container when I run airflow dags or if I run pytest tests/dags/test_my_dag.py. However, if I run cd dags; python -m my_dag or cd dags; python my_dag.py I get this error:

from google.cloud import secretmanager as secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)

I can add from google.cloud import bigquery in the line right above this line and that works OK. It appears to literally just be a problem with this particular package.

Why does it matter if pytest and airflow commands succeed? Because, I have another environment where I am trying to run dataflow jobs from the command-line and I get this same error. And unfortunately I don't think I can bypass this error in that environment for several reasons.

UPDATE 6

I have narrowed down the error to an issue with the google.cloud namespace and the secretmanager package within that namespace in the __init__.py file.

If I add from google.cloud import secretmanager to airflow/dags/__init__.py and then try to run python -m dags.my_dag.py, I receive this error but with a slightly different stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/workspace/airflow/dags/__init__.py", line 3, in <module>
    from google.cloud import secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)

OLD INFORMATION

I am 95% sure that it's still a path problem and that pytest and airflow are fixing something I'm not aware of that isn't handled when I try to manually run the python script.

Things I have tried:

cd /airflow; python setup.py develop --user
cd /airflow; pip install -e . --user
cd /airflow/dags; pip install -r requirements.txt --user

UPDATE

As per requests in the comments, here are the contents of requirements.txt:

boto3>=1.7.84
google-auth==1.11.2
google-cloud-bigtable==1.2.1
google-cloud-bigquery==1.24.0
google-cloud-spanner==1.14.0
google-cloud-storage==1.26.0
google-cloud-logging==1.14.0
google-cloud-secret-manager>=0.2.0
pycloudsqlproxy>=0.0.15
pyconfighelper>=0.0.7
pymysql==0.9.3
setuptools==45.2.0
six==1.14.0

And I accidentally omitted the --user flags from the pip and python installation command examples above. In my container environment everything is installed into the user's home directory using --user and NOT in the global site-packages directory.

UPDATE 2

I've added the following code to the file that is generating the error:

print('***********************************************************************************')
import sys
print(sys.path)
from google.cloud import secretmanager_v1beta1 as secretmanager
print('secretmanager.__file__: {}'.format(secretmanager.__file__))

From airflow list_dags:

['/home/app/.local/bin', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/workspace/airflow', '/usr/local/lib/python3.7/site-packages', '/workspace/airflow/dags', '/workspace/airflow/config', '/workspace/airflow/plugins']
secretmanager.__file__: /home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg/google/cloud/secretmanager_v1beta1/__init__.py

From python my_dag.py:

['/workspace/airflow/dags', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/icentris_ml_airflow-0.0.0-py3.7.egg', '/usr/local/lib/python3.7/site-packages']

UPDATE 3 tree airflow/dags

airflow/dags
├── __init__.py
├── __pycache__
│   ├── __init__.cpython-37.pyc
│   ├── bq_to_cs.cpython-37.pyc
│   ├── bq_to_wrench.cpython-37.pyc
│   ├── fetch_cloudsql_tables-bluesun.cpython-37.pyc
│   ├── fetch_cloudsql_tables.cpython-37.pyc
│   ├── fetch_app_tables-bluesun.cpython-37.pyc
│   ├── fetch_app_tables.cpython-37.pyc
│   ├── gcs_to_cloudsql.cpython-37.pyc
│   ├── gcs_to_s3.cpython-37.pyc
│   ├── lake_to_staging.cpython-37.pyc
│   ├── schedule_dfs_sql_to_bq-bluesun.cpython-37.pyc
│   ├── schedule_dfs_sql_to_bq.cpython-37.pyc
│   ├── app_to_bq_initial_load-bluesun.cpython-37.pyc
│   ├── app_to_lake-bluesun.cpython-37.pyc
│   └── app_to_lake.cpython-37.pyc
├── bq_to_wrench.py
├── composer_variables.json
├── my_ml_airflow.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── lake_to_staging.py
├── libs
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-37.pyc
│   │   ├── checkpoint.cpython-37.pyc
│   │   └── utils.cpython-37.pyc
│   ├── checkpoint.py
│   ├── io
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   └── __init__.cpython-37.pyc
│   │   └── gcp
│   │       ├── __init__.py
│   │       ├── __pycache__
│   │       │   ├── __init__.cpython-37.pyc
│   │       │   └── storage.cpython-37.pyc
│   │       └── storage.py
│   ├── shared -> /workspace/shared/
│   └── utils.py
├── requirements.txt
├── table_lists
│   └── table-list.json
└── templates
    └── sql
        ├── lake_to_staging.contacts.sql
        ├── lake_to_staging.orders.sql
        └── lake_to_staging.users.sql

11 directories, 41 files

UPDATE 4

I tried fixing it so that sys.path looked the same when running python dags/my_dag.py as it does when running airflow list_dags or pytest test_my_dag.py.

Still get the same error.

Looking at a more recent version of documentation, I noticed that you should be able to just do from google.cloud import secretmanager. Which gave me the same result (works with airflow and pytest, not when trying to run directly).

At this point, my best guess is that it has something to do with namespace magic, but I'm not sure?

Answer 1

It have to be installed via terminal: pip install google-cloud-secret-manager Because package name is not secretmanager but google-cloud-secret-manager

Answer 2

Similar to Noah's answer, this fixed the issue for me without having to import an unneeded module:

import google.cloud.secretmanager as secretmanager