How to use Bazel's py_library imports argument

Question

I'm trying to use Boto3 in a Bazel built project but can't seem to get the correct import for the library. Because of the Boto git repository, all the sources are in folders named botocore and boto3 in the root of the repository. The imports are all boto3.boto3, with the first corresponding the name of the external dependency and second being the root folder in which the reside. How do I use the imports attribute of py_binary and py_library rules to import from the inner boto3 instead of the other one?

This is what my workspace looks like:

//WORKSPACE

BOTOCORE_BUILD_FILE = """

py_library(
    name = "botocore",
    srcs = glob([ "botocore/**/*.py" ]),
    imports = [ "botocore" ],
    visibility = [ "//visibility:public" ],
)

"""

_BOTO3_BUILD_FILE = """

py_library(
    name = "boto3",
    srcs = glob([ "boto3/**/*.py" ]),
    imports = [ "boto3" ],
    deps = [ "@botocore//:botocore" ],
    visibility = [ "//visibility:public" ],
)

"""

new_git_repository(
    name = "botocore",
    commit = "cc3da098d06392c332a60427ff434aa51ba31699",
    remote = "https://github.com/boto/botocore.git",
    build_file_content = _BOTOCORE_BUILD_FILE,
)

new_git_repository(
    name = "boto3",
    commit = "8227503d7b1322b45052a16b197ac41fedd634e9", # 1.4.4
    remote = "https://github.com/boto/boto3.git",
    build_file_content = _BOTO3_BUILD_FILE,
)

//BUILD

py_binary(
    name = "example",
    srcs = [ "example.py" ],
    deps = [
        "@boto3//:boto3",
    ],
)

//example.py

import boto3

boto3.client('')

Checking for the contents of the build folder

$ ls bazel-bin/example.runfiles/*
bazel-bin/example.runfiles/__init__.py bazel-bin/example.runfiles/MANIFEST

bazel-bin/example.runfiles/boto3:
boto3  __init__.py

bazel-bin/example.runfiles/botocore:
botocore  __init__.py

When I try to run the example script I get AttributeError: 'module' object has no attribute 'client' I can import boto3.boto3 but then using anything in it results in missing dependencies such as boto3.sessions because everything is nested in <target-name>.boto3

Answer 1

I think you're on the right track, but you're running into a subtle problem due to the ordering of the python sys.path.

If I run your example and print out sys.path in example.py, I see that the path contains in order:

bazel-out/local-fastbuild/bin/example.runfiles
bazel-out/local-fastbuild/bin/example.runfiles/boto3/boto3
bazel-out/local-fastbuild/bin/example.runfiles/boto3

The second line is due to the imports = ['boto3'] in your WORKSPACE file.

I think you want the third line to be where you get import boto3 from, because you want python to see bazel-out/local-fastbuild/bin/example.runfiles/boto3/boto3/__init__.py.

So when python evaluates import boto3, it sees bazel-out/local-fastbuild/bin/example.runfiles/boto3/__init__.py from the first entry and uses that, instead of bazel-out/local-fastbuild/bin/example.runfiles/boto3/boto3/__init__.py from the third entry.

I think the answer here, is to name your "workspace" something other than the directory it contains. For example:

# WORKSPACE
new_git_repository(
  name = "boto3_archive",
  commit = "8227503d7b1322b45052a16b197ac41fedd634e9", # 1.4.4
  remote = "https://github.com/boto/boto3.git",
  build_file_content = _BOTO3_BUILD_FILE,
)

# BUILD
py_binary(
  name = "example",
  srcs = [ "example.py" ],
  deps = [
    "@boto3_archive//:boto3",
  ],
)

When I do this in your example, I get the following error: ImportError: No module named dateutil.parser, which I think is progress.

Answer 2

Files

$ tree
.
├── BUILD
├── WORKSPACE
├── requirements.txt
└── src
    └── main.py

requirements.txt

$ cat requirements.txt 
boto3==1.13.4

WORKSPACE

$ cat WORKSPACE 
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "rules_python",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.0.2/rules_python-0.0.2.tar.gz",
    strip_prefix = "rules_python-0.0.2",
    sha256 = "b5668cde8bb6e3515057ef465a35ad712214962f0b3a314e551204266c7be90c",
)
load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

# Only needed if using the packaging rules.
load("@rules_python//python:pip.bzl", "pip_repositories")

pip_repositories()


# Python external packages installation
load(
    "@rules_python//python:pip.bzl", "pip3_import"
)

pip3_import(
    name = "lambda_deps",
    requirements = "//:requirements.txt", # Top level requirements.txt file
)

load("@lambda_deps//:requirements.bzl", "pip_install")
pip_install()

BUILD

$ cat BUILD
load(
    "@lambda_deps//:requirements.bzl",
    "requirement"
)

py_binary(
    name = 's3_test',
    main = 'main.py',
    srcs = ['src/main.py'],
    deps = [
        requirement('boto3')
    ]
)

src/main.py

$ cat src/main.py
import boto3


def hello_boto3():
    print('hello', boto3.client('s3'))


if __name__ == '__main__':
    hello_boto3()

Build and run

$ bazel clean && bazel build //:s3_test
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed target //:s3_test (24 packages loaded, 1234 targets configured).
INFO: Found 1 target...
Target //:s3_test up-to-date:
  bazel-bin/s3_test
INFO: Elapsed time: 11.666s, Critical Path: 0.25s
INFO: 0 processes.
INFO: Build completed successfully, 5 total actions

$ ./bazel-bin/s3_test
hello <botocore.client.S3 object at 0x7ff1b6686b38>