Skip to content

Bazel Python Setup

Set up python bazel environment. As of 2023/04, the recommended way of python-bazel environment is to use standalone rules_python. Also, there is a trend to use bzlmod (with MODULE.bazel instead of WORKSPACE to have module-level setup), see Bzlmod User Guide for more details.

Python Repo Setup

In order to use py bazel rules, we need to make a bazel repository to be a python repository. Once rules_python is enabled inside a bazel repository, we can start to use py_binary, py_library, py_test, etc.

# filename: MODULE.bazel

# Python Rules: rules_python 0.20.0
bazel_dep(name = "rules_python", version = "0.20.0")

pip = use_extension("@rules_python//python:extensions.bzl", "pip")

pip.parse(
    name = "pip",
    requirements_lock = "//:requirements_lock.txt",
)

use_repo(pip, "pip")

# filename: WORKSPACE
# initiate py repository
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "a644da969b6824cc87f8fe7b18101a8a6c57da5db39caa6566ec6109f37d2141",
    strip_prefix = "rules_python-0.20.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.20.0/rules_python-0.20.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

The build/test targets are relatively straightforward for basic py_binary, py_library, and py_test, but there are other useful rules, which I will leave it for a future study. For more information, see Documentations

Python Runtime: hermetic v.s. local host machine

Because python scripts always needs a python runtime to run, and doesn't automatically get compiled into a binary. Which python would run the script would be the issue. By default, bazel would search for python3 and python in the system PATH. In this case, if we run bazel build ..., we will see the bazel-bin/ directory generates an external/ directory, in which there will at least be a /bazel_tools subdirectory for a python wrapper shell scripts. 

bazel-bin
├── external
│   └── bazel_tools
│       └── tools
│           └── python
│               └── py3wrapper.sh
└── hello_world
    ├── main
    ├── main.runfiles
    │   ├── bazel_tools
    │   │   └── ...
    │   ├── __main__
    │   │   ├── external
    │   │   └── hello_world
    │   │       ├── __init__.py
    │   │       ├── main -> /home/feitong/.cache/bazel/_bazel_feitong/9c77a8aefa1ae729b6db1213c36c96f4/execroot/__main__/bazel-out/k8-fastbuild/bin/hello_world/main
    │   │       ├── main.py -> /media/feitong/Data/Projects/monorepo/experimental/python_hello_world/workspace_version/hello_world/main.py
    │   │       └── random_number_generator.py -> /media/feitong/Data/Projects/monorepo/experimental/python_hello_world/workspace_version/hello_world/random_number_generator.py
    │   └── MANIFEST
    └── main.runfiles_manifest
In this case, when you run bazel-bin/hello_world/main, it will use the python runtime on your computer. For details, you can check the py3wrapper.sh files.

If you want to make sure your python binary can run everywhere, and do not depend on a specific python version on your computer, you can make python environment hermetic, and configure python toolchain in your repository.

# filename: MODULE.bazel

# (Optional) Register a specific python toolchain instead of using the host version
python = use_extension("@rules_python//python:extensions.bzl", "python")

python.toolchain(
    name = "python3_10",
    python_version = "3.10",
    # configure_coverage_tool = True,
)

use_repo(python, "python3_9_toolchains")

register_toolchains(
    "@python3_9_toolchains//:all",
)

# filename: WORKSPACE

# register python toolchain. In this case it will use hermetic python.
load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python3_10",
    # Available versions are listed in @rules_python//python:versions.bzl.
    # We recommend using the same version your team is already standardized on.
    python_version = "3.10",
    register_coverage_tool = True,
)

load("@python3_10//:defs.bzl", "interpreter")

If you want to know what python toolchains are provided by the rules_python, you can check @rules_python//python:versions.bzl. If you have registered python toolchains, then when you build a python target, you will find the bazel-bin file directory is different.

  1. you don't have top level bazel-bin/external for a python wrapper anymore.
  2. in the bazel-bin/main.runfiles, you will see the python runtime, e.g. python3_10_x86_64-unknown-linux-gnu, in which it has python binary and multiple necessary libraries.
  3. bazel would generate an __init__.py file, if you don't have that in your source file.
# an example bazel-bin directory structure without hermetic python setup.
bazel-bin
└── hello_world
    ├── main  # this is a code-gen python file that can be run
    ├── main.runfiles
    │   ├── __init__.py
    │   ├── __main__
    │   ├── MANIFEST
    │   └── python3_10_x86_64-unknown-linux-gnu
    └── main.runfiles_manifest

Dependencies and third-party libraries

Todo

To get diagnostic information about which dependencies introduce version requirements, you can run the find_requirements aspect on your target:

bazel build <your target> \
  --aspects=@rules_python//python:defs.bzl%find_requirements \
  --output_groups=pyversioninfo

Reference