Fail to generate ARM64 compliant docker image from Linux/aarch64 container

[Won Cho]

🐞 bug report

Affected Rule

The issue is caused by the rule: py_image

Is this a regression?

I cannot tell.

Description

My Macbook M1 has a linux (aarch64 platform) docker container that has all of my project setup for bazel, and I do all development inside the container. Bazel py_image produced a tarball docker image file. Since it was built from the linux/aarch machine, I expected the tarball image to be aarch compliant. But its targeted platform is x86_64 (or amd64).

🔬 Minimal Reproduction

In WORKSPACE.bazel,

  
container_pull(
    name = "python_arm64",
    architecture = "arm64v8",
    digest = "sha256:6d55a17e24a29d204da7d5d62b7f90f9f88691e36a09add24a287dc10ec46bd1",
    registry = "index.docker.io",
    repository = "python:3.10",
)
  

In BUILD.bazel,

  
py_binary(
    name = "python_shell_simple",
    srcs = ["python_shell_simple.py"],
    main = "python_shell_simple.py",
    python_version = "PY3ONLY",
    srcs_version = "PY3ONLY",
    deps = [
        requirement("botocore"),
        requirement("boto3"),
    ],
)
  

Then run this command,
$ bazel build //batch/file_process:python_shell_simple_image.tar

Copy the tar file to host machine. In my case, Apple M1 macos' filesystem. Load the docker image.
$ docker load -i python_shell_simple_image.tar

Launch a new docker container.
docker run --rm -it bazel/batch/file_process:python_shell_simple_image

🔥 Exception or Error

**WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested**

🌍 Your Environment

Operating System:

  
Ubuntu linux/aarch64 container running in Macbook M1 Monterey MacOS.
  

Output of bazel version:

  
6.0.0
  

Rules_docker version:

  
v0.25.0
  

Anything else relevant?

[Martynas Cibulskis]

Having the same issue, was this resolved?

[Won Cho]

I ended up launching linux docker container (aarch64 cpu) from my macbook M1, and use it for running bazel.

RUN apt-get install -y software-properties-common
RUN apt-get -y --no-install-recommends install apt-utils
RUN apt-get -qq -y install make build-essential > /dev/null
RUN apt-get -qq -y install **crossbuild-essential-amd64** > /dev/null
RUN apt-get -qq -y install libbz2-dev libffi-dev libgdbm-compat-dev libgdbm-dev \
    liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma \
    lzma-dev pkg-config tk-dev uuid-dev zlib1g-dev gfortran
RUN apt-get -qq -y install gdb lcov git llvm wget xz-utils vim curl zip \
    unzip zsh screen iputils-ping > /dev/null

The crossbuild package for ubuntu worked for me.

In my .bazelrc,

build --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64
build:platform_build --incompatible_enable_cc_toolchain_resolution
build:gcc_suite --crosstool_top=//infrastructure/bazel/toolchain:gcc_suite
build:gcc_suite --cpu=amd64
build:gcc_suite --host_crosstool_top=@bazel_tools//tools/cpp:toolchain

Then use this command to cross-compile to x86_64 compatible binary
bazel build --config=gcc_suite //acme/...

[Won Cho]

The gcc_suite toolchain config was done manually. I'm not 100% sure if I did it correctly though.

load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES")
load(
    "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl",
    "feature",
    "flag_group",
    "flag_set",
    "tool_path",
)

all_link_actions = [
    ACTION_NAMES.cpp_link_executable,
    ACTION_NAMES.cpp_link_dynamic_library,
    ACTION_NAMES.cpp_link_nodeps_dynamic_library,
]

def _impl(ctx):
    tool_paths = [
        tool_path(
            name = "gcc",
            path = "/usr/bin/x86_64-linux-gnu-gcc",
        ),
        tool_path(
            name = "ld",
            path = "/usr/bin/x86_64-linux-gnu-ld",
        ),
        tool_path(
            name = "ar",
            path = "/usr/bin/x86_64-linux-gnu-ar",
        ),
        tool_path(
            name = "cpp",
            path = "/usr/bin/x86_64-linux-gnu-cpp",
        ),
        tool_path(
            name = "gcov",
            path = "/usr/bin/x86_64-linux-gnu-gcov",
        ),
        tool_path(
            name = "nm",
            path = "/usr/bin/x86_64-linux-gnu-nm",
        ),
        tool_path(
            name = "objdump",
            path = "/usr/bin/x86_64-linux-gnu-objdump",
        ),
        tool_path(
            name = "strip",
            path = "/usr/bin/x86_64-linux-gnu-strip",
        ),
    ]

    features = [
        feature(
            name = "default_linker_flags",
            enabled = True,
            flag_sets = [
                flag_set(
                    actions = all_link_actions,
                    flag_groups = ([
                        flag_group(
                            flags = [
                                "-lstdc++",
                            ],
                        ),
                    ]),
                ),
            ],
        ),
    ]

    return cc_common.create_cc_toolchain_config_info(
        abi_libc_version = "unknown",
        abi_version = "unknown",
        features = features,
        compiler = "gcc",
        ctx = ctx,
        cxx_builtin_include_directories = [
            "/usr/lib/gcc-cross/x86_64-linux-gnu/11/include",
            "/usr/x86_64-linux-gnu/include",
            "/usr/x86_64-linux-gnu/include/c++/11/x86_64-linux-gnu/",
        ],
        host_system_name = "local",
        target_cpu = "amd64",
        target_libc = "unknown",
        target_system_name = "local",
        toolchain_identifier = "amd64-toolchain",
        tool_paths = tool_paths,
    )

cc_toolchain_config = rule(
    attrs = {},
    provides = [CcToolchainConfigInfo],
    implementation = _impl,
)

[Kevin Park]

Any workaround or fix that does not include running Bazel in a container?

[Jason Banich]

What @kpark-hrp said

[Kevin Park]

My workaround.

1. Any container_pull, mark it with architecture arg. For example,

container_pull(
    name = "debian_arm64_base",
    architecture = "arm64v8",
    digest = "sha256:e9739b2eb6a410c953e5e79a2b99cf0f4b61dbf9437e625ee34f531fa5692b19",  # stable-slim
    registry = "docker.io",
    repository = "library/debian",
)

2. Similar but slightly different architecture tag for any container you build on top of base.

app_layer(
    ...,
    architecture = "arm64",
    ...,
)

This worked for me.