Bazel and Pants support?

[alan707]

Are there any plans for the bktec tool to support Bazel and Pants test frameworks. Is there a public roadmap we could take a peek at?

Thank you!

[jameshill]

Hey @alan707,

Thanks for reaching out, indeed there's a public roadmap here you can take a peek at: https://buildkite.com/platform/roadmap/ under the Test Engine tab.

Bazel is indeed on that roadmap and we're evaluating our options with respects to streaming the Build Event Protocol into Test Engine.

We've not done discovery on Pants, are you currently evaluating the two? Can you tell us a bit more about your objectivs/goals and what you're looking to achieve and what timeframes you're working with?

All the best,
James

[jasonwbarnett]

Funny timing — I was just evaluating adding pants support to Test Analytics. I was able to get it working with the pytest runner using minimal code changes:

diff --git c/internal/runner/pytest.go w/internal/runner/pytest.go
index 868bccb..909e2c7 100644
--- c/internal/runner/pytest.go
+++ w/internal/runner/pytest.go
@@ -23,12 +23,6 @@ func (p Pytest) Name() string {
 }

 func NewPytest(c RunnerConfig) Pytest {
-       if !checkPythonPackageInstalled("buildkite_test_collector") { // python import only use underscore
-               fmt.Fprintln(os.Stderr, "Error: Required Python package 'buildkite-test-collector' is not installed.")
-               fmt.Fprintln(os.Stderr, "Please install it with: pip install buildkite-test-collector.")
-               os.Exit(1)
-       }
-
        if c.TestCommand == "" {
                c.TestCommand = "pytest {{testExamples}} --json={{resultPath}}"
        }
@@ -125,8 +119,6 @@ func (p Pytest) commandNameAndArgs(cmd string, testCases []string) (string, []st

        if strings.Contains(cmd, "{{testExamples}}") {
                cmd = strings.Replace(cmd, "{{testExamples}}", testExamples, 1)
-       } else {
-               cmd = cmd + " " + testExamples
        }

        cmd = strings.Replace(cmd, "{{resultPath}}", p.ResultPath, 1)

I’m curious if Buildkite would prefer to extend support for pants using the existing pytest runner, or if you’d prefer a dedicated pants runner. Their implementations could be relatively similar. Another option is to use the same runner with additional logic in the detection phase, depending on whether BUILDKITE_TEST_ENGINE_TEST_RUNNER is set to pants.

I’m looking for input before investing too much time, but I did want to report that I have this working with pants as of today. It doesn’t yet support test splitting or other advanced features, as we heavily rely on pants’ --changed-since flag for test selection.

I had to compile the test-engine-client using the changes above and then I set

export BUILDKITE_TEST_ENGINE_TEST_RUNNER=pytest
export BUILDKITE_TEST_ENGINE_TEST_CMD="pants --changed-since=origin/main --changed-dependents=transitive --test-shard="${matrix}" test --report=true --output=all -- --json={{resultPath}}"
bktec

Please note:

• RE: environment variables
  • There are more that I exported, but aren't particularly relevant as they don't deviate from the standard set of instructions.
• RE: pants arguments
  • --report=true isn't necessary, but we're using it for the junit-annotate buildkite plugin.
  • --output=all isn't necessary either. It's personal preference.
  • --matrix argument is also personal preference. We have 15K+ tests across 30 shards.
• There are steps missing here that are pants specific such has adding buildkite-test-collector to the pytest pants resolve. I'll leave a trail below.

# pants.toml

[test]
extra_env_vars = [
    "BUILDKITE_*", # Adding in preparation of pants 2.25 (when wildcard support was added)
    "BUILDKITE_ANALYTICS_TOKEN",
    "BUILDKITE_BRANCH",
    "BUILDKITE_BUILD_ID",
    "BUILDKITE_BUILD_NUMBER",
    "BUILDKITE_BUILD_URL",
    "BUILDKITE_COMMIT",
    "BUILDKITE_JOB_ID",
    "BUILDKITE_MESSAGE"
]

[pytest]
install_from_resolve = "pytest"
requirements = ["//3rdparty/python:pytest"]
execution_slot_var = "PANTS_EXECUTION_SLOT"

# 3rdparty/python/pytest-requirements.txt

buildkite-test-collector>=1.0.3,<2.0.0
pytest-asyncio>=0.20.3,<1.0.0
pytest-cov>=3.0.0,<5.0.0
pytest-mock>=3.0.0,<4.0.0
pytest-xdist>=3.5.0,<4.0.0
pytest>=7.0.0,<9.0.0

# 3rdparty/python/BUILD

python_requirements(
    name="pytest",
    source="pytest-requirements.txt",
    resolve="pytest",
    overrides={
        "pytest": {"entry_point": "pytest:console_main"},
    },
)

If anyone has questions just shout.