-
Notifications
You must be signed in to change notification settings - Fork 6
Allow throttling testing based on PR labels #182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Avoid redundant testing for Make builds, on the assumption that CMake handles most of them: - Skip all GPU and HVX testing - Skip the error, warning, performance, and tutorials (basically, just correctness and generator) - Continue to skip python & apps where we used to - Continue to skip wasm for Make entirely
Followup from recent Halide meeting, this changes build/test defaults to require opt-in for GPU and/or HVX testing, based on labels:
We vary testing depending on how a PR is labeled:
- if 'buildbot_test_nothing' is present, no testing is done and the rest are ignored. ('skip_buildbots' is a synonym for this.)
- if 'buildbot_test_everything' is present, we do all possible testing.
- Otherwise, we default to doing (at least) baseline CPU testing on all targets.
- if 'buildbot_test_gpu' is present, we also test all GPU targets (cuda, opencl, metal, d3d12, etc)
- if 'buildbot_test_hvx' is present, we also test Hexagon
This has been (lightly) tested on a local buildbot and seems pretty close, but needs more testing on a proper setup.
|
It would be nice if these labels corresponded to something that already exists: eg. the actual target/feature names, the app names, or the CTest label names. It's fine to have meta-labels like We also still need to do the work of labeling the GPU-enabled correctness tests and the like, so we don't re-run the exact same test with irrelevant target flags. I'm also not sold on the naming convention. Having |
|
How bad would it be to dust this off and land it? |
|
I'm inclined to argue it's dangerous to disable parts of the test suite. It has happened many times that changing something that feels unrelated at first glance breaks some codegen on another platform. What I think might be useful is to have a Perhaps another useful application of the same mechanism would be to disable LLVM-main-build or LLVM-main-build-and-test, if we already know LLVM-main is broken. This will save time on the buildbot machines for useful work. Suggestion for the tag: |
Followup from recent Halide meeting, this changes build/test defaults to require opt-in for GPU and/or HVX testing, based on labels:
With this in place, we vary testing depending on how a PR is labeled:
This has been (lightly) tested on a local buildbot and seems pretty close, but needs more testing on a proper setup.