build: build native library once and share across CI test jobs

[andygrove]

Summary

This PR optimizes CI build times by building the native Rust library once and sharing it across all test jobs, instead of rebuilding it in every job.

Changes

PR Build Linux (pr_build_linux.yml):

• Add build-native job that builds libcomet.so once and uploads as artifact
• All 36 test jobs download the pre-built library instead of rebuilding
• Rust tests run in parallel with build-native (they use debug builds)

PR Build macOS (pr_build_macos.yml):

• Add build-native job that builds libcomet.dylib once
• All 18 test jobs download the pre-built library instead of rebuilding

Spark SQL Tests (spark_sql_test.yml):

• Add build-native job that builds once
• All ~48 test jobs download the pre-built library instead of rebuilding
• Add skip-native-build input to setup-spark-builder action

Shared Changes:

• Add Cargo caching throughout
• Add faster CI Cargo profile in native/Cargo.toml (no LTO, parallel codegen) with same overflow behavior as release
• Add skip-native-build input to java-test action

Estimated Savings

Workflow	Before	After	Savings
PR Build Linux	36 Rust builds	1	~175-525 min
PR Build macOS	18 Rust builds	1	~85-255 min
Spark SQL Tests	48 Rust builds	1	~235-705 min
Total	102 Rust builds	3	~495-1485 min

Additionally, the CI Cargo profile compiles ~30-50% faster than full release (no LTO).

[hsiang-c]

👍

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.00%. Comparing base (f09f8af) to head (2a7801f).
⚠️ Report is 868 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3249      +/-   ##
============================================
+ Coverage     56.12%   60.00%   +3.87%     
- Complexity      976     1428     +452     
============================================
  Files           119      170      +51     
  Lines         11743    15796    +4053     
  Branches       2251     2608     +357     
============================================
+ Hits           6591     9478    +2887     
- Misses         4012     5000     +988     
- Partials       1140     1318     +178     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wForget]

Can we use ${{ runner.os }}-cargo-ci- generated by build-native here?

[andygrove]

The linux-test-rust job uses debug builds (cargo test) while build-native uses the CI profile, so their native/target artifacts are incompatible. Sharing the same cache key would cause cache thrashing, where each job overwrites the other's compiled artifacts, if I'm understanding this correctly.

[wForget]

Thanks @andygrove , this change looks good to me.

The cache storage is approaching its limit: https://github.com/apache/datafusion-comet/actions/caches

[andygrove]

Thanks @andygrove , this change looks good to me.

The cache storage is approaching its limit: https://github.com/apache/datafusion-comet/actions/caches

Thanks. #3251 will help with this.