feat(fitness): comprehensive system fitness checks for worker startup validation

[deanq]

Summary

Implemented comprehensive fitness check system that validates worker readiness at startup. Includes 7 built-in system checks (memory, disk, network, GPU) plus a simple API for users to register custom checks. All checks provide clear visibility in production logs with intelligent, scalable validation thresholds.

Custom User-Defined Checks

Users can register custom fitness checks using a simple decorator:

import runpod

@runpod.serverless.register_fitness_check
def check_model_files():
    """Verify required model files exist."""
    from pathlib import Path
    model_path = Path("/models/my-model.safetensors")
    if not model_path.exists():
        raise RuntimeError(f"Model not found: {model_path}")

@runpod.serverless.register_fitness_check
async def check_external_api():
    """Verify external API is accessible."""
    import aiohttp
    async with aiohttp.ClientSession() as session:
        async with session.get("https://api.example.com/health", timeout=5) as resp:
            if resp.status != 200:
                raise RuntimeError(f"API health check failed: {resp.status}")

def handler(job):
    return {"output": "success"}

runpod.serverless.start({"handler": handler})

Key Features:

• Simple decorator: @runpod.serverless.register_fitness_check
• Supports both sync and async functions
• User checks run before built-in system checks
• Raise exceptions on failure (no return value needed)

Documentation: Complete guide at docs/serverless/worker_fitness_checks.md with examples for GPU validation, file checks, environment variables, and external service connectivity.

System Fitness Checks

All Workers (3 checks):

1. Memory Check - Validates sufficient RAM available

   • Default: 4GB minimum
   • Configurable: `RUNPOD_MIN_MEMORY_GB`

2. Disk Space Check - Validates adequate disk space

   • Default: 10% of total disk must be free
   • Automatically scales with machine size
   • Configurable: `RUNPOD_MIN_DISK_PERCENT`

3. Network Connectivity Check - Validates internet access

   • Default: 5s timeout to 8.8.8.8:53
   • Configurable: `RUNPOD_NETWORK_CHECK_TIMEOUT`

GPU Workers (4 additional checks):
4. CUDA Version Check - Validates CUDA driver version

• Default: CUDA 11.8+
• Configurable: `RUNPOD_MIN_CUDA_VERSION`

5. CUDA Initialization Check - Verifies GPU accessibility

   • Tests actual device initialization
   • Validates each GPU's memory allocation
   • Catches runtime failures early

6. GPU Compute Benchmark - Quick performance validation

   • Matrix multiplication test
   • Default: 100ms max execution time
   • Configurable: `RUNPOD_GPU_BENCHMARK_TIMEOUT`

7. GPU Binary Test - Comprehensive GPU health via native binary

   • CUDA driver availability
   • NVML initialization
   • Memory allocation per GPU

Behavior

• Execution: Runs once at worker startup, before accepting jobs
• Failure: Worker exits immediately with code 1, marked unhealthy
• Success: All checks pass, worker begins accepting jobs
• CPU Workers: GPU checks skip silently (same code works for CPU/GPU)

Example Production Logs

GPU Worker Startup:
```
[info]--- Starting Serverless Worker | Version 1.8.2.dev ---
[info]Running 7 fitness check(s)...
[info]Memory check passed: 181.49GB available (of 187.85GB total)
[info]Disk space check passed: 9.88GB free (98.8% available)
[info]Network connectivity passed: Connected to 8.8.8.8 (23ms)
[info]CUDA version check passed: 12.7 (minimum: 11.8)
[info]CUDA initialization passed: 2 device(s) initialized successfully
[info]GPU compute benchmark passed: Matrix multiply completed in 42ms
[info]GPU binary test passed: 2 GPU(s) healthy (CUDA 12.7)
[info]All fitness checks passed.
```

CPU Worker Startup:
```
[info]--- Starting Serverless Worker | Version 1.8.2.dev ---
[info]Running 3 fitness check(s)...
[info]Memory check passed: 359.41GB available (of 377.18GB total)
[info]Disk space check passed: 9.88GB free (98.8% available)
[info]Network connectivity passed: Connected to 8.8.8.8 (23ms)
[info]All fitness checks passed.
```

Configuration

All thresholds are configurable via environment variables:

```dockerfile

Dockerfile

ENV RUNPOD_MIN_MEMORY_GB=8.0
ENV RUNPOD_MIN_DISK_PERCENT=15.0
ENV RUNPOD_MIN_CUDA_VERSION=12.0
ENV RUNPOD_NETWORK_CHECK_TIMEOUT=10
ENV RUNPOD_GPU_BENCHMARK_TIMEOUT=2
```

Or in Python:
```python
import os
os.environ["RUNPOD_MIN_MEMORY_GB"] = "8.0"
os.environ["RUNPOD_MIN_DISK_PERCENT"] = "15.0"
```

Test Coverage

• Memory availability detection
• Disk space validation across different disk sizes
• Network connectivity testing
• CUDA version parsing and comparison
• GPU device initialization verification
• GPU compute performance benchmarking
• Edge cases for cross-platform compatibility

Implementation Details

• Auto-registers checks at worker startup
• Deferred registration avoids circular imports
• Graceful degradation on missing tools
• Comprehensive error messages for debugging
• Platform-aware (Linux /proc/meminfo fallback for memory)

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[Copilot]

Pull request overview

This PR implements a comprehensive fitness check system for worker startup validation, consisting of user-defined custom checks and 7 built-in system resource checks (memory, disk, network, and GPU-specific validations). The system validates worker readiness before accepting jobs, with intelligent thresholds that scale automatically and clear production logging.

Key Changes:

• Added user-facing decorator API (@runpod.serverless.register_fitness_check) for custom health checks
• Implemented 7 automatic system checks (3 for all workers, 4 additional for GPU workers)
• Included GPU test binary with native CUDA memory allocation validation
• Comprehensive documentation and examples for user-defined checks

Reviewed changes

Copilot reviewed 28 out of 31 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
runpod/serverless/modules/rp_fitness.py	Core fitness check registration and execution system
runpod/serverless/modules/rp_system_fitness.py	Built-in system resource checks (memory, disk, network, CUDA)
runpod/serverless/modules/rp_gpu_fitness.py	GPU-specific health checks using native binary and fallback
runpod/serverless/__init__.py	Exported register_fitness_check to public API
runpod/serverless/worker.py	Integrated fitness checks into worker startup flow
docs/serverless/worker_fitness_checks.md	Complete user documentation with examples
tests/test_serverless/test_modules/test_fitness.py	Unit tests for core fitness system
tests/test_serverless/test_modules/test_system_fitness.py	Unit tests for system resource checks
tests/test_serverless/test_modules/test_gpu_fitness.py	Unit tests for GPU check system
build_tools/gpu_test.c	Native CUDA binary source for GPU memory testing

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 31 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jhcipar]

Maybe not something for right now, but I feel like it'd be nice to have some instrumentation for timing with these - like out of the box, debug logs will display how long each fitness check took.