Add startup benchmark configs for vLLM

[desertfire]

Summary: Add startup test configurations for all platforms to support vLLM's compilation time benchmarking added in vllm-project/vllm#33183.

[huydhn]

A quick round of benchmark test https://github.com/pytorch/pytorch-integration-testing/actions/runs/21457061378

[desertfire]

A quick round of benchmark test https://github.com/pytorch/pytorch-integration-testing/actions/runs/21457061378

I can see the compilation time json files in the generated artifact.

[huydhn]

Yeah, but I think I need to check their content. A quick query on ClickHouse surprisingly doesn't show what I expect. So, stay tuned

[desertfire]

Yeah, but I think I need to check their content. A quick query on ClickHouse surprisingly doesn't show what I expect. So, stay tuned

Are you saying the compilation time metrics were not stored in a certain format as you expected? https://github.com/vllm-project/vllm/blob/c6e7404cc5713a926e8b6c187b5f197a5436e9ff/vllm/benchmarks/startup.py#L96 is how those metrics are stored. Feel free to twist it as you may need.