Skip to content

LLM judge infra improvements #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pjin-nvidia wants to merge 26 commits into
base: main
Choose a base branch
from
Open

LLM judge infra improvements #258

pjin-nvidia wants to merge 26 commits into from

Conversation

@pjin-nvidia
Copy link
Contributor

@pjin-nvidia pjin-nvidia commented Oct 30, 2025

No description provided.

@pjin-nvidia
Infra improvements for equivalence LLM judge.
900b30d 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
No concurrency limit by default.
04a131f 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@copy-pr-bot
Copy link

copy-pr-bot bot commented Oct 30, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pjin-nvidia pjin-nvidia changed the title [WIP] Equivalence LLM judge infra improvements Equivalence LLM judge infra improvements Oct 30, 2025
@pjin-nvidia
Judge prompt template file.
ef3f9b0 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Set simple agent max steps = 1.
3ca052d 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Fix pydantic validation.
617f564 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/equiv-judge-infra
b9828af 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Refactor wrapped server client post w/ an empty openai response.
cbd9e48 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Set default prompt template file path.
03e5f90 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Dummy model.
c1f608d 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Same response id as VLLM.
4032509 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/equiv-judge-infra
c8891cd 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia pjin-nvidia changed the title Equivalence LLM judge infra improvements LLM judge infra improvements Nov 13, 2025
@pjin-nvidia pjin-nvidia marked this pull request as ready for review November 13, 2025 14:55
@pjin-nvidia
Prefer model_post_init.
7a10196 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Add test.
7f4c392 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Unset max_steps.
d679229 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Fix test.
90481b4 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Move the POST wrapping back into the judge server.
b3df997 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Fix test.
4d612b9 
Signed-off-by: Peter Jin <pjin@nvidia.com>
bxyu-nvidia
bxyu-nvidia reviewed Nov 13, 2025
View reviewed changes
@pjin-nvidia
Create an empty response, optionally providing responses create param…
32ec31d 
…s to fill the rseponse.

Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Refactor.
7dbd1e4 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Lint.
2b1c3a5 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Pydantic. Add test assert.
b952c5a 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Test.
18d0ea1 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Test.
3549881 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Test.
9dd719a 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Using empty_response flow with responses create params.
93af881 
Signed-off-by: Peter Jin <pjin@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bxyu-nvidia bxyu-nvidia bxyu-nvidia approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pjin-nvidia @bxyu-nvidia