Runpod SDK versions > 1.7.10 send all requests to the same worker even with 10 active workers, and 50 max workers set

[ashleykleynhans]

Describe the bug
Runpod SDK 1.7.12 that attempts to fix a bug in version 1.7.11 does fix the initial bug in local testing, but its still broken in Runpod serverless because its sending every single request to the same workers, leading to massive delay times while the requests are waiting for the worker to become available.

To Reproduce
Steps to reproduce the behavior:

1. Create a serverless endpoint that uses Python SDK version 1.7.12.
2. Deploy the endpoint with multiple max workers and some active workers.
3. Send a bunch of concurrent requests.
4. Observe that all requests are being sent to the same worker instead of multiple workers.

Expected behavior
If an endpoint is configured to have multiple max workers and active workers, depending on the queue configuration, new requests should be spread across workers and not all sent to the same worker.

Screenshots

Additional context

Reverting to SDK version 1.7.10 (since 1.7.11 is also broken) resolves the issue.

[ashleykleynhans]

By the way, I believe the versioning for 1.7.11 and 1.7.12 is also incorrect.

There was clearly a major change introduced, so 1.7.11 should have actually been version 1.8.0, and version 1.7.12 should be 1.8.1.

Point releases are bug fix/patch releases and should not contain any major/potentially breaking changes. They should be safe to deploy without worrying about breaking anything, whereas if they had been major releases, users would be aware that they should be weary and spend more time testing.

Please review https://semver.org/ which is best practice for versioning.

[justinwlin]

Thanks for the ticket! On the repro of the issue we had three people unable to reproduce. Maybe if you have a different reproduction, feel free lmk (in case maybe I didn't repro right) and I'll continue to look deeper / am still testing to make sure we didn't miss anything.

I will leave this ticket open till we're sure that this is resolved and in case others report similarly.

Attached is steps to my reproduction steps:

Investigation

Code used on my end to test the reproduction:

https://github.com/justinwlin/Runpod-1.7.12-Python-SDK-Repro/blob/main/handler.py

The setup that we did for reproduction was:

0 min workers, 10 max workers
Concurrency: lambda x: 1 and 3 (testing nonconcurrent and concurrent workload)
20 second delay on the job as a pseudo-processing time.

Simulated a curl command to the endpoint 10 back to back requests:

for i in {1..10}; do curl -s -X POST https://api.runpod.ai/v2/00v60a3087cgph/run \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer xxx' \
  -d '{"input":{"prompt":"Your prompt"}}'; echo; done

Base Line on 1.7.10:

Acts as expected, cycling through workers as previous status-quo.

Testing on 1.7.12, concurrency test of 3, 20 second delay:

(Testing through the console)
Workers act as expected, in groups of threes, with a 20 second execution time, with 1.7.12 being logged.

(Testing through bash shell command, picking up the jobs in groups of 3)

Tested on 1.7.12, concurrency of 1, 20 second delay:

(Tested through bash shell command)
We can see proper cycling through workers behavior:

[weihzh]

not exactly the same issue, but my docker somehow stops working (stop, remove container) before it even reaches runpod.serverless.start.
Downgrading the version to 1.7.0 did work, so some major changes definitely happened between these two versions.

[justinwlin]

@weihzh Do you have any repository that maybe I can see if I can repro on? Thanks!

[ashleykleynhans]

This issue is still present. The only version that works for me is 1.7.10.
Even 1.8.1 sends the requests all to the same worker, even with 10 active workers.
All requests wait in the queue while waiting for the other requests on the ONE worker to complete.

Jobs stuck in queue:

Same worker used:

Active workers running but not being used:

This happens regardless of whether I do a new deployment to change the docker image, or whether I scale all of my workers down and back up again.

[ashleykleynhans]

@justinmerrell I do not have permission to reopen the issue, please could you reopen it for me?

[ashleykleynhans]

I had to downgrade the SDK back to version 1.7.10 to resolve the issue once again.

[ashleykleynhans]

Could the issue perhaps be related to the fact that I am using network storage on my endpoint?

My endpoint is 100% fine and processing requests as expected after downgrading the SDK to version 1.7.10, so there is definitely some kind of issue with newer versions. I am surprised other users have not experienced the same issue.

My endpoint id is 1d6w6kzsx42qpm if it helps. It uses the default scaling policy.