Better handling of batching of search in MG replicated mode #1718
+26
−21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-project-automation
bot
moved this to Todo
in Vector Search, ML, & Data Mining Release Board
cjnolet
added
bug
cjnolet
moved this from Todo
to In Progress
in Vector Search, ML, & Data Mining Release Board
Contributor
|
I would expect CAGRA to be thread safe. We expect that we can search using multiple threads to achieve large throughput for concurrent small batch size queries. We should fix this in CAGRA. |
tfeher
approved these changes
Jan 21, 2026
Contributor
tfeher
left a comment
tfeher
left a comment
There was a problem hiding this comment.
The changes look good to me as a workaround. But we should find the root cause, please open an issue to track it.
Member
|
@viclafargue cam you also link the issue in a comment in the code before this is merged? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Answers #1720
In multi-GPU replicated mode, the search query is divided in batches. These batches are ran in parallel with OpenMP. In some cases, there may be more batches than available GPUs causing a thread safety issue (at least for CAGRA indices). This change solves the issue. Each rank gets its own thread, that thread handles all batches for that rank sequentially. This prevents concurrent access to the same GPU from multiple threads.