Skip to content

Fix kafkastreamer consumer group race conditions and backpressure handling#175

Open
Copilot wants to merge 5 commits intomainfrom
copilot/review-kafkastreamer-adapter
Open

Fix kafkastreamer consumer group race conditions and backpressure handling#175
Copilot wants to merge 5 commits intomainfrom
copilot/review-kafkastreamer-adapter

Conversation

Copy link
Contributor

Copilot AI commented Feb 13, 2026

The kafkastreamer adapter had race conditions during rebalances and unbuffered channels causing consumer evictions.

Changes

Ready channel race fix

  • ConsumeClaim called multiple times during rebalances → replaced channel send with sync.Once

Iterator backpressure fix

  • Unbuffered iterator channels blocked ConsumeClaim → added 100-element buffer
  • Prevents rebalance timeout evictions under load

Event ID correction

  • Changed from m.Offset + 1 to m.Offset for proper Kafka offset semantics

Code cleanup

  • Removed duplicate unreachable select case in connector.go
  • Renamed fixes_test.goconsumer_group_internal_test.go
// Before: race on rebalance
func (mp *msgProcessor) ConsumeClaim(...) {
    mp.ready <- true  // Multiple sends possible
    ...
}

// After: safe rebalance handling
func (mp *msgProcessor) ConsumeClaim(...) {
    mp.readyOnce.Do(func() { mp.ready <- true })
    ...
}

Applied to both msgProcessor and connectorProcessor. Tests verify concurrent ConsumeClaim calls and buffer capacity.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 3 commits February 13, 2026 16:47
…el race, and event ID

Co-authored-by: andrewwormald <20083774+andrewwormald@users.noreply.github.com>
Co-authored-by: andrewwormald <20083774+andrewwormald@users.noreply.github.com>
…mentation

Co-authored-by: andrewwormald <20083774+andrewwormald@users.noreply.github.com>
Copilot AI changed the title [WIP] Review kafkastreamer adapter for scaling assumptions Fix kafkastreamer adapter scaling issues and Kafka consumer group correctness Feb 13, 2026
Copilot AI requested a review from andrewwormald February 13, 2026 16:56
Co-authored-by: andrewwormald <20083774+andrewwormald@users.noreply.github.com>
Copilot AI changed the title Fix kafkastreamer adapter scaling issues and Kafka consumer group correctness Rename fixes_test.go to consumer_group_internal_test.go Feb 13, 2026
@andrewwormald andrewwormald marked this pull request as ready for review February 16, 2026 12:05
@sonarqubecloud
Copy link

Copilot AI changed the title Rename fixes_test.go to consumer_group_internal_test.go Fix kafkastreamer consumer group race conditions and backpressure handling Feb 16, 2026
Copilot AI requested a review from andrewwormald February 16, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants