Skip to content

Conversation

@jgallimore
Copy link
Member

No description provided.

jbonofre and others added 30 commits March 7, 2025 14:27
(cherry picked from commit 4b22a4c)
(cherry picked from commit be361fd)
…pache#1419)

This commit makes several improvements and fixes for syncing durable
subscriptions when a network bridge connects.

1) A bug was fixed during durable sync that would cause the clientId
to not always be included for durables in the subscription list which
could cause a loop to be created as the other broker would not be able
to tell where the network subscription came from.

2) During reactivation when dynamicOnly is false and durable sync is set
to true, we make sure to include the TTL information (full broker path)
from the online consumer attached to durables so that TTL info is properly
 propagated so we don't incorrectly create demand. Thisonly works if
consumers are online, so for TTL > 1 it is still recommended to set
dynamicOnly to true and allow only online consumers drive demand.

3) For TTL 1, we can handle sync correctly on restarts even if durables
are offline and missing consumer TTL info because we know that we should
ignore proxy durables (bridge durables for other bridges) entirely because
they will be > 1 hop away.

4) Some other minor improvements were made like filtering everything if
TTL is 0 and also consolidating logic.

(cherry picked from commit 953737c)
(cherry picked from commit 182c598)
This commit fixes multiple problems with handling message expiration on
durable topic subscriptions.

1) Memory usage tracking is fixed on expiration by correctly
decrementing the counter inside AbstractStoreCursor when calling the
remove(message) method, which was previously missed.
2) A new refrence type is used to wrap references in TopicStorePrefetch
so that if multiple subscriptions share a reference in their cursors
each one can expire the message. Previously only one would expire as the
message would be marked as expired and skipped.
3) On client expiration, the references are properly decremented so
memory tracking is correct.
4) The expiration thread for Topics has been improved to be much more
efficient for KahaDB by only scanning for expired messages if there are
durables eligible for expiration. The thread also now checks the index
to see if expired messages are associated with the subs still so we
don't expire the same sub multiple times. Only messages that need to
still be processed are returned which further cuts down memory usage.

(cherry picked from commit 4abcfa1)
(cherry picked from commit 9a1f00b)
This allows the store to check if the memory usage is full in the system
before continuing to try and load messages to expire on durable subs

(cherry picked from commit 6dac231)
apache#1447)

This fixes the broker so multiple removals are no longer done for the
same message leading to having to search the entire non persistent
pending list. Durable subscriptions now check the persistence type of
the message so the cursor will no longer search everything in a
non-persistent pending list when the message is persistent.

(cherry picked from commit 5a3abbc)
This fixes the test by correctly waiting for expiration and greatly
speeds things up by reducing how many messages are sent

(cherry picked from commit 6581ed7)
This fixes the clear() method so that when clearing the memory map it
will decrement memory usage, and when clearing the disk list it
will destroy and reset the list for future writes.

(cherry picked from commit b1e8441)
…che#1474)

Update the KahaDB scheduled tasks to catch any runtime exceptions thrown
by the configured IOExceptionHandler. This will prevent the tasks from
being killed and no longer running if the IOExceptionHandler does
not shut down the broker.

(cherry picked from commit 2eb0d66)
(cherry picked from commit 98611c2)
(cherry picked from commit e2c1a1a)
- Auto-resolved pom.xml conflicts:
  - Prefer incoming version bumps (e.g., 5.19.1)
  - Preserved local SCM and distributionManagement details
Copy link
Member

@cesarhernandezgt cesarhernandezgt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 , I ran both base and PR CI jobs and PR is good to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants