PS-10245 feature: Implement receiving binlog events in GTID mode (part 5) #86
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t 5) https://perconadev.atlassian.net/browse/PS-10245 GTID-based replication mode is now supported. Along with writing data to binlog files 'binsrv::storage' class now also creates and maintains binlog file metadata - separate files that follows the '<binlog_file_name>.json' naming convention and store actual flushed file size (in both position-based and GTID-based replication modes) and accumulated GTID set (only in GTID-based replication mode). In addition, upon storage initialization, we now restore the GTID set from the latest binlog metadata file in order to continue GTID-based replication from this set. Implemented additional logic that discards events from the latest incomplete transaction from the storage buffer upon MySQL server disconnect. Main application extended with additional diagnostic messages informing about binlog files creation / opening / closing / reusing. In GTID mode 'binsrv::storage' class now also keeps track of the GTIDs associated with processed events that helps with updating binlog metadata files. 'binsrv::storage' constructor now performs additional checks to ensure that each file in the binlog index has a corresponding binlog file metadata ('<binlog_file_name>.json') and the size from that metadata file matches the actual binlog file size. 'binsrv::basic_storage_backend::open_stream()' method now returns the size of the opened file or 0 if the file was reated. Concreate implementations ('binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend') updated correspondingly. 'binsrv::storage::open_binlog()' method now returns extra info about file creating / opening: - created - opened_empty - opened_at_magic_paylod_offset - opened_with_data_present This helps with making a decision which events 'binsrv::event::reader_context' should expect next. 'binsrv::event::reader_context' extended with additional logic that helps to determine if the processed event is "info-only" and should not be written to the binlog files (this concerns artificial RORATE, FORMAT_DESCRIPTION and PREVIOUS_GTID_LOG events). Improved 'util::nv_tuple_to_json()' function - we now create keys for optional objects only when they are non-empty. Added new 'binlog_streaming.resume_streaming' MTR test case that checks various continuation scenarios bot the Binlog Server utility (starting from an empty storage, starting from a file that has just magic payload in it, receiving ROTATE as the last event, etc.). The testcase has 4 combinations for position-based / GTID-based replication in buffered / unbuffered (immediately flushing) mode. It is now possible to specify to set '$binsrv_replication_mode' (to either 'position' or 'gtid') before including the 'set_up_binsrv_environment.inc' MTR include file to set desired Binlog Server utility replication mode.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://perconadev.atlassian.net/browse/PS-10245
GTID-based replication mode is now supported.
Along with writing data to binlog files 'binsrv::storage' class now also creates and maintains binlog file metadata - separate files that follows the '<binlog_file_name>.json' naming convention and store actual flushed file size (in both position-based and GTID-based replication modes) and accumulated GTID set (only in GTID-based replication mode). In addition, upon storage initialization, we now restore the GTID set from the latest binlog metadata file in order to continue GTID-based replication from this set.
Implemented additional logic that discards events from the latest incomplete transaction from the storage buffer upon MySQL server disconnect.
Main application extended with additional diagnostic messages informing about binlog files creation / opening / closing / reusing.
In GTID mode 'binsrv::storage' class now also keeps track of the GTIDs associated with processed events that helps with updating binlog metadata files.
'binsrv::storage' constructor now performs additional checks to ensure that each file in the binlog index has a corresponding binlog file metadata ('<binlog_file_name>.json') and the size from that metadata file matches the actual binlog file size.
'binsrv::basic_storage_backend::open_stream()' method now returns the size of the opened file or 0 if the file was reated. Concreate implementations ('binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend') updated correspondingly.
'binsrv::storage::open_binlog()' method now returns extra info about file creating / opening:
'binsrv::event::reader_context' should expect next.
'binsrv::event::reader_context' extended with additional logic that helps to determine if the processed event is "info-only" and should not be written to the binlog files (this concerns artificial RORATE, FORMAT_DESCRIPTION and PREVIOUS_GTID_LOG
events).
Improved 'util::nv_tuple_to_json()' function - we now create keys for optional objects only when they are non-empty.
Added new 'binlog_streaming.resume_streaming' MTR test case that checks various continuation scenarios bot the Binlog Server utility (starting from an empty storage, starting from a file that has just magic payload in it, receiving ROTATE as the last event, etc.). The testcase has 4 combinations for position-based / GTID-based replication in buffered / unbuffered (immediately flushing) mode.
It is now possible to specify to set '$binsrv_replication_mode' (to either 'position' or 'gtid') before including the 'set_up_binsrv_environment.inc' MTR include file to set desired Binlog Server utility replication mode.