Skip to content

Conversation

@mxm
Copy link
Contributor

@mxm mxm commented Jan 26, 2026

Under high load due to limited CI resources and parallel test execution, the test pipeline can run multiple Flink checkpoints before all records have been processed. This produces a slightly different but not semantically wrong outcome.

In the below reported test failure, there won't be a positional delete because the Flink checkpoint cuts off the writer optimization to convert an equality delete to a positional delete.

TestDynamicIcebergSink > testCommitsOncePerTableBranchAndCheckpoint() FAILED
    java.lang.AssertionError: 
    Expecting map:
      {"added-data-files"="1", "added-delete-files"="1", "added-equality-delete-files"="1", "added-equality-deletes"="1", "added-files-size"="1128", "added-records"="1", "changed-partition-count"="1", "engine-name"="flink", "engine-version"="2.1.0", "flink.job-id"="7a390dfdf54bc3d7028708e2aa46c54c", "flink.max-committed-checkpoint-id"="2", "flink.operator-id"="27651a86dc01e60882f6b181172fad4c", "iceberg-version"="Apache Iceberg aa63060 (commit aa630608f79d06c1c3929801fd7266b082170758)", "total-data-files"="5", "total-delete-files"="2", "total-equality-deletes"="2", "total-files-size"="4478", "total-position-deletes"="0", "total-records"="6"}
    to contain entries:
      ["total-equality-deletes"="1",
        "total-position-deletes"="1",
        "total-records"="6"]
    but the following map entries had different values:
      ["total-equality-deletes"="2" (expected: "1"), "total-position-deletes"="0" (expected: "1")]
        at org.apache.iceberg.flink.sink.dynamic.TestDynamicIcebergSink.testCommitsOncePerTableBranchAndCheckpoint(TestDynamicIcebergSink.java:1005)

The important part of the test is to ensure comitting once per checkpoint which is asserted further below.

This closes #15139.

mxm added 2 commits January 26, 2026 17:49
See apache#15139

Under high load due to limited CI resources and parallel test execution, the
test pipeline can run multiple Flink checkpoints before all records have been
processed. This produces a slightly different but not semantically wrong outcome.

The important part of the test is to ensure comitting once per checkpoint which
is asserted further below.

This closes apache#15139.
@github-actions github-actions bot added the flink label Jan 26, 2026
Copy link
Contributor

@singhpk234 singhpk234 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me !

Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ty!

@kevinjqliu kevinjqliu merged commit ec93fe2 into apache:main Jan 27, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky Test TestDynamicIcebergSink > testCommitsOncePerTableBranchAndCheckpoint

4 participants