Fix(stream): fix ci case issues

[JinqingKuang]

Description

Fix(stream): fix ci case issues

Issue(s)

• Fix: https://project.feishu.cn/taosdata_td/defect/detail/6495889818
• Fix: https://project.feishu.cn/taosdata_td/defect/detail/6600066555
• Fix: https://project.feishu.cn/taosdata_td/defect/detail/6635280371
• Fix: https://project.feishu.cn/taosdata_td/defect/detail/6644387794
• Fix: https://project.feishu.cn/taosdata_td/defect/detail/6740517318

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello @JinqingKuang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on resolving several CI-related issues within the stream processing module. It introduces critical fixes for how stream version timestamps are managed and communicated, especially in scenarios where WAL logs are unavailable or when no data is returned. The changes also refine the complex logic governing sliding window stream triggers and the processing of historical data. Complementing these core fixes, the PR includes substantial updates to the test suite, adding new verification scenarios and improving existing test utilities to ensure the robustness and correctness of stream functionalities.

Highlights

• Stream Version Timestamp Handling: The verTime field in stream messages is now consistently set and transmitted in microseconds, improving precision and consistency in stream processing, especially when WAL logs are not found or when no data is returned.
• Sliding Window Trigger Logic Refinement: Enhanced the calculation of end keys and handling of partial data for sliding window stream triggers, addressing potential inaccuracies in historical data processing and calcRange management.
• Expanded Stream Test Coverage: Significantly increased test coverage for stream processing, particularly for stream6 in the idmp_vehicle use case, by adding new trigger and detailed verification methods.
• Test Framework Improvements: Introduced a new utility function checkHaveSameResult in the test framework to simplify and standardize the comparison of SQL query results and their memory usage in tests.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• include/common/streamMsg.h
  • Added a comment to the verTime field to explicitly state its unit as microseconds.
• source/dnode/vnode/src/vnd/vnodeStream.c
  • Modified processWalVerMetaNew and prepareIndexMetaData to set verTime to 0 when ver is reset, or to the current timestamp in microseconds when WAL logs are not found.
  • Updated vnodeProcessStreamWalMetaNewReq and vnodeProcessStreamWalMetaDataNewReq to include verTime in RPC responses when TSDB_CODE_STREAM_NO_DATA is returned.
• source/libs/new-stream/src/streamTriggerTask.c
  • Adjusted stTriggerTaskAddRecalcRequest to correctly determine scanRange.ekey for sliding window triggers with interval > sliding.
  • Changed stTriggerTaskGetDelay to use microseconds for now and latestVersionTime in delay calculations.
  • Modified stRealtimeContextProcPullRsp to handle RPC responses that may contain both ver and verTime.
  • Implemented new logic in stHistoryGroupAddCalcParam to ignore partial data for specific sliding window triggers and updated finishTs calculation.
• test/cases/18-StreamProcessing/02-Stream/test_stream_check_name.py
  • Replaced a custom loop for checking stream running status with the tdStream.checkStreamStatus() utility.
• test/cases/18-StreamProcessing/20-UseCase/test_idmp_vehicle.py
  • Integrated new verification calls (verifyA_stream6_sub1, triggerB_stream6, verifyB_stream6, verifyB_stream6_sub1) into the idmp_vehicle test suite.
  • Added detailed triggerA_stream6, verifyA_stream6, verifyA_stream6_sub1, triggerB_stream6, verifyB_stream6, verifyB_stream6_sub1 methods to expand test cases for stream6.
  • Updated verifyA_stream5 and verifyA_stream7 to utilize the new tdSql.checkHaveSameResult method for result comparison.
• test/ci/cases.task
  • Removed the 90,, prefix from the test_tsma.py entry, adjusting its CI execution configuration.
• test/new_test_framework/utils/sql.py
  • Added a new method checkHaveSameResult to TDSql for comparing the results of two SQL queries and their memory usage.

Activity

• The pull request addresses multiple identified issues, as indicated by the linked defect IDs in the description.
• Automated test cases have been executed and passed, confirming the stability of the changes.
• Test coverage has been maintained, ensuring no regressions were introduced.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces several fixes for stream processing, primarily to resolve CI test case failures. The main changes include adding a verTime field to track version timestamps in RPC responses, correcting logic for sliding window calculations, and updating boundary checks in history processing. The changes are logical and address specific bugs. I have a couple of suggestions to improve code readability and maintainability by replacing pointer arithmetic with array-style indexing and addressing code duplication.

[Copilot]

Pull request overview

This pull request fixes multiple CI test case issues related to stream processing functionality. The changes address timing synchronization bugs, add comprehensive test coverage for stream6 scenarios, and fix critical logic errors in stream trigger task handling.

Changes:

• Fixed timestamp unit handling from nanoseconds to microseconds for stream delay calculation
• Added verTime (microsecond timestamp) tracking in stream WAL responses for better synchronization
• Enhanced test coverage with new stream6 verification methods and improved test organization
• Fixed window parameter comparison logic to correctly skip windows outside the calculation range
• Refactored stream status checking to use centralized utility function

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/new_test_framework/utils/sql.py	Added checkHaveSameResult helper method to verify two SQL queries match expected data
test/ci/cases.task	Removed priority override for test_tsma.py test case
test/cases/18-StreamProcessing/20-UseCase/test_idmp_vehicle.py	Added comprehensive stream6 trigger/verify methods for both phase A and B, cleaned up comments
test/cases/18-StreamProcessing/02-Stream/test_stream_check_name.py	Refactored to use centralized tdStream.checkStreamStatus() utility
source/libs/new-stream/src/streamTriggerTask.c	Fixed timestamp conversion from ns to us, added sliding window end handling, fixed window parameter comparison logic, added partial data filtering
source/dnode/vnode/src/vnd/vnodeStream.c	Added verTime initialization and tracking, updated response size to include both ver and verTime
include/common/streamMsg.h	Added clarifying comment that verTime is in microseconds

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.