Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Watermark Implementation for Async Data Movement #2376

Merged
merged 37 commits into from
Jan 31, 2025
Merged

feat: Watermark Implementation for Async Data Movement #2376

merged 37 commits into from
Jan 31, 2025

Conversation

yhl25
Copy link
Contributor

@yhl25 yhl25 commented Jan 30, 2025

fixes #2365

yhl25 and others added 30 commits January 4, 2025 18:21
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Vigith Maurice <[email protected]>
Signed-off-by: Vigith Maurice <[email protected]>
Signed-off-by: Vigith Maurice <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Vigith Maurice <[email protected]>
Signed-off-by: Vigith Maurice <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Signed-off-by: Yashash H L <[email protected]>
Copy link

codecov bot commented Jan 30, 2025

Codecov Report

Attention: Patch coverage is 92.89204% with 291 lines in your changes missing coverage. Please review.

Project coverage is 69.71%. Comparing base (d4e5859) to head (4548fa3).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...t/numaflow-core/src/watermark/processor/manager.rs 82.45% 80 Missing ⚠️
rust/numaflow-core/src/watermark/isb/wm_fetcher.rs 92.94% 54 Missing ⚠️
...numaflow-core/src/pipeline/isb/jetstream/writer.rs 81.67% 35 Missing ⚠️
rust/numaflow-core/src/pipeline.rs 83.12% 27 Missing ⚠️
rust/numaflow-core/src/config/pipeline.rs 91.70% 17 Missing ⚠️
rust/numaflow-core/src/tracker.rs 85.36% 12 Missing ⚠️
rust/numaflow-core/src/message.rs 74.41% 11 Missing ⚠️
...numaflow-core/src/pipeline/isb/jetstream/reader.rs 89.00% 11 Missing ⚠️
rust/numaflow-core/src/watermark/isb.rs 97.48% 11 Missing ⚠️
.../numaflow-core/src/watermark/processor/timeline.rs 96.59% 10 Missing ⚠️
... and 9 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2376      +/-   ##
==========================================
+ Coverage   68.14%   69.71%   +1.56%     
==========================================
  Files         352      361       +9     
  Lines       46527    49934    +3407     
==========================================
+ Hits        31708    34813    +3105     
- Misses      13737    14045     +308     
+ Partials     1082     1076       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Vigith Maurice <[email protected]>
@vigith vigith marked this pull request as ready for review January 31, 2025 05:02
@vigith vigith requested review from whynowy and vigith as code owners January 31, 2025 05:02
Signed-off-by: Vigith Maurice <[email protected]>
rust/Cargo.toml Outdated
Comment on lines 43 to 44
#[profile.release]
#lto = "fat"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Comment on lines 69 to 70
pipeline_name: "default-pl",
vertex_name: "default-vtx",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not empty string?

Signed-off-by: Yashash H L <[email protected]>
@yhl25 yhl25 enabled auto-merge (squash) January 31, 2025 05:58
@yhl25 yhl25 merged commit 084be89 into main Jan 31, 2025
28 checks passed
@yhl25 yhl25 deleted the watermark branch January 31, 2025 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Watermark changes to support async data movement
2 participants