Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: initial rule engine + memo table #22

Merged
merged 47 commits into from
Feb 10, 2025
Merged

feat: initial rule engine + memo table #22

merged 47 commits into from
Feb 10, 2025

Conversation

AlSchlo
Copy link
Collaborator

@AlSchlo AlSchlo commented Feb 5, 2025

Problem

The persistent memo layer and the rule engine are the core components that facilitate query plan transformation.

Having an initial setup is important for people to work on the same codebase simultaneously.

Summary of changes

  • A memo table implementation backed by SQLite.
  • Low-level IR for the rule engine proof concept.

@codecov-commenter
Copy link

codecov-commenter commented Feb 5, 2025

Codecov Report

Attention: Patch coverage is 70.65391% with 368 lines in your changes missing coverage. Please review.

Project coverage is 70.6%. Comparing base (3962965) to head (758671f).

Files with missing lines Patch % Lines
optd-core/src/storage/memo.rs 81.9% 91 Missing ⚠️
optd-core/src/operators/relational/physical/mod.rs 0.0% 76 Missing ⚠️
...-core/src/engine/interpreter/expressions/values.rs 75.7% 32 Missing ⚠️
...c/operators/relational/physical/join/merge_join.rs 0.0% 26 Missing ⚠️
...rc/operators/relational/physical/join/hash_join.rs 0.0% 21 Missing ⚠️
...d-core/src/engine/interpreter/expressions/plans.rs 0.0% 18 Missing ⚠️
...ators/relational/physical/join/nested_loop_join.rs 0.0% 16 Missing ⚠️
optd-core/src/operators/scalar/mod.rs 69.5% 14 Missing ⚠️
optd-core/src/operators/relational/logical/mod.rs 78.5% 12 Missing ⚠️
...c/operators/relational/physical/scan/table_scan.rs 0.0% 12 Missing ⚠️
... and 10 more
Additional details and impacted files
Files with missing lines Coverage Δ
optd-core/src/test_utils.rs 100.0% <100.0%> (ø)
optd-core/src/cascades/mod.rs 99.2% <99.2%> (ø)
optd-core/src/operators/scalar/add.rs 50.0% <50.0%> (ø)
optd-core/src/operators/scalar/column_ref.rs 62.5% <62.5%> (ø)
optd-core/src/operators/scalar/constants.rs 50.0% <50.0%> (ø)
optd-core/src/operators/scalar/equal.rs 50.0% <50.0%> (ø)
...td-core/src/operators/relational/logical/filter.rs 33.3% <33.3%> (ø)
optd-core/src/operators/relational/logical/scan.rs 50.0% <50.0%> (ø)
optd-core/src/operators/relational/logical/join.rs 50.0% <50.0%> (ø)
optd-core/src/storage/transaction.rs 91.5% <91.5%> (ø)
... and 11 more

AlSchlo and others added 4 commits February 4, 2025 23:38
Signed-off-by: Yuchen Liang <[email protected]>
Signed-off-by: Yuchen Liang <[email protected]>
@yliang412 yliang412 changed the title Alexis/rule engine (DRAFT) feat: initial rule engine + memo table Feb 8, 2025
@yliang412 yliang412 marked this pull request as ready for review February 8, 2025 18:19
@AlSchlo AlSchlo merged commit a94829a into main Feb 10, 2025
12 checks passed
@AlSchlo AlSchlo deleted the alexis/rule_engine branch February 10, 2025 00:40
yliang412 added a commit that referenced this pull request Feb 14, 2025
## Problem

With the initial representation and storage added in #4 and #22, we now
want to support the full pipeline going from parsing SQL, optimizing the
plan using optd, and executing the query in Datafusion.

## Summary of changes

- Integrate all @SarveshOO7's good work in
#10
- Added one mock physical implementation rule + operator for each
logical operator
- Refactor scalar operator storage and reduce code bloat.
- Add physical storage tables and memo API.
- Bump MSRV to 1.81.0 to be compatible with datafusion 45.0.0:
apache/datafusion#14330

---------

Signed-off-by: Yuchen Liang <[email protected]>
Co-authored-by: SarveshOO7 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants