feat(core): new task graph without rules and merging #55

yliang412 · 2025-04-05T06:24:38Z

Problem

We need to adapt the optimizer implementation to use the new task graph design.

Summary of changes

Added task creation and ensuring logic.
Added an in-memory memo implementation, with certain operations mocked.
- The memo merge is naive and does not trigger recursive merges.
Added new client interface and showcase inter-query optimization.
Added tests that optimize based on a preloaded memo + 0 rules.

Future work

Merging is still WIP.

codecov-commenter · 2025-04-05T06:26:20Z

Codecov Report

Attention: Patch coverage is 38.64900% with 1435 lines in your changes missing coverage. Please review.

Project coverage is 79.2%. Comparing base (07f8873) to head (15ec949).

Files with missing lines	Patch %	Lines
optd-core/src/optimizer/merge.rs	0.0%	342 Missing ⚠️
optd-core/src/memo/memory.rs	53.1%	259 Missing ⚠️
optd-core/src/optimizer/tasks/mod.rs	7.6%	144 Missing ⚠️
optd-core/src/optimizer/jobs.rs	2.8%	103 Missing ⚠️
optd-core/src/optimizer/tasks/implement_expr.rs	0.0%	86 Missing ⚠️
optd-core/src/optimizer/handlers.rs	26.9%	84 Missing ⚠️
optd-core/src/optimizer/tasks/transform_expr.rs	0.0%	77 Missing ⚠️
optd-core/src/optimizer/tasks/fork_costed.rs	0.0%	45 Missing ⚠️
...d-core/src/optimizer/tasks/continue_with_costed.rs	0.0%	42 Missing ⚠️
optd-core/src/optimizer/tasks/fork_logical.rs	0.0%	42 Missing ⚠️
... and 11 more

Additional details and impacted files

Files with missing lines	Coverage Δ
optd-core/src/cir/operators.rs	`100.0% <100.0%> (ø)`
optd-core/src/cir/rules.rs	`100.0% <ø> (+100.0%)`	⬆️
optd-core/src/memo/merge_repr.rs	`98.6% <100.0%> (ø)`
optd-core/src/optimizer/tasks/optimize_plan.rs	`100.0% <100.0%> (ø)`
optd-dsl/src/analyzer/map.rs	`96.5% <ø> (ø)`
optd-core/src/optimizer/egest.rs	`33.5% <0.0%> (+33.5%)`	⬆️
optd-core/src/memo/mod.rs	`53.8% <53.8%> (ø)`
optd-core/src/optimizer/client.rs	`88.8% <88.8%> (ø)`
optd-core/src/error.rs	`0.0% <0.0%> (ø)`
optd-core/src/optimizer/tasks/explore_group.rs	`83.3% <83.3%> (ø)`
... and 16 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Yuchen Liang <[email protected]>

Tasks are objectives. They might launch jobs that do actual work Signed-off-by: Yuchen Liang <[email protected]>

Co-authored-by: Yuchen Liang <[email protected]>

Signed-off-by: Yuchen Liang <[email protected]>

…ember` Signed-off-by: Yuchen Liang <[email protected]>

Signed-off-by: Yuchen Liang <[email protected]>

… removed task graph

## Problem To facilitate integration of the long-standing PR #55, this PR deletes the entire optimizer module so that we can slowly add stuff back in.

… dependencies

## Problem Some of the names in `mod.rs` are super confusing. ## Summary of changes Changes the names of several of the types. Look at `memo/mod.rs` for the changes, everything else is just autorenamed.

This will now allow us to merge the code much more easily into main. Furthermore, it will allow us to test the code using the DSL and it's compiled HIR.

fixed all clippy warnings except for one

connortsui20 · 2025-05-02T15:57:36Z

Closing while @SarveshOO7 and @yliang412 work on testing. Once that is done, we can make a new PR that will be easier to review.

DONT DELETE THE BRANCH

yliang412 changed the title ~~[WIP] feat(core): new task graph without merging~~ [WIP] feat(core): new task graph without rules and merging Apr 5, 2025

yliang412 and others added 28 commits April 5, 2025 20:15

refactor Message

2fb0c7b

Signed-off-by: Yuchen Liang <[email protected]>

add some new tasks

b9fc601

wip

7ea3aa0

Signed-off-by: Yuchen Liang <[email protected]>

changed the process new_physical_partial and new_logical_partial

af4f0aa

mark uncompiled part of process_new_physical_partial todo

6b0f15e

Signed-off-by: Yuchen Liang <[email protected]>

rename task to be created to avoid confusion

ad5a2d8

Tasks are objectives. They might launch jobs that do actual work Signed-off-by: Yuchen Liang <[email protected]>

WIP

4bffa2e

Co-authored-by: Yuchen Liang <[email protected]>

tasks refactor WIP

39b5ad0

Signed-off-by: Yuchen Liang <[email protected]>

Finished some explore group functions

a4d924d

task graph WIP

ee6793a

Signed-off-by: Yuchen Liang <[email protected]>

remove subscriber; clarified memo.get_best_optimized_expr

920d5e4

Signed-off-by: Yuchen Liang <[email protected]>

remove one todo

b10a989

Signed-off-by: Yuchen Liang <[email protected]>

complete group + goal subscription

1773cf5

Signed-off-by: Yuchen Liang <[email protected]>

completed process_new_physical_partial

2c914f5

wip: not sure whether we should do this

9326e26

register the optimize_goal_in and cost_expr_in in `receive_new_goal_m…

7cb4d5c

…ember` Signed-off-by: Yuchen Liang <[email protected]>

remove logical property derivation during ingest

d79c2de

Signed-off-by: Yuchen Liang <[email protected]>

forward best costed to goal

7913d13

Signed-off-by: Yuchen Liang <[email protected]>

add cost to continue_with_costed task

5d70451

Signed-off-by: Yuchen Liang <[email protected]>

implement create_optimize_plan_task

d8f9d64

Signed-off-by: Yuchen Liang <[email protected]>

finish optimize_plan and add a new client API

62bfac6

Signed-off-by: Yuchen Liang <[email protected]>

fix infinite future error

b5b5a54

Signed-off-by: Yuchen Liang <[email protected]>

finish derive/retrieve property

8146ee7

Signed-off-by: Yuchen Liang <[email protected]>

rename message_tx/rx to engine_tx/rx to making things explcit

67fd5e0

Signed-off-by: Yuchen Liang <[email protected]>

add memo only test

f7b49f2

Signed-off-by: Yuchen Liang <[email protected]>

make update_physical_expr_cost eager

1d3559b

Signed-off-by: Yuchen Liang <[email protected]>

turn off and on optimizer

3946265

Signed-off-by: Yuchen Liang <[email protected]>

fix clippy

41d12f6

Signed-off-by: Yuchen Liang <[email protected]>

SarveshOO7 added 5 commits April 20, 2025 21:50

completed cascading merges and goal merges in memo table

1b39290

whoops! didn't need that

f9e70fa

handle physical stale physical expression whose cost tasks need to be…

b642931

… removed task graph

fixed asyn recursion; in-memory memo complete

42844dc

remove stuff to get a buildable version for now

87822de

connortsui20 mentioned this pull request Apr 24, 2025

Remove optimizer #82

Merged

connortsui20 added a commit that referenced this pull request Apr 24, 2025

Remove optimizer (#82)

70dbd27

## Problem To facilitate integration of the long-standing PR #55, this PR deletes the entire optimizer module so that we can slowly add stuff back in.

SarveshOO7 and others added 11 commits April 26, 2025 22:22

deletion of tasks and a mini rewrite of the merge result

ac851a1

deduplicate the publishers

f35cc73

small issue in group merge fixed. plus goal merge result updated

10d41ac

typo

92f0b49

typo 2.0

235c939

goal creation can have no source

6c65828

merge code, tehcnically complete

5e73313

merge goals while being able to delete stuff and not have cyclic goal…

ca4b48f

… dependencies

rename types (#94)

96b325d

## Problem Some of the names in `mod.rs` are super confusing. ## Summary of changes Changes the names of several of the types. Look at `memo/mod.rs` for the changes, everything else is just autorenamed.

copied the structure and code over from main

c6e2f3d

This will now allow us to merge the code much more easily into main. Furthermore, it will allow us to test the code using the DSL and it's compiled HIR.

fixed all compiler errors in the optimizer from merging

1bbbd88

fixed all clippy warnings except for one

SarveshOO7 force-pushed the yuchen/optimizer-with-new-task-graph branch from 6a8690a to 1bbbd88 Compare May 1, 2025 20:15

SarveshOO7 and others added 8 commits May 1, 2025 16:28

created a mock catalog

2299d94

wrote the simple HIR for the test

bed789f

rename to PropagateBestExpression

c15b26c

move memo module out of core

0494252

Merge branch 'main' into yuchen/optimizer-with-new-task-graph

dcd694b

fix merge conflicts

70ab35c

format

85b064f

fix clippy

839d8cd

connortsui20 force-pushed the yuchen/optimizer-with-new-task-graph branch from 5fb448b to 839d8cd Compare May 2, 2025 15:49

connortsui20 closed this May 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): new task graph without rules and merging #55

feat(core): new task graph without rules and merging #55

yliang412 commented Apr 5, 2025 •

edited

Loading

codecov-commenter commented Apr 5, 2025 •

edited

Loading

connortsui20 commented May 2, 2025 •

edited

Loading

feat(core): new task graph without rules and merging #55

feat(core): new task graph without rules and merging #55

Conversation

yliang412 commented Apr 5, 2025 • edited Loading

Problem

Summary of changes

Future work

codecov-commenter commented Apr 5, 2025 • edited Loading

Codecov Report

connortsui20 commented May 2, 2025 • edited Loading

yliang412 commented Apr 5, 2025 •

edited

Loading

codecov-commenter commented Apr 5, 2025 •

edited

Loading

connortsui20 commented May 2, 2025 •

edited

Loading