-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add SPI for delegating row expression optimizer #24144
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @tdcmeehan.
Just some initial comments
presto-main/src/main/java/com/facebook/presto/server/testing/TestingPrestoServer.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java
Show resolved
Hide resolved
17fd8e1
to
563611e
Compare
Codenotify: Notifying subscribers in CODENOTIFY files for diff 3f4e4c3...4f56709. No notifications. |
563611e
to
948cf13
Compare
948cf13
to
d25b1b6
Compare
97d0c3b
to
8c8d1f4
Compare
...s/src/test/java/com/facebook/presto/tests/expressions/TestDelegatingExpressionOptimizer.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/server/testing/TestingPrestoServer.java
Outdated
Show resolved
Hide resolved
...-main/src/main/java/com/facebook/presto/sql/relational/DelegatingRowExpressionOptimizer.java
Outdated
Show resolved
Hide resolved
...-main/src/main/java/com/facebook/presto/sql/relational/DelegatingRowExpressionOptimizer.java
Outdated
Show resolved
Hide resolved
...-main/src/main/java/com/facebook/presto/sql/relational/DelegatingRowExpressionOptimizer.java
Outdated
Show resolved
Hide resolved
3cbc135
to
6e124d2
Compare
9d80d3e
to
2cf08d2
Compare
Updates:
Thanks for the comments, please take another @ZacBlanco @rschlussel @aaneja @pdabre12 |
616bb09
to
226a3c3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two comments, otherwise lgtm
...o-main/src/test/java/com/facebook/presto/sql/expressions/TestExpressionOptimizerManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java
Show resolved
Hide resolved
226a3c3
to
92e8cd3
Compare
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/SystemSessionProperties.java
Outdated
Show resolved
Hide resolved
|
||
public ExpressionOptimizer getExpressionOptimizer(ConnectorSession connectorSession) | ||
{ | ||
checkArgument(connectorSession instanceof FullConnectorSession, "connectorSession is not an instance of FullConnectorSession"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i don't like assuming that a session is a connectorSession. it breaks the abstractions. In this case all we care about is the session property. can we instead add a getSystemProperty() method to ConnectorSession?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I agree this is not ideal, I'm of the opinion that it's worse to expose getSystemProperty
to the SPI, because this allows all plugins to access system properties that were previously only known to the engine or runtime. While not ideal, I think this approach is a bit less worse because it's within presto-main
, and we have examples of this being done for system connector and metadata tables provided to the JDBC driver. I do think that we should probably think of a better abstraction for how we provide session properties to plugins which are not connectors, and I've added a TODO and will open a Github issue to track it. Let me know if you think this is reasonable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair criticism and suggestion to punt on resolving this. i do feel like the problem i'm talking about isn't about plugins that aren't connectors, it's easy enough to implement passing them ConnectorSessions. The thing that makes this require a ConnectorSession vs. a Session is that it gets called from connector optimizers. and we had similar issue with stats stuff a while back. The plugin separation break down when connectors call into stuff from presto-main.
92e8cd3
to
4bc5d7e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just two nits, otherwise LGTM.
presto-main/src/main/java/com/facebook/presto/sql/relational/RowExpressionOptimizer.java
Outdated
Show resolved
Hide resolved
...in/java/com/facebook/presto/sql/planner/optimizations/CteProjectionAndPredicatePushDown.java
Show resolved
Hide resolved
4bc5d7e
to
551c188
Compare
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java
Outdated
Show resolved
Hide resolved
551c188
to
4f56709
Compare
The runtime should consolidate to the `ExpressionOptimizerProvider` factory so that it can be customized without significant refactoring.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for presto-native-execution changes.
Description
As part of RFC-0006, we need to support out of process expression evaluation. Add support for pluggable expression optimization and planner support to utilize the new SPI.
Motivation and Context
RFC-0006. See #24126 for larger changes that include the Presto sidecar as described in the RFC.
Impact
No impact by default as the old in-memory evaluation is the default.
Test Plan
Tests have been added.
Contributor checklist
Release Notes