-
Notifications
You must be signed in to change notification settings - Fork 114
Support SQL views #3680
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Support SQL views #3680
Conversation
fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/View.java
Outdated
Show resolved
Hide resolved
fdb-relational-api/src/main/java/com/apple/foundationdb/relational/api/metadata/View.java
Show resolved
Hide resolved
...cord-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/RawView.java
Outdated
Show resolved
Hide resolved
fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/View.java
Outdated
Show resolved
Hide resolved
fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/RecordMetaDataBuilder.java
Outdated
Show resolved
Hide resolved
} | ||
for (RecordMetaDataProto.PView viewProto: metaDataProto.getViewsList()) { | ||
final RawView rawView = (RawView)PlanSerialization.dispatchFromProtoContainer( | ||
new PlanSerializationContext(DefaultPlanSerializationRegistry.INSTANCE, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like piggy-backing on plan serialization to just be able to do the dispatch on a multitude of views here . How many kinds of views do we have right now (I think one) -- how many do you think we will have later?
Suggestions:
- extricate the logic from the plan serialization logic and make the specific plan serialization logic use and share that logic with this code.
- dynamic dispatch in the same way as that logic but with a switch statement if the number of actual implementations is like
1
or2
ever.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, I originally wanted to introduce a forward-looking representation of metadata that leaves the door open for introducing other flavors of views, such as fully expanded, partially expanded, and so on. But I don't think this is realistic, moreover, views in their current shape at least, are pure metadata artifacts, and you're right for calling out their useless dependence of the plan serialization.
Having said that, I changed the metadata representation of the views such that it is now a pure metadata artifacts whose Ser/De has nothing to do anymore with the plan serialization API.
This introduces support for SQL views. Views are virtual tables defined by SQL queries that are compiled lazily when referenced, providing a way to encapsulate complex queries and present simplified interfaces to users.
Implementation Overview
Views are stored as raw SQL text in the catalog and compiled on-demand during query execution. The implementation follows a layered architecture:
At the core layer, we introduce
RawView
which represents the persisted view definition. Views are stored in the metadata alongside user-defined functions and are serialized/deserialized using the existing protobuf infrastructure. TheRecordMetaData
andRecordMetaDataBuilder
classes now maintain a map of views that get persisted with the schema.At the relational API layer,
RecordLayerView
wraps the raw view definition with compilation capabilities. Each view contains a lazy compilation function that transforms the SQL text into a logical plan when the view is first referenced. This compiled plan can be cached to avoid recompilation on subsequent accesses.The query parser has been extended to recognize view references in
FROM
clauses. When a view is encountered, the semantic analyzer retrieves the view definition from the catalog and triggers compilation by invoking the view's compilation function. The resulting logical operator is then seamlessly integrated into the query plan, allowing views to be used just like regular tables.Features
CREATE VIEW
syntax and stored as SQL text in the schema templateThe implementation includes comprehensive testing through YAML integration tests that validate various view scenarios including simple filters, nested views, CTEs within views, and views referencing user-defined functions.
This fixes #3670.