Skip to content

query: syntax to include metadata as part of the program. #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
prataprc opened this issue Aug 6, 2018 · 0 comments
Open

query: syntax to include metadata as part of the program. #6

prataprc opened this issue Aug 6, 2018 · 0 comments

Comments

@prataprc
Copy link
Member

prataprc commented Aug 6, 2018

Query, being inspired from JQ, allows the program to be
applied on one or more input document, where document can be
any json encoded value (or encoded in json equivalent format).
While this is useful to process JSON value in sed and
awk like fashion, it has some serious short-comings
when used in SQL fashion.

  • Key, big data systems are predominanty based on {key,value}
    document storage as its primary storage. To that extent we need
    to include key as part of the query-program.

  • Metadata, at big data scale, documents are predominantly
    distributed. This along with features like, append only values,
    create-modify-read timestamps, uuid, document versioning etc..
    adds one or more metadata fields to the actual document-value.

The solution we are proposing is:

  • To keep metadata separate from actual document value.
  • To include both document and metadata as inputs to query program.
  • Right now query programs can access the document value using
    . (dot) syntax, here after we can allow query programs to use # (hash)
    syntax to access the documents metadata.
  • Key can be made available via metadata, like #_key.
  • While document (.) can be any JSON value, metadata is always
    a set of {key,value} pairs, where Key is of string-type and value
    can be any valid JSON value.
@prataprc prataprc changed the title query: enhance query syntax to include metadata as part of the language. query: syntax to include metadata as part of the language. Aug 6, 2018
@prataprc prataprc changed the title query: syntax to include metadata as part of the language. query: syntax to include metadata as part of the program. Aug 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant