Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to specify the database schema in addition to 'database' (e.g. for starrocks) #52

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

LaurinBrechter
Copy link

I am currently running dbt, superset and apache starrocks (I know it's kinda niche) and want to also use dbt-superset-lineage for this. In starrocks, there is no such thing as a database, there's only a schema. Therefore, I have to specify my dbt models like this:

sources:
  - name: data
    schema: ...

specifying my models like this

sources:
  - name: data
    database: ...

would raise an error in dbt. dbt-superset-lineage doesn't work as of now as it only reads the database key in the yaml file and can't find the tables as database is None

In this PR I have added the option to specify a database or a schema.

  • When both database and schema are given (like e.g. in postgres), only this combination will be pulled from.
  • When either of the 2 are given we ignore the other (like in starrocks where schema would be given but database ignored)
  • When none are given, we pull from all tables.

This also gives people more fine-grained control when they work with a DWH that has schema and database

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant