Pre-Aggregations with BigQuery: Access Denied despite persistent storage setup

[huavanson]

I'm currently using Cube.js as the backend for a Synmetrix dashboard and I’ve encountered an issue related to pre-aggregations when using BigQuery as the datasource.

Even though I have already set up persistent storage and configured the CUBESTORE_REMOTE_DIR, I’m still getting the following error when Cube tries to build pre-aggregations:

Error: Access Denied: Project xxx-dev: User does not have bigquery.datasets.create permission in project xxx-dev.

Interestingly, this issue does not happen when using PostgreSQL as the datasource.

From my understanding, once pre-aggregations are built, Cube Store should store the results in the persistent storage (as Parquet files), and future queries from the frontend that match pre-aggregations should be handled entirely by Cube Store workers — without involving the original datasource (BigQuery in this case). So I'm confused why Cube.js is still trying to access BigQuery during the query phase.

Could you please confirm whether:

My understanding about how pre-aggregations are used (with Cube Store taking over the query execution) is correct, and

Why Cube.js still needs bigquery.datasets.create permissions even when persistent storage is set up?

Any insights or clarification would be greatly appreciated.

Thanks in advance!

[igorlukanin]

I’m still getting the following error when Cube tries to build pre-aggregations:

Interestingly, this issue does not happen when using PostgreSQL as the datasource.

Hi @huavanson, this is the essence of what seems to be happening here. When using BigQuery as the data source, Cube is unable to build the pre-aggregations due to the lack of permissions. The error message that you're getting explicitly states that. It also explains why you're not getting the same error message for Postgres.

From my understanding, once pre-aggregations are built, Cube Store should store the results in the persistent storage (as Parquet files), and future queries from the frontend that match pre-aggregations should be handled entirely by Cube Store workers — without involving the original datasource (BigQuery in this case).

This is a correct understanding.

In your case, the pre-aggregations fail to build due to the lack of permissions.