Adds example and distribution modules #47

prodriguezdefino · 2023-06-29T18:17:18Z

Also fixes a NPE when the underlying table in BQ gets expired, now returns a proper exception.

…ed table is expired and no longer accessible.

...igquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java

.../src/main/java/com/google/cloud/flink/bigquery/source/split/BigQuerySourceSplitAssigner.java

sonatype-lift · 2023-06-29T18:20:20Z

...-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java

+ * launching the job from Flink's Rest API is known the single quotes are not supported and will
+ * make the pipeline fail. As a workaround for that case using \u0027 as a replacement will make it
+ * work, example {@code "TIMESTAMP_TRUNC(ingestion_timestamp, HOUR) = \u00272023-06-20
+ * 19:00:00\u0027"}.


UnicodeEscape: Using unicode escape sequences for printable ASCII characters is obfuscated, and potentially dangerous.

Suggested change

* 19:00:00\u0027"}.

* 19:00:00'"}.

❗❗ 2 similar findings have been found in this PR

🔎 Expand here to view all instances of this finding

File Path Line Number

flink-connector-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java 46

flink-connector-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java 45

Visit the Lift Web Console to find more details in your report.

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

@sonatype-lift ignore

...bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java

...uery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java

sonatype-lift · 2023-06-29T18:20:26Z

...ry/src/main/java/com/google/cloud/flink/bigquery/common/utils/GoogleCredentialsSupplier.java

+
+    private static Credentials createCredentialsFromFile(String file) {
+        try {
+            return GoogleCredentials.fromStream(new FileInputStream(file));


PATH_TRAVERSAL_IN: This API (java/io/FileInputStream.(Ljava/lang/String;)V) reads a file whose location might be specified by user input

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

sonatype-lift · 2023-06-29T18:20:28Z

...-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/source/BigQuerySource.java

+ */
+@AutoValue
+@PublicEvolving
+public abstract class BigQuerySource<OUT>


AutoValueCouldNotWrite: Could not write generated class com.google.cloud.flink.bigquery.source.AutoValue_BigQuerySource: javax.annotation.processing.FilerException: Attempt to recreate a file for type com.google.cloud.flink.bigquery.source.AutoValue_BigQuerySource

❗❗ 3 similar findings have been found in this PR

🔎 Expand here to view all instances of this finding

File Path Line Number

flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java 34

flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java 41

flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java 34

Visit the Lift Web Console to find more details in your report.

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

...tor-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/utils/SchemaTransform.java

sonatype-lift · 2023-06-29T18:20:32Z

🛠 Lift Auto-fix

Some of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.¹

# Download the patch
curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/flink-bigquery-connector/47.diff -o lift-autofixes.diff

# Apply the patch with git
git apply lift-autofixes.diff

# Review the changes
git diff

Want it all in a single command? Open a terminal in your project's directory and copy and paste the following command:

curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/flink-bigquery-connector/47.diff | git apply

Once you're satisfied, commit and push your changes in your project.

You can preview the patch by opening the patch URL in the browser. ↩

… field since the dynamic table source and the source impl needs to store the read options as property and it becomes not serializable after changing the read options to use an optional property.

prodriguezdefino · 2023-11-02T23:23:09Z

/gcbrun

prodriguezdefino · 2023-11-03T18:34:46Z

/gcbrun

…ed table is expired and no longer accessible.

…driguezdefino/flink-bigquery-connector into add_example_and_shadedsqljar

jayehwhyehentee · 2023-11-24T06:52:52Z

/gcbrun

jayehwhyehentee · 2023-11-24T08:06:57Z

/gcbrun

jayadeep-jayaraman · 2023-11-24T08:24:19Z

...-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java

+            throws Exception {
+
+        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+        env.enableCheckpointing(60000L);


Can we pass this as a user provided configuration and have this value as a default if user does not pass the value

This is a configuration for the Flink application, not the connector. The default behaviour, as defined by Flink itself, is that checkpointing is disabled. Here is the official doc for more details.

I meant 60000L can be passed as a configuration instead of hardcoding. So that users can run this with different checkpoint intervals

jayadeep-jayaraman · 2023-11-24T08:25:52Z

...-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java

+
+        env.fromSource(bqSource, WatermarkStrategy.noWatermarks(), "BigQuerySource")
+                .flatMap(new FlatMapper(recordPropertyToAggregate))
+                .keyBy(t -> t.f0)


let's have meaningful variable names

Changed t to mappedTuple.

...-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java

jayehwhyehentee · 2023-11-27T08:34:38Z

/gcbrun

jayadeep-jayaraman · 2023-11-27T12:04:37Z

...-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java

+            throws Exception {
+
+        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+        env.enableCheckpointing(60000L);


I meant 60000L can be passed as a configuration instead of hardcoding. So that users can run this with different checkpoint intervals

jayehwhyehentee · 2023-11-27T12:45:27Z

/gcbrun

prodriguezdefino added 12 commits June 27, 2023 13:31

cleaning up the main branch for new version.

eb9dc8f

updates in ignore and readme files

0695673

prepping the pom addition, added parent's compliance tools

e36cc44

adding parent pom and the connector impl project pom

e167803

adding common functionalities

8043378

added the bigquery services wrapper and factories

b455b4e

creates the split, its state and the enumerator state

27cd837

added configs, split reader and split assignment impls

4d4b60f

applying recommendations from sonartype-lift

0567b58

adding the Datastream source implementation for BigQuery

9006714

added Table API implementation for BigQuery

d5d95bf

adding the example and shaded distro jar, fixes a NPE when the provid…

1263768

…ed table is expired and no longer accessible.

sonatype-lift bot reviewed Jun 29, 2023

View reviewed changes

...igquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Jun 29, 2023

View reviewed changes

.../src/main/java/com/google/cloud/flink/bigquery/source/split/BigQuerySourceSplitAssigner.java Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Jun 29, 2023

View reviewed changes

...bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Jun 29, 2023

View reviewed changes

...uery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Jun 29, 2023

View reviewed changes

...tor-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/utils/SchemaTransform.java Outdated Show resolved Hide resolved

prodriguezdefino added 9 commits July 6, 2023 13:24

merge changes from master (previous pom deletion resolution)

5743292

fixing the package name for the schema namespace

cab9115

Merge branch 'common_code_source' into bq_services_wrappers

3375d54

merged main branch and took care of few lift comments

849f769

Merge branch 'bq_services_wrappers' into source_splits

fd70b95

merge from source_split

0f18d14

fixing lift recommendations and spotless

6b08119

Merge branch 'split_assigner_and_reader' into source_functionality

5973d26

Merge branch 'source_functionality' into table_api

2d9635d

prodriguezdefino added 4 commits November 1, 2023 17:14

adding serializable autovalue annotation to avoid storing an Optional…

87780c5

… field since the dynamic table source and the source impl needs to store the read options as property and it becomes not serializable after changing the read options to use an optional property.

addressing comments from review

3f4d1be

addressed comments from review

c8eb789

Merge branch 'table_api' into add_example_and_shadedsqljar

fd79610

prodriguezdefino added a commit to prodriguezdefino/flink-bigquery-connector that referenced this pull request Nov 2, 2023

merge from GoogleCloudDataproc#47

8008260

prodriguezdefino added 2 commits November 3, 2023 11:24

partition list in 1024 chunks

cce6567

Merge branch 'table_api' into add_example_and_shadedsqljar

f36f91f

jayehwhyehentee and others added 3 commits November 23, 2023 21:16

Address review comments

5260073

adding the example and shaded distro jar, fixes a NPE when the provid…

936076b

…ed table is expired and no longer accessible.

Merge branch 'add_example_and_shadedsqljar' of https://github.com/pro…

7977576

…driguezdefino/flink-bigquery-connector into add_example_and_shadedsqljar

Merge branch 'main' into add_example_and_shadedsqljar

68d0bf9

jayadeep-jayaraman reviewed Nov 24, 2023

View reviewed changes

...-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java Show resolved Hide resolved

Address review comments

c9f73ee

jayehwhyehentee requested review from jayadeep-jayaraman and jayehwhyehentee November 27, 2023 08:39

jayehwhyehentee mentioned this pull request Nov 27, 2023

Use Open Clover for code coverage analysis #48

Merged

jayadeep-jayaraman requested changes Nov 27, 2023

View reviewed changes

Make checkpoint interval configurable in example jar

27c34d0

jayehwhyehentee requested a review from jayadeep-jayaraman November 27, 2023 12:46

jayadeep-jayaraman approved these changes Nov 27, 2023

View reviewed changes

jayadeep-jayaraman merged commit 337134d into GoogleCloudDataproc:main Nov 27, 2023
4 checks passed

jayehwhyehentee deleted the add_example_and_shadedsqljar branch December 12, 2023 05:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds example and distribution modules #47

Adds example and distribution modules #47

prodriguezdefino commented Jun 29, 2023

sonatype-lift bot Jun 29, 2023

prodriguezdefino Jul 11, 2023

sonatype-lift bot Jun 29, 2023

sonatype-lift bot Jun 29, 2023

sonatype-lift bot commented Jun 29, 2023

prodriguezdefino commented Nov 2, 2023

prodriguezdefino commented Nov 3, 2023

jayehwhyehentee commented Nov 24, 2023

jayehwhyehentee commented Nov 24, 2023

jayadeep-jayaraman Nov 24, 2023

jayehwhyehentee Nov 27, 2023

jayadeep-jayaraman Nov 27, 2023

jayadeep-jayaraman Nov 24, 2023

jayehwhyehentee Nov 27, 2023

jayehwhyehentee commented Nov 27, 2023

jayadeep-jayaraman Nov 27, 2023

jayehwhyehentee commented Nov 27, 2023

File Path	Line Number
flink-connector-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java	46
flink-connector-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java	45

Command	Usage
`@sonatype-lift ignore`	Leave out the above finding from this PR
`@sonatype-lift ignoreall`	Leave out all the existing findings from this PR
`@sonatype-lift exclude <file\|issue\|path\|tool>`	Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file

File Path	Line Number
flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java	34
flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java	41
flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java	34

Adds example and distribution modules #47

Adds example and distribution modules #47

Conversation

prodriguezdefino commented Jun 29, 2023

sonatype-lift bot Jun 29, 2023

Choose a reason for hiding this comment

prodriguezdefino Jul 11, 2023

Choose a reason for hiding this comment

sonatype-lift bot Jun 29, 2023

Choose a reason for hiding this comment

sonatype-lift bot Jun 29, 2023

Choose a reason for hiding this comment

sonatype-lift bot commented Jun 29, 2023

🛠 Lift Auto-fix

Footnotes

prodriguezdefino commented Nov 2, 2023

prodriguezdefino commented Nov 3, 2023

jayehwhyehentee commented Nov 24, 2023

jayehwhyehentee commented Nov 24, 2023

jayadeep-jayaraman Nov 24, 2023

Choose a reason for hiding this comment

jayehwhyehentee Nov 27, 2023

Choose a reason for hiding this comment

jayadeep-jayaraman Nov 27, 2023

Choose a reason for hiding this comment

jayadeep-jayaraman Nov 24, 2023

Choose a reason for hiding this comment

jayehwhyehentee Nov 27, 2023

Choose a reason for hiding this comment

jayehwhyehentee commented Nov 27, 2023

jayadeep-jayaraman Nov 27, 2023

Choose a reason for hiding this comment

jayehwhyehentee commented Nov 27, 2023