Added test to validate recovery when StorageRead client fails #50
Conversation
…ed table is expired and no longer accessible.
…rom time to time, the pipeline should be able to recover reading from the beginning of the split or the last checkpointed offset
![sonatype-lift[bot]](https://avatars.githubusercontent.com/in/9843?s=60&v=4){kind=small}
...igquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/google/cloud/flink/bigquery/source/split/BigQuerySourceSplitAssigner.java
Outdated
Show resolved
Hide resolved
...-bigquery-examples/src/main/java/org/apache/flink/examples/gcp/bigquery/BigQueryExample.java
Outdated
Show resolved
Hide resolved
...bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java
Outdated
Show resolved
Hide resolved
...uery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java
Outdated
Show resolved
Hide resolved
...uery/src/main/java/com/google/cloud/flink/bigquery/table/restrictions/BigQueryPartition.java
Outdated
Show resolved
Hide resolved
|
|
||
| private static Credentials createCredentialsFromFile(String file) { | ||
| try { | ||
| return GoogleCredentials.fromStream(new FileInputStream(file)); |
There was a problem hiding this comment.
PATH_TRAVERSAL_IN: This API (java/io/FileInputStream.(Ljava/lang/String;)V) reads a file whose location might be specified by user input
ℹ️ Expand to see all @sonatype-lift commands
You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.
| Command | Usage |
|---|---|
@sonatype-lift ignore |
Leave out the above finding from this PR |
@sonatype-lift ignoreall |
Leave out all the existing findings from this PR |
@sonatype-lift exclude <file|issue|path|tool> |
Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file |
Note: When talking to LiftBot, you need to refresh the page to see its response.
Click here to add LiftBot to another repo.
| */ | ||
| @AutoValue | ||
| @PublicEvolving | ||
| public abstract class BigQuerySource<OUT> |
There was a problem hiding this comment.
AutoValueCouldNotWrite: Could not write generated class com.google.cloud.flink.bigquery.source.AutoValue_BigQuerySource: javax.annotation.processing.FilerException: Attempt to recreate a file for type com.google.cloud.flink.bigquery.source.AutoValue_BigQuerySource
❗❗ 3 similar findings have been found in this PR
🔎 Expand here to view all instances of this finding
| File Path | Line Number |
|---|---|
| flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java | 34 |
| flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java | 41 |
| flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java | 34 |
Visit the Lift Web Console to find more details in your report.
ℹ️ Expand to see all @sonatype-lift commands
You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.
| Command | Usage |
|---|---|
@sonatype-lift ignore |
Leave out the above finding from this PR |
@sonatype-lift ignoreall |
Leave out all the existing findings from this PR |
@sonatype-lift exclude <file|issue|path|tool> |
Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file |
Note: When talking to LiftBot, you need to refresh the page to see its response.
Click here to add LiftBot to another repo.
...tor-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/utils/SchemaTransform.java
Outdated
Show resolved
Hide resolved
🛠 Lift Auto-fixSome of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.1 # Download the patch
curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/flink-bigquery-connector/50.diff -o lift-autofixes.diff
# Apply the patch with git
git apply lift-autofixes.diff
# Review the changes
git diffWant it all in a single command? Open a terminal in your project's directory and copy and paste the following command: curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/flink-bigquery-connector/50.diff | git applyOnce you're satisfied, commit and push your changes in your project. Footnotes |
|
/gcbrun |
|
/gcbrun |
|
|
||
| private final Iterator<T> realIterator; | ||
| private final Double errorPercentage; | ||
| private final Random random = new Random(); |
There was a problem hiding this comment.
I wouldn't prefer using Random. If you absolutely have to, please use a fixed seed so that it's deterministic.
An alternative suggestion is to have it fail on every N-th invocation of next().
There was a problem hiding this comment.
Using random with seed
This pertains to identifying whether table is BQ native or external, which is relevant if we want to support external tables in the future.
|
/gcbrun |
|
/gcbrun |
The most common error, when reading large streams, occurs when iterating on the ServerStream. This new test should be able to demonstrate the pipeline is able to recover the execution and obtain the expected results by restarting the read from the beginning of the split or the last checkpointed offset.