Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds shell script for unbounded table read #71

Closed
wants to merge 32 commits into from
Closed

Adds shell script for unbounded table read #71

wants to merge 32 commits into from

Conversation

prashastia
Copy link
Collaborator

Adds shell script that

  1. Creates the asynchronously running unbounded job - the dataproc job is created in mode unbounded
  2. Dynamically adds partitions - insert_dynamic_partitions.py is executed with the necessary parameters.
  3. kills the dataproc job - As the read has completed and its correctness needs to be checked further.

@prashastia
Adds a new module for nightly tests.
a4d0d25 
This module is similar to the BigQueryExample. A few changes to count the number of records and log them.
@prashastia
Modifies the docstring.
f674638
@prashastia
Modifies the docstring.
6ec22dc
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
f0cd825 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
c814bd9 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
f5d3f31 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
b9cad97 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
b3ce92b 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
4e6ef67 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Modifies IntegrationTest to check query correctness.
dd8e5be
@prashastia
Adds spotless:apply.
137030d 
comments CODECOV_TOKEN usage.
@prashastia
Renames the table_read file to bounded_table_read.sh
957ebf0
@prashastia
Addresses review comments.
2ff4032
@prashastia
Creates separate shell script for bounded jobs.
394132d
@prashastia
Adds a new common shell script for common actions performed for both …
5425806 
…bounded and unbounded source.
@prashastia
Addresses review comments.
ea79601
@prashastia
Addresses review comments.
c6a564f
@prashastia
Formats the file.
a2a49b3
@prashastia
Fixes checkstyle violations.
3da1b14
@prashastia
Fixes checkstyle violations.
0f34c03
@prashastia
Fixes checkstyle violations.
0822681
@prashastia
Fixes checkstyle violations.
1770526
@prashastia
Changes test name, adds a new e2e test for checking table read for ta…
e416b1f 
…ble with complex schema.
@prashastia
Changes test name, adds a new e2e test for checking table read for ta…
471aca2 
…ble with complex schema.
@prashastia
Adds a new e2e test for checking query read.
68d48d0
@prashastia
Fixes cause of error in query read execution.
44f39f6
@prashastia
Fixes cause of error in query read execution.
b8a5883
@prashastia
Adds a new e2e test for checking large table ~200GBs read.
46ac1e6
@prashastia
Adds utils.py - a class containing implementations for writing recor…
74afc54 
…ds to different tables required for the e2e tests.
@prashastia
Adds table_type.py - a class containing various methods for record g…
41a0639 
…eneration.

 The records can be string, long, records type.
@prashastia
insert_dynamic_partitions.py A python script to insert partitions inc…
92835ec 
…rementally into the partitioned table. Used by the e2e unbounded source testing script.
@prashastia
Adds shell script that creates the asynchronously running unbounded j…
2925f6c 
…ob, Dynamically adds partitions and then kills the dataproc job.
@prashastia
Copy link
Collaborator Author

/gcbrun

@prashastia prashastia closed this Dec 23, 2023
@prashastia prashastia deleted the nightly-tests-unbounded-read-3 branch December 24, 2023 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jayehwhyehentee jayehwhyehentee Awaiting requested review from jayehwhyehentee

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@prashastia