Skip to content

Commit 01e998c

Browse files
jprakash-dbjackyhu-db
jprakash-db
and
jackyhu-db
authoredDec 27, 2024··
PySQL Connector split into connector and sqlalchemy (#444)
* Modified the gitignore file to not have .idea file * [PECO-1803] Splitting the PySql connector into the core and the non core part (#417) * Implemented ColumnQueue to test the fetchall without pyarrow Removed token removed token * order of fields in row corrected * Changed the folder structure and tested the basic setup to work * Refractored the code to make connector to work * Basic Setup of connector, core and sqlalchemy is working * Basic integration of core, connect and sqlalchemy is working * Setup working dynamic change from ColumnQueue to ArrowQueue * Refractored the test code and moved to respective folders * Added the unit test for column_queue Fixed __version__ Fix * venv_main added to git ignore * Added code for merging columnar table * Merging code for columnar * Fixed the retry_close sesssion test issue with logging * Fixed the databricks_sqlalchemy tests and introduced pytest.ini for the sqla_testing * Added pyarrow_test mark on pytest * Fixed databricks.sqlalchemy to databricks_sqlalchemy imports * Added poetry.lock * Added dist folder * Changed the pyproject.toml * Minor Fix * Added the pyarrow skip tag on unit tests and tested their working * Fixed the Decimal and timestamp conversion issue in non arrow pipeline * Removed not required files and reformatted * Fixed test_retry error * Changed the folder structure to src / databricks * Removed the columnar non arrow flow to another PR * Moved the README to the root * removed columnQueue instance * Revmoved databricks_sqlalchemy dependency in core * Changed the pysql_supports_arrow predicate, introduced changes in the pyproject.toml * Ran the black formatter with the original version * Extra .py removed from all the __init__.py files names * Undo formatting check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * BIG UPDATE * Refeactor code * Refractor * Fixed versioning * Minor refractoring * Minor refractoring * Changed the folder structure such that sqlalchemy has not reference here * Fixed README.md and CONTRIBUTING.md * Added manual publish * On push trigger added * Manually setting the publish step * Changed versioning in pyproject.toml * Bumped up the version to 4.0.0.b3 and also changed the structure to have pyarrow as optional * Removed the sqlalchemy tests from integration.yml file * [PECO-1803] Print warning message if pyarrow is not installed (#468) Print warning message if pyarrow is not installed Signed-off-by: Jacky Hu <[email protected]> * [PECO-1803] Remove sqlalchemy and update README.md (#469) Remove sqlalchemy and update README.md Signed-off-by: Jacky Hu <[email protected]> * Removed all sqlalchemy related stuff * generated the lock file * Fixed failing tests * removed poetry.lock * Updated the lock file * Fixed poetry numpy 2.2.2 issue * Workflow fixes --------- Signed-off-by: Jacky Hu <[email protected]> Co-authored-by: Jacky Hu <[email protected]>
·
v4.0.3v4.0.0
1 parent f9d6ef1 commit 01e998c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+467
-4911
lines changed
 

‎.github/workflows/code-quality-checks.yml

+51
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,57 @@ jobs:
5858
#----------------------------------------------
5959
- name: Run tests
6060
run: poetry run python -m pytest tests/unit
61+
run-unit-tests-with-arrow:
62+
runs-on: ubuntu-latest
63+
strategy:
64+
matrix:
65+
python-version: [ 3.8, 3.9, "3.10", "3.11" ]
66+
steps:
67+
#----------------------------------------------
68+
# check-out repo and set-up python
69+
#----------------------------------------------
70+
- name: Check out repository
71+
uses: actions/checkout@v2
72+
- name: Set up python ${{ matrix.python-version }}
73+
id: setup-python
74+
uses: actions/setup-python@v2
75+
with:
76+
python-version: ${{ matrix.python-version }}
77+
#----------------------------------------------
78+
# ----- install & configure poetry -----
79+
#----------------------------------------------
80+
- name: Install Poetry
81+
uses: snok/install-poetry@v1
82+
with:
83+
virtualenvs-create: true
84+
virtualenvs-in-project: true
85+
installer-parallel: true
86+
87+
#----------------------------------------------
88+
# load cached venv if cache exists
89+
#----------------------------------------------
90+
- name: Load cached venv
91+
id: cached-poetry-dependencies
92+
uses: actions/cache@v2
93+
with:
94+
path: .venv-pyarrow
95+
key: venv-pyarrow-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ github.event.repository.name }}-${{ hashFiles('**/poetry.lock') }}
96+
#----------------------------------------------
97+
# install dependencies if cache does not exist
98+
#----------------------------------------------
99+
- name: Install dependencies
100+
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
101+
run: poetry install --no-interaction --no-root
102+
#----------------------------------------------
103+
# install your root project, if required
104+
#----------------------------------------------
105+
- name: Install library
106+
run: poetry install --no-interaction --all-extras
107+
#----------------------------------------------
108+
# run test suite
109+
#----------------------------------------------
110+
- name: Run tests
111+
run: poetry run python -m pytest tests/unit
61112
check-linting:
62113
runs-on: ubuntu-latest
63114
strategy:

‎.github/workflows/integration.yml

-2
Original file line numberDiff line numberDiff line change
@@ -55,5 +55,3 @@ jobs:
5555
#----------------------------------------------
5656
- name: Run e2e tests
5757
run: poetry run python -m pytest tests/e2e
58-
- name: Run SQL Alchemy tests
59-
run: poetry run python -m pytest src/databricks/sqlalchemy/test_local

0 commit comments

Comments
 (0)
Please sign in to comment.