Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

varun-edachali-dbx · 2025-05-26T13:30:00Z

What type of PR is this?

Refactor

Description

Create ResultSet interface and make the existing ResultSet (as ThriftResultSet) implement it. Concrete Thrift Backend layer should return ThriftResultSet from execute commands.
Generalise the return types of some functions in the backend interface to prevent alignment with Thrift (create new Status enum to avoid using Thrift's Status type).

How is this tested?

Unit tests
- some of the existing unit tests were slightly altered to account for the introduction of the interface. No unit tests were removed or introduced.
E2E Tests
Manually
N/A

Related Tickets & Documents

https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?usp=sharing
https://databricks.atlassian.net/browse/PECOBLR-445?atlOrigin=eyJpIjoiMjA3ZTljYTUzNDgxNDY4MmFhMThjYjI5MWY0Yjc0MTIiLCJwIjoiaiJ9

github-actions · 2025-05-26T13:30:14Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-26T15:01:49Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-28T03:31:46Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-28T03:40:13Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

new codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

…t to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: varun-edachali-dbx <[email protected]>

* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * fixed format Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> --------- Signed-off-by: Sai Shree Pradhan <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

More conditions to run github actions Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

* Modified the gitignore file to not have .idea file * [PECO-1803] Splitting the PySql connector into the core and the non core part (#417) * Implemented ColumnQueue to test the fetchall without pyarrow Removed token removed token * order of fields in row corrected * Changed the folder structure and tested the basic setup to work * Refractored the code to make connector to work * Basic Setup of connector, core and sqlalchemy is working * Basic integration of core, connect and sqlalchemy is working * Setup working dynamic change from ColumnQueue to ArrowQueue * Refractored the test code and moved to respective folders * Added the unit test for column_queue Fixed __version__ Fix * venv_main added to git ignore * Added code for merging columnar table * Merging code for columnar * Fixed the retry_close sesssion test issue with logging * Fixed the databricks_sqlalchemy tests and introduced pytest.ini for the sqla_testing * Added pyarrow_test mark on pytest * Fixed databricks.sqlalchemy to databricks_sqlalchemy imports * Added poetry.lock * Added dist folder * Changed the pyproject.toml * Minor Fix * Added the pyarrow skip tag on unit tests and tested their working * Fixed the Decimal and timestamp conversion issue in non arrow pipeline * Removed not required files and reformatted * Fixed test_retry error * Changed the folder structure to src / databricks * Removed the columnar non arrow flow to another PR * Moved the README to the root * removed columnQueue instance * Revmoved databricks_sqlalchemy dependency in core * Changed the pysql_supports_arrow predicate, introduced changes in the pyproject.toml * Ran the black formatter with the original version * Extra .py removed from all the __init__.py files names * Undo formatting check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * BIG UPDATE * Refeactor code * Refractor * Fixed versioning * Minor refractoring * Minor refractoring * Changed the folder structure such that sqlalchemy has not reference here * Fixed README.md and CONTRIBUTING.md * Added manual publish * On push trigger added * Manually setting the publish step * Changed versioning in pyproject.toml * Bumped up the version to 4.0.0.b3 and also changed the structure to have pyarrow as optional * Removed the sqlalchemy tests from integration.yml file * [PECO-1803] Print warning message if pyarrow is not installed (#468) Print warning message if pyarrow is not installed Signed-off-by: Jacky Hu <[email protected]> * [PECO-1803] Remove sqlalchemy and update README.md (#469) Remove sqlalchemy and update README.md Signed-off-by: Jacky Hu <[email protected]> * Removed all sqlalchemy related stuff * generated the lock file * Fixed failing tests * removed poetry.lock * Updated the lock file * Fixed poetry numpy 2.2.2 issue * Workflow fixes --------- Signed-off-by: Jacky Hu <[email protected]> Co-authored-by: Jacky Hu <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

* Removed python3.8 support * Minor fix Signed-off-by: varun-edachali-dbx <[email protected]>

Support for Py till 3.12 Signed-off-by: varun-edachali-dbx <[email protected]>

* Increased the number of retry attempts allowed (#486) Updated the number of attempts allowed * bump version to 3.7.1 (#487) bumped up version * Refractore * Minor change Signed-off-by: varun-edachali-dbx <[email protected]>

bumped up the version Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-02T06:28:50Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-02T08:42:07Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-02T09:06:03Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-02T10:09:09Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-02T11:30:18Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-03T01:40:00Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-03T01:54:02Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-03T04:16:56Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-03T04:42:15Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

jayantsing-db

thanks. a minor comment for a separate PR.

jayantsing-db · 2025-06-03T05:08:17Z

src/databricks/sql/backend/thrift_backend.py

+        return ThriftResultSet(
+            connection=cursor.connection,
+            execute_response=execute_response,
+            thrift_client=self,


we already pass connection which has session which in turn has the backend? can we look into normalising this init params (in a separate PR)

varun-edachali-dbx had a problem deploying to azure-prod May 26, 2025 13:30 — with GitHub Actions Failure

varun-edachali-dbx temporarily deployed to azure-prod May 26, 2025 15:01 — with GitHub Actions Inactive

varun-edachali-dbx had a problem deploying to azure-prod May 26, 2025 15:01 — with GitHub Actions Failure

varun-edachali-dbx force-pushed the fetch-interface branch from 25158cc to 9b265c1 Compare May 26, 2025 15:02

varun-edachali-dbx had a problem deploying to azure-prod May 26, 2025 15:02 — with GitHub Actions Failure

varun-edachali-dbx temporarily deployed to azure-prod May 26, 2025 15:44 — with GitHub Actions Inactive

varun-edachali-dbx had a problem deploying to azure-prod May 28, 2025 03:31 — with GitHub Actions Failure

varun-edachali-dbx had a problem deploying to azure-prod May 28, 2025 03:40 — with GitHub Actions Failure

varun-edachali-dbx and others added 16 commits May 28, 2025 03:40

ensure backend client returns a ResultSet type in backend tests

8ea5cf4

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

3eba92a

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: correct client tests

6735fd5

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

12388e8

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess comments (nit)

853e6c3

Signed-off-by: varun-edachali-dbx <[email protected]>

Update CODEOWNERS (#562)

4f8b54c

new codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

Update github actions run conditions (#569)

2c2984e

More conditions to run github actions Signed-off-by: varun-edachali-dbx <[email protected]>

fix: thrift_backend->backend, ResultSet -> ThriftResultSet

4979422

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: thrift_backend -> backend in e2e test_driver

6361e85

Signed-off-by: varun-edachali-dbx <[email protected]>

Removed CI CD for python3.8 (#490)

e5ac8c6

* Removed python3.8 support * Minor fix Signed-off-by: varun-edachali-dbx <[email protected]>

Added CI CD upto python 3.12 (#491)

e06fd2f

Support for Py till 3.12 Signed-off-by: varun-edachali-dbx <[email protected]>

Merging changes from v3.7.1 release (#488)

7126f97

* Increased the number of retry attempts allowed (#486) Updated the number of attempts allowed * bump version to 3.7.1 (#487) bumped up version * Refractore * Minor change Signed-off-by: varun-edachali-dbx <[email protected]>

Bumped up to version 4.0.0 (#493)

b1245da

bumped up the version Signed-off-by: varun-edachali-dbx <[email protected]>

stronger typing in resultSet behaviour

dfb4cc3

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 2, 2025 06:28 — with GitHub Actions Failure

remove redundant patch in test

6a9f0e2

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 2, 2025 08:41 — with GitHub Actions Failure

add has_been_closed_server_side assertion

0b2e83f

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 2, 2025 09:05 — with GitHub Actions Failure

remove redundancies in tests

1f0c81f

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 2, 2025 10:09 — with GitHub Actions Failure

more robust close check

aa7207e

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 2, 2025 11:30 — with GitHub Actions Failure

use normalised state in e2e test

fde4634

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 3, 2025 01:39 — with GitHub Actions Failure

simplify corrected test

54f2b93

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 3, 2025 01:53 — with GitHub Actions Inactive

add line gaps after multi-line pydocs for consistency

db73ecf

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 3, 2025 04:16 — with GitHub Actions Failure

varun-edachali-dbx requested a review from jayantsing-db June 3, 2025 04:28

use normalised CommandState type in ExecuteResponse

cf8a629

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 3, 2025 04:42 — with GitHub Actions Inactive

jayantsing-db approved these changes Jun 3, 2025

View reviewed changes

varun-edachali-dbx merged commit 3c78ed7 into sea-migration Jun 3, 2025
22 of 23 checks passed

Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

Uh oh!

Conversation

varun-edachali-dbx commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Description

How is this tested?

Related Tickets & Documents

Uh oh!

github-actions bot commented May 26, 2025

Uh oh!

github-actions bot commented May 26, 2025

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

jayantsing-db left a comment

Choose a reason for hiding this comment

Uh oh!

jayantsing-db Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

varun-edachali-dbx commented May 26, 2025 •

edited

Loading