PERF: Optimize pooled connection return while preserving transaction safety

[Sumit Sarabhai (sumitmsft)]

AB#48076

Summary

This PR now replaces the earlier revert with a targeted pooling optimization. It retains the transaction, failure-discard, pool-capacity and native handle-lifetime protections from #777.

• Skip pool sanitation only when a prior successful rollback/autocommit restoration established clean state and no subsequent statement allocation or uncertain operation invalidated it. A new login or deferred reset alone never establishes clean state.
• Retained statement aliases, raw-handle exposure and unknown connection attributes remain conservative. Valid scalar login timeouts such as timeout=30 do not permanently disable the fast path.
• Move close-time transaction handling into native code and consolidate required sanitation into one autocommit probe, metadata invalidation and GIL-release scope. Never turn autocommit on after a failed rollback.
• Preserve cleanup gates, discard/capacity recovery, pool-generation and shutdown protections. Extend the existing explicit-transaction integration cases and add deterministic native failure/call-count coverage.

Scope and limitations

Used or uncertain connections still execute full transaction sanitation. This does not assume autocommit=True means there cannot be an explicit SQL transaction, and it does not remove safety synchronization to improve timings. End-to-end performance improvements and live SQL behavior remain subject to validation; no blanket regression-elimination claim is made.

Validation

• Windows x64 Release extension and native test executable built successfully.
• 43 deterministic native sanitation cases passed; CTest passed.
• 24 targeted Python tests passed using the freshly built extension.
• Positive-timeout coverage verifies that timeout 30 reaches physical login and that 100 subsequent unused pooled leases issue no sanitation calls.
• Three expanded live-SQL integration cases collected successfully but were not executed locally. Normal cross-platform PR validation and performance evaluation are pending.

[github-actions]

PR Performance Report

✅ No regression detected

No consistent slowdowns detected across all 2 environments.

0 IMPROVEMENTS 0 SLOWDOWNS 2/2 ENVIRONMENTS

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task	Before	After	Paired change	Result
Connection opening	10.401 ms	10.236 ms	-2.0%	no signal
SELECT queries	1.046 ms	1.045 ms	-0.6%	no signal
Row insertion	33.558 ms	33.504 ms	-0.2%	no signal
Executemany inserts	154.035 ms	156.027 ms	+2.2%	no signal
Fetch-all queries	119.800 ms	119.728 ms	-1.4%	no signal
Row-by-row fetching	14.119 ms	14.558 ms	+0.7%	no signal
Batched row fetching	119.758 ms	115.298 ms	-3.0%	no signal
Transaction commit and rollback	110.741 ms	109.540 ms	-1.1%	no signal
Arrow row fetching	94.748 ms	92.807 ms	-2.3%	no signal
100,000-row insertion	432.196 ms	430.478 ms	-1.0%	no signal
Row fetching in batches of 100	120.182 ms	120.007 ms	-0.2%	no signal
Row fetching in batches of 10,000	119.337 ms	133.886 ms	+12.5%	no signal
Repeated positional queries	32.428 ms	32.214 ms	-0.6%	no signal
Repeated named-parameter queries	34.447 ms	35.055 ms	+2.6%	no signal
Legacy 100,000-row insertion	427.486 ms	351.255 ms	-19.0%	no signal
Insertion with explicit input sizes	485.856 ms	482.487 ms	-0.7%	no signal
Joined aggregation queries	182.996 ms	179.373 ms	-0.8%	no signal
Large joined-result fetching	177.176 ms	175.455 ms	+0.5%	no signal
1.2-million-row fetching	3424.313 ms	3415.328 ms	+0.2%	no signal
Common table expression queries	5.323 ms	5.248 ms	-1.2%	no signal
256 KiB VARCHAR(MAX) / fetchall()	1.288 ms	1.223 ms	-5.1%	no signal

Unix / SQL Server 2025

Database task	Before	After	Paired change	Result
Connection opening	97.196 ms	96.364 ms	-0.3%	no signal
SELECT queries	1.114 ms	1.087 ms	-1.1%	no signal
Row insertion	34.834 ms	34.874 ms	-0.4%	no signal
Executemany inserts	149.413 ms	151.991 ms	-0.2%	no signal
Fetch-all queries	122.176 ms	120.992 ms	+0.2%	no signal
Row-by-row fetching	14.310 ms	14.678 ms	+3.6%	no signal
Batched row fetching	116.801 ms	117.438 ms	+0.6%	no signal
Transaction commit and rollback	117.128 ms	117.830 ms	+1.3%	no signal
Arrow row fetching	93.829 ms	94.652 ms	+1.1%	no signal
100,000-row insertion	485.807 ms	433.588 ms	-10.5%	no signal
Row fetching in batches of 100	122.961 ms	122.434 ms	-0.9%	no signal
Row fetching in batches of 10,000	125.140 ms	127.255 ms	+5.6%	no signal
Repeated positional queries	34.053 ms	34.239 ms	+0.5%	no signal
Repeated named-parameter queries	36.423 ms	36.903 ms	+1.3%	no signal
Legacy 100,000-row insertion	351.545 ms	353.536 ms	-1.1%	no signal
Insertion with explicit input sizes	499.796 ms	494.416 ms	-3.8%	no signal
Joined aggregation queries	161.665 ms	159.487 ms	-0.9%	no signal
Large joined-result fetching	184.764 ms	185.922 ms	+0.9%	no signal
1.2-million-row fetching	3541.185 ms	3559.394 ms	+1.8%	no signal
Common table expression queries	5.090 ms	5.166 ms	+0.1%	no signal
256 KiB VARCHAR(MAX) / fetchall()	1.510 ms	1.527 ms	+1.2%	no signal

Build and measurement details

ADO build 178409

PR head: e7444e70f56e31d9e3c9c37365fc97901ec7ee85
Base: d849a09119c831d945521d5333cfb9bdb6e0561d
Measured merge: 6a26f626521a9b5c8d68e7a74879a94010b9ccad

• Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4295.3; 5 paired comparisons and 1 warmup.
• Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5005.3; 5 paired comparisons and 1 warmup.

A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical cleanup, destructor, handle-lifecycle, and disconnect-synchronization issues block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 4 · 1

Open (5)

• Native connection handle remains attached after cleanup failure · New
• Connection destructor can terminate on disconnect failure · New
• Concurrent statement allocation races with connection disconnect · New
• SQLFreeHandle failure incorrectly clears live handle · New
• Returned connection can enter a replacement pool generation · New

What changed in this PR

Reverts PR #777 to restore prior pooled-connection behavior while retaining result-metadata invalidation.

Changes:

• Removes pooled transaction sanitization and related regression tests.
• Restores previous native connection, handle cleanup, and pool-return behavior.
• Removes the reverted changelog entry.

File	Summary
tests/​test_009_pooling.py	Removes pooling and native lifecycle regression tests.
tests/​test_006_exceptions.py	Removes close-failure tests.
mssql_python/​pybind/​ddbc_bindings.h	Removes cleanup-state APIs.
mssql_python/​pybind/​ddbc_bindings.cpp	Restores earlier handle-management behavior.
mssql_python/​pybind/​connection/​connection.h	Removes pool-sanitation interfaces.
mssql_python/​pybind/​connection/​connection.cpp	Restores prior disconnect and pool-return behavior.
mssql_python/​pybind/​connection/​connection_pool.h	Removes discard/origin-pool APIs.
mssql_python/​pybind/​connection/​connection_pool.cpp	Restores key-based pool returns.
mssql_python/​connection.py	Restores the previous close flow.
CHANGELOG.md	Removes the reverted fix entry.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 82%

🎯 Overall Coverage 84%

📈 Total Lines Covered: 9488 out of 11193 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/connection.py (100%)
• mssql_python/pybind/connection/connection.cpp (82.7%): Missing lines 291,310,331,348-349,369-370,476-480,484-485,753-758,929,986,988-989

Summary

• Total: 140 lines
• Missing: 24 lines
• Coverage: 82%

mssql_python/pybind/connection/connection.cpp

Lines 287-295

  287 
  288 void Connection::commit() {
  289     PERF_TIMER("Connection::commit");
  290     _poolClean = false;
! 291     _poolSessionReset = false;
  292     if (!_dbcHandle) {
  293         ThrowStdException("Connection handle not allocated");
  294     }
  295     updateLastUsed();

---

Lines 306-314

  306 
  307 void Connection::rollback() {
  308     PERF_TIMER("Connection::rollback");
  309     _poolClean = false;
! 310     _poolSessionReset = false;
  311     if (!_dbcHandle) {
  312         ThrowStdException("Connection handle not allocated");
  313     }
  314     updateLastUsed();

---

Lines 327-335

  327     PERF_TIMER("Connection::setAutocommit");
  328     if (!enable) {
  329         _poolClean = false;
  330         _poolSessionReset = false;
! 331     }
  332     if (!_dbcHandle) {
  333         ThrowStdException("Connection handle not allocated");
  334     }
  335     clearResultMetadata();

---

Lines 344-353

  344         py::gil_scoped_release release;
  345         ret = SQLSetConnectAttr_ptr(_dbcHandle->get(), SQL_ATTR_AUTOCOMMIT,
  346                                     reinterpret_cast<SQLPOINTER>(static_cast<SQLULEN>(value)), 0);
  347     }
! 348     if (!SQL_SUCCEEDED(ret)) {
! 349         _poolClean = false;
  350         checkError(ret);
  351     }
  352     if (value == SQL_AUTOCOMMIT_ON) {
  353         LOG("Autocommit enabled");

---

Lines 365-374

  365     SQLINTEGER value;
  366     SQLINTEGER string_length;
  367     SQLRETURN ret = SQLGetConnectAttr_ptr(_dbcHandle->get(), SQL_ATTR_AUTOCOMMIT, &value,
  368                                           sizeof(value), &string_length);
! 369     if (!SQL_SUCCEEDED(ret)) {
! 370         _poolClean = false;
  371         checkError(ret);
  372     }
  373     return value == SQL_AUTOCOMMIT_ON;
  374 }

---

Lines 472-489

  472     }
  473 
  474     if (py::isinstance<py::int_>(value)) {
  475         // Get the integer value
! 476         int64_t longValue;
! 477         try {
! 478             longValue = value.cast<int64_t>();
! 479         } catch (const py::cast_error&) {
! 480             _poolProofDisabled = true;
  481             throw;
  482         } catch (const py::error_already_set&) {
  483             _poolProofDisabled = true;
! 484             throw;
! 485         }
  486         if (scalarLoginTimeout &&
  487             (longValue < 0 ||
  488              static_cast<uint64_t>(longValue) > std::numeric_limits<SQLUINTEGER>::max())) {
  489             _poolProofDisabled = true;

---

Lines 749-762

  749                                             reinterpretU16stringAsSqlWChar(rollbackQuery), SQL_NTS);
  750                     if (!SQL_SUCCEEDED(ret)) {
  751                         ErrorInfo error = SQLReadError(SQL_HANDLE_STMT, statement, ret);
  752                         statementError = error.sqlState.length() == 5
! 753                             ? "SQLSTATE:" + error.sqlState + ":" + error.ddbcErrorMsg
! 754                             : error.ddbcErrorMsg;
! 755                     }
! 756                     SQLRETURN freeRet = SQLFreeHandle_ptr(SQL_HANDLE_STMT, statement);
! 757                     if (SQL_SUCCEEDED(ret) && !SQL_SUCCEEDED(freeRet)) {
! 758                         ErrorInfo error = SQLReadError(SQL_HANDLE_STMT, statement, freeRet);
  759                         statementError = error.sqlState.length() == 5
  760                             ? "SQLSTATE:" + error.sqlState + ":" + error.ddbcErrorMsg
  761                             : error.ddbcErrorMsg;
  762                         ret = freeRet;

---

Lines 925-933

  925         // SQLDisconnect failure using the existing connection and children.
  926         if (!_usePool && !rollbackBeforeDisconnect) {
  927             throw;
  928         }
! 929         // Never retain a connection whose transaction state could not be
  930         // sanitized. Release capacity and preserve the original cleanup error.
  931         try {
  932             ConnectionPoolManager::getInstance().discardConnection(_originPool, _conn);
  933         } catch (...) {

---

Lines 982-993

  982     return conn->allocStatementHandle();
  983 }
  984 
  985 py::object Connection::getInfo(SQLUSMALLINT infoType) const {
! 986     _poolClean = false;
  987     _poolSessionReset = false;
! 988     if (infoType == SQL_DRIVER_HDBC || infoType == SQL_DRIVER_HENV ||
! 989         infoType == SQL_DRIVER_HSTMT || infoType == SQL_DRIVER_HLIB) {
  990         _poolProofDisabled = true;
  991     }
  992     if (!_dbcHandle) {
  993         ThrowStdException("Connection handle not allocated");

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 62.6%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 79.3%
mssql_python.pybind.connection.connection_pool.cpp: 82.3%
mssql_python.pybind.connection.connection.cpp: 82.7%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.fetch_temporal.hpp: 92.1%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[Copilot]

Copilot review overview

🟡 Changes recommended

Native tests are not part of standard validation, and exception tests no longer cover the failure paths their names describe.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

• Opt-in native tests are excluded from standard CI · New
• Rollback failure test no longer exercises rollback failure · New

Resolved since last review (5)

• SQLFreeHandle failure incorrectly clears live handle
• Concurrent statement allocation races with connection disconnect
• Connection destructor can terminate on disconnect failure
• Native connection handle remains attached after cleanup failure
• Returned connection can enter a replacement pool generation

[Copilot]

Copilot review overview

🔵 Needs a closer look

Enable the native sanitation regression suite in normal CI.

Review effort: Lite
Findings: 2

Open (2)

• Rollback failure test no longer exercises rollback failure
• Opt-in native tests are excluded from standard CI

[Copilot]

Copilot review overview

🔵 Needs a closer look

Address the manual-lease state bug, restore Python-boundary failure coverage, and run native sanitation tests in CI.

Review effort: Lite
Findings: 2

Open (2)

• Rollback failure test no longer exercises rollback failure
• Opt-in native tests are excluded from standard CI

Previously missed (1)

In code that hasn't changed since last review

Autocommit read failure coverage was removed

tests/​test_006_exceptions.py:304

This replacement likewise removes the simulated get_autocommit failure, so the test named test_close_cleans_up_after_autocommit_read_failure no longer covers that error path. Because the probe now runs inside native close, add a Python-boundary test that can inject/observe a native probe failure, or rename this test so the suite does not falsely claim coverage it no longer provides.