REFACTOR: remove bundled ODBC driver from mssql-python wheel; require mssql-python-odbc (Phase 2)

[Jahnvi Thakkar (jahnvi480)]

Work Item / Issue Reference

AB#46836

---

Summary

This pull request makes significant updates to the build and packaging process for the mssql-python and mssql-python-odbc Python packages, reflecting the transition to Phase 2 where the ODBC driver is fully separated into its own package. The pipeline now always builds both packages together, ensures the external ODBC wheel is installed before testing, and simplifies and streamlines build and test stages. Documentation and prompts are also updated to reflect these structural changes.

Build and Pipeline Simplification:

• The pipeline now always builds both mssql-python and mssql-python-odbc packages together; parameters and logic for building only one package have been removed, simplifying configuration and execution. [1] [2]
• The ODBC build is consolidated into a single ODBC_BuildAll stage that produces all platform wheels from one host, replacing multiple per-platform ODBC build stages.
• All mssql-python build stages now depend on the ConsolidateOdbc stage and install the external ODBC wheel before running tests, validating the package split end-to-end during the build. The previous post-consolidation "TestBothWheels" stages are removed as redundant.

Documentation and Prompt Updates:

• Documentation and prompts are updated to reference the new mssql_python_odbc/libs location for ODBC binaries and to clarify that these are now maintained in the standalone mssql-python-odbc package.

Platform and Security Improvements:

• The macOS build pool is updated to use the latest available image (macos-latest, macOS-15 Sequoia).
• The SBOM (Software Bill of Materials) is now always named after the primary mssql-python wheel, reflecting the unified build process.

Release Pipeline Robustness:

• The release pipeline now derives the expected ODBC package version directly from mssql_python_odbc/__init__.py, ensuring the version pin is always in sync and eliminating the need for manual updates when the ODBC version changes.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 60%

🎯 Overall Coverage 82%

📈 Total Lines Covered: 7368 out of 8963 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/pybind/ddbc_bindings.cpp (60.0%): Missing lines 1164,1178-1180,1476,1478,1486-1487

Summary

• Total: 20 lines
• Missing: 8 lines
• Coverage: 60%

mssql_python/pybind/ddbc_bindings.cpp

Lines 1160-1168

  1160         if (!externalComplete) {
  1161             LOG("GetOdbcLibsBaseDir: mssql_python_odbc present at '%s' but its ODBC driver "
  1162                 "binaries are missing or incomplete for this platform",
  1163                 parentDir.string().c_str());
! 1164             ThrowStdException(
  1165                 "The 'mssql-python-odbc' package is installed but its ODBC driver binaries "
  1166                 "are missing or incomplete for this platform. Reinstall it with: "
  1167                 "pip install --force-reinstall mssql-python-odbc");
  1168         }

---

Lines 1174-1184

  1174             // Phase 2: the standalone package is required. Turn the missing
  1175             // dependency into a clear, actionable error instead of a fallback.
  1176             LOG("GetOdbcLibsBaseDir: required package mssql_python_odbc is not installed (%s)",
  1177                 e.what());
! 1178             ThrowStdException(
! 1179                 "The required 'mssql-python-odbc' package (which ships the ODBC driver "
! 1180                 "binaries) is not installed. Install it with: pip install mssql-python-odbc");
  1181         }
  1182         // A different import-time error means the package is installed but
  1183         // broken; surface it instead of silently masking the real problem.
  1184         LOG("GetOdbcLibsBaseDir: importing mssql_python_odbc failed unexpectedly (%s); "

---

Lines 1472-1482

  1472     // the actionable "install mssql-python-odbc" error into a hard crash. So we
  1473     // capture any failure as an exception_ptr inside the callable and rethrow it
  1474     // below, in a normal context that pybind11 can translate into a Python
  1475     // exception. The stored error persists, so every subsequent call re-raises the
! 1476     // same actionable message instead of silently proceeding without a driver.
  1477     std::call_once(m_onceFlag, [this]() {
! 1478         try {
  1479             LoadDriverOrThrowException();
  1480             m_driverLoaded = true;
  1481         } catch (...) {
  1482             m_loadError = std::current_exception();

---

Lines 1482-1491

  1482             m_loadError = std::current_exception();
  1483         }
  1484     });
  1485     if (m_loadError) {
! 1486         std::rethrow_exception(m_loadError);
! 1487     }
  1488 }
  1489 
  1490 // SqlHandle definition
  1491 SqlHandle::SqlHandle(SQLSMALLINT type, SQLHANDLE rawHandle) : _type(type), _handle(rawHandle) {}

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.6%
mssql_python.__init__.py: 77.6%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.pybind.connection.connection_pool.cpp: 81.4%
mssql_python.pybind.connection.connection.cpp: 84.3%
mssql_python.logging.py: 85.5%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR finalizes the Phase 2 split of the bundled ODBC driver into the standalone mssql-python-odbc package, makes the external package required at runtime, and updates build/test pipelines to produce and consume the new wheel layout.

Changes:

• Make mssql_python_odbc required for driver resolution (no fallback to bundled mssql_python/libs), with clearer runtime errors.
• Update ODBC wheel packaging to include only the target platform’s libs/ subtree and enable single-host cross-building of all ODBC wheels.
• Refactor CI pipelines to build all ODBC wheels in one Windows stage, install the ODBC wheel before pytest in platform builds, and remove now-redundant “both wheels” installation test stages.

Reviewed changes

Copilot reviewed 17 out of 85 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_015_utf8_path_handling.py	Updates tests to locate driver libs in mssql_python_odbc (but still includes a fallback).
setup_odbc.py	Packages only the target platform’s ODBC libs/ subtree; adds env overrides for cross-building and forces wheel to trust supplied platform tags.
setup.py	Removes inclusion/exclusion rules for bundled libs/ since the driver is now shipped separately.
mssql_python/pybind/ddbc_bindings.cpp	Requires mssql_python_odbc and throws actionable errors if missing/incomplete; removes fallback behavior.
mssql_python/pybind/build.bat	Copies VC++ runtime from mssql_python_odbc/libs/.../vcredist instead of mssql_python/libs.
eng/pipelines/pr-validation-pipeline.yml	Updates validation paths to the new mssql_python_odbc/libs/... locations.
OneBranchPipelines/stages/wheel-installation-test-windows-stage.yml	Deleted: prior end-user “both wheels” installation test stage (Windows).
OneBranchPipelines/stages/wheel-installation-test-stage.yml	Deleted: prior end-user “both wheels” installation test stage (Linux).
OneBranchPipelines/stages/wheel-installation-test-macos-stage.yml	Deleted: prior end-user “both wheels” installation test stage (macOS).
OneBranchPipelines/stages/build-windows-single-stage.yml	Adds optional dependency/install of consolidated mssql-python-odbc wheel before pytest.
OneBranchPipelines/stages/build-odbc-windows-stage.yml	Deleted: replaced by a single-host “build all ODBC wheels” stage.
OneBranchPipelines/stages/build-odbc-macos-stage.yml	Deleted: replaced by a single-host “build all ODBC wheels” stage.
OneBranchPipelines/stages/build-odbc-linux-stage.yml	Deleted: replaced by a single-host “build all ODBC wheels” stage.
OneBranchPipelines/stages/build-odbc-all-stage.yml	New: builds all 7 ODBC wheels on one Windows agent via ODBC_TARGET_* overrides and validates wheel contents.
OneBranchPipelines/stages/build-macos-single-stage.yml	Adds optional dependency/install of consolidated mssql-python-odbc wheel before pytest.
OneBranchPipelines/stages/build-linux-single-stage.yml	Adds optional dependency/install of consolidated mssql-python-odbc wheel before pytest (via PIP_FIND_LINKS).
OneBranchPipelines/build-release-package-pipeline.yml	Always builds both packages; makes mssql-python stages depend on ConsolidateOdbc; swaps 7 ODBC stages for one build-all stage.
.gitignore	Stops ignoring mssql_python_odbc/libs/ now that it’s the committed source of truth.

Suppressed comments (2)

tests/test_015_utf8_path_handling.py:1

• This test helper still falls back to mssql_python when mssql_python_odbc is missing/incomplete, but the runtime resolver (GetOdbcLibsBaseDir) now requires the external package and throws if it’s missing or incomplete. The fallback makes the tests inconsistent with production behavior and can mask packaging issues. Recommendation (mandatory): remove the fallback and instead fail (or explicitly pytest.skip) when mssql_python_odbc is not importable or does not contain a complete libs/ payload for the platform.
  setup_odbc.py:1
• When ODBC_TARGET_PLATFORM_TAG is set, get_platform_info() returns an empty arch if ODBC_TARGET_ARCH is unset. That empty value is later used to build the target libs/ globs (e.g., libs/windows/{arch}), which can silently produce wrong/empty package_data and yield broken wheels. Recommendation (mandatory): validate the override inputs—require a non-empty, expected arch for win* and *linux* tags (and optionally validate allowed values per tag), and raise a clear error when the override is incomplete/invalid.

[Gaurav Sharma (bewithgaurav)]

need to add some failure case tests, and some improvement suggestions

Please review it again