Test library execution against Clickhouse engine

There has been a couple of recent issues in which test execution failed
in certain scenarios (#48 and #53). These weren't caught by the
libraries test as it would require executing actual tests against actual
SQL queries. This commit will add a framework for running these tests
against a running database engine (Clickhouse in the first instance) to
better enable these kinds of bugs to be caught.

Author: smoothml

.github/workflows/test-package.yml

@@ -16,6 +16,12 @@ jobs:
       fail-fast: false
       matrix:
         python-version: ["3.9", "3.10", "3.11"]
+    services:
+      clickhouse:
+        image: clickhouse/clickhouse-server:24.1.5.6
+        ports:
+          - 8123:8123
+          - 9000:9000
 
     steps:
     - uses: actions/checkout@v3
@@ -36,5 +42,9 @@ jobs:
         # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
         poetry run flake8 . --count --exit-zero --max-complexity=10 --max-line-length=119 --statistics --ignore E203,E266,E501,W503
     - name: Test with pytest
+      env:
+        SQL_MOCK_CLICKHOUSE_HOST: 127.0.0.1
+        SQL_MOCK_CLICKHOUSE_PORT: 8123
+        SQL_MOCK_CLICKHOUSE_USER: default
       run: |
         poetry run pytest tests/

CONTRIBUTION.md

@@ -45,22 +45,42 @@ Once you have Poetry, you can install the project's dependencies:
 poetry install --all-extras
 ```
 
-### 3. Pre-Commit Hooks
+### 3. External Dependencies
 
-This project uses pre-commit hooks to ensure code quality. To install the hooks, run:
+This project uses several external dependencies in the developer workflow. These are:
+
+- [Pre-commit](https://pre-commit.com/) hooks are used to ensure code quality. Install the hooks with
+    ```
+    poetry run pre-commit install
+    ```
+    This will set up the necessary hooks to check code formatting, linting, and other code quality checks before each commit.
+- [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/) for running integrations tests (see below).
+- [Make](https://www.gnu.org/software/make/) is used as convenience wrapper around some of the more convoluted commands. This is optional as you can always run the commands directly (see the `Makefile` for the actual commands being run).
+
+### 4. Running Tests
+
+We use [pytest](https://docs.pytest.org/en/latest/) for running tests. You can run all the tests with:
 
 ```bash
-poetry run pre-commit install
+make test
 ```
 
-This will set up the necessary hooks to check code formatting, linting, and other code quality checks before each commit.
+There are two types of tests: those testing the internal functions and methods of the library and those testing the test execution with example queries. The latter require a running database instance to be available (either locally or remotely). Note, integration testing is currently only supported for Clickhouse. You can run internal testing only with:
 
-### 4. Running Tests
+```bash
+make test-unit
+```
 
-We use [pytest](https://docs.pytest.org/en/latest/) for running tests. You can run all the tests with:
+Running integration tests locally requires [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/). First, start the local database services with:
+
+```bash
+docker compose up -d
+```
+
+Then you can run integration tests with:
 
 ```bash
-poetry run pytest tests/
+make test-integration
 ```
 
 ### 5. Environment Variables

Makefile

@@ -0,0 +1,18 @@
+.DEFAULT_GOAL := help
+SHELL := /bin/bash
+
+.PHONY: help
+help: ## Show all available commands
+	@awk 'BEGIN {FS = ":.*##"; printf "Usage: make \033[36m<target>\033[0m\n"} /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-13s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST);
+
+.PHONY: test
+test: ## Run test pipeline
+	poetry run pytest tests/
+
+.PHONY: test-integration
+test-integration: ## Run integration tests
+	poetry run pytest -m "integration" tests/
+
+.PHONY: test-unit
+test-unit: ## Run unit tests
+	poetry run pytest -m "not integration" tests/

compose.yaml

@@ -0,0 +1,10 @@
+services:
+  clickhouse:
+    image: clickhouse/clickhouse-server:24.1.5.6
+    ports:
+      - 8123:8123
+      - 9000:9000
+    ulimits:
+      nofile:
+        soft: "262144"
+        hard: "262144"

pyproject.toml

@@ -75,9 +75,12 @@ flake8 = "^6.1.0"
 requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
 
-# Ignore Google Bigquery namespace deprecation warnings
 [tool.pytest.ini_options]
+# Ignore Google Bigquery namespace deprecation warnings
 filterwarnings = [
     "ignore:Deprecated call to `pkg_resources\\.declare_namespace\\('.*'\\):DeprecationWarning",
     "ignore::DeprecationWarning:google.rpc",
 ]
+markers = [
+    "integration: Integration tests requiring running database instances"
+]

tests/sql_mock/clickhouse/test_query_execution.py

@@ -0,0 +1,57 @@
+import os
+
+import pytest
+
+from sql_mock.clickhouse import column_mocks as col
+from sql_mock.clickhouse.table_mocks import ClickHouseTableMock
+from sql_mock.table_mocks import table_meta
+
+pytestmark = pytest.mark.integration
+
+
+@pytest.fixture(autouse=True)
+def set_env():
+    if not os.getenv("SQL_MOCK_CLICKHOUSE_HOST"):
+        os.environ["SQL_MOCK_CLICKHOUSE_HOST"] = "localhost"
+    if not os.getenv("SQL_MOCK_CLICKHOUSE_PORT"):
+        os.environ["SQL_MOCK_CLICKHOUSE_PORT"] = "8123"
+    if not os.getenv("SQL_MOCK_CLICKHOUSE_USER"):
+        os.environ["SQL_MOCK_CLICKHOUSE_USER"] = "default"
+    if not os.getenv("SQL_MOCK_CLICKHOUSE_PASSWORD"):
+        os.environ["SQL_MOCK_CLICKHOUSE_PASSWORD"] = ""
+
+
+def test_simple_query():
+    query = """SELECT
+        user_id,
+        count() AS sessions
+    FROM sessions
+    GROUP BY user_id
+    """
+
+    @table_meta(table_ref="sessions")
+    class SessionsMock(ClickHouseTableMock):
+        user_id = col.String(default="foo")
+
+    @table_meta(query=query)
+    class ResultMock(ClickHouseTableMock):
+        user_id = col.String(default="foo")
+        sessions = col.Int(default=0)
+
+    sessions_mock = SessionsMock.from_dicts(
+        [
+            {"user_id": "a"},
+            {"user_id": "a"},
+            {"user_id": "a"},
+            {"user_id": "b"},
+        ],
+    )
+
+    result = ResultMock.from_mocks(input_data=[sessions_mock])
+
+    expected = [
+        {"user_id": "a", "sessions": 3},
+        {"user_id": "b", "sessions": 1},
+    ]
+
+    result.assert_equal(expected)