TIDESQL
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

A pluggable write and space optimized storage engine for MariaDB using TidesDB.


INSTALLATION
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

You can choose to run install.sh, for instructions run ./install.sh --help

The install script will build TidesDB and MariaDB from source.
You can specify the versions if you want.  The script will install everything
and make TidesDB available as an engine to utilize.

Below are manual install instructions for those who want to configure everything
themselves, though with the install script you can modify configuration files
after the fact.

LINUX (Ubuntu/Debian)
---------------------

1. Install dependencies:

   sudo apt update
   sudo apt install -y build-essential cmake ninja-build bison flex \
     libzstd-dev liblz4-dev libsnappy-dev libncurses-dev libssl-dev \
     libxml2-dev libevent-dev libcurl4-openssl-dev pkg-config

2. Install TidesDB library:

   git clone --depth 1 https://github.com/tidesdb/tidesdb.git tidesdb-lib
   cd tidesdb-lib && mkdir build && cd build
   cmake ..
   make -j$(nproc)
   sudo make install
   sudo ldconfig

3. Clone MariaDB and copy TidesDB storage engine:

   git clone --depth 1 --branch 12.1 https://github.com/MariaDB/server.git mariadb-server
   cd mariadb-server
   git submodule update --init --recursive
   cp -r /path/to/tidesql/tidesdb storage/
   cp -r /path/to/tidesql/mysql-test/suite/tidesdb mysql-test/suite/

4. Build MariaDB:

   mkdir build && cd build
   cmake ..
   make -j$(nproc)

5. Run tests (from the build directory):

   cd mysql-test
   perl mtr --suite=tidesdb --parallel=4 --force


MACOS
-----

1. Install dependencies:

   brew install cmake ninja bison snappy lz4 zstd openssl@3

2. Install TidesDB library:

   # Ensure Xcode SDK is used (not CommandLineTools)
   sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
   export SDKROOT=$(xcrun --show-sdk-path)

   git clone --depth 1 https://github.com/tidesdb/tidesdb.git tidesdb-lib
   cd tidesdb-lib && mkdir build && cd build
   cmake .. -DCMAKE_OSX_SYSROOT=${SDKROOT} -DOPENSSL_ROOT_DIR=$(brew --prefix openssl@3)
   make -j$(sysctl -n hw.ncpu)
   sudo make install

3. Clone MariaDB and copy TidesDB storage engine:

   git clone --depth 1 --branch 12.1 https://github.com/MariaDB/server.git mariadb-server
   cd mariadb-server
   git submodule update --init --recursive
   cp -r /path/to/tidesql/tidesdb storage/
   cp -r /path/to/tidesql/mysql-test/suite/tidesdb mysql-test/suite/

4. Build MariaDB:

   # Remove CommandLineTools SDK to prevent header conflicts
   sudo rm -rf /Library/Developer/CommandLineTools/SDKs/MacOSX*.sdk

   export SDKROOT=$(xcrun --show-sdk-path)
   export CC=$(xcrun -find clang)
   export CXX=$(xcrun -find clang++)

   mkdir build && cd build
   cmake .. \
     -DCMAKE_C_COMPILER=${CC} \
     -DCMAKE_CXX_COMPILER=${CXX} \
     -DCMAKE_OSX_SYSROOT=${SDKROOT} \
     -DCMAKE_C_FLAGS="-Wno-nullability-completeness" \
     -DCMAKE_CXX_FLAGS="-Wno-nullability-completeness" \
     -DWITH_SSL=bundled \
     -DWITH_PCRE=bundled \
     -G Ninja
   cmake --build . --parallel $(sysctl -n hw.ncpu)

   Note: The CommandLineTools SDK removal is needed because CMake may find
   headers in /Library/Developer/CommandLineTools/SDKs/MacOSX*.sdk which
   causes C/C++ header path conflicts with libc++.

5. Run tests (from the build directory):

   cd mysql-test
   perl mtr --suite=tidesdb --parallel=4 --force


WINDOWS
-------

1. Install prerequisites:

   - Visual Studio 2022 with C++ workload
   - CMake (choco install cmake)
   - Strawberry Perl (choco install strawberryperl)
   - WinFlexBison (download from GitHub releases)

2. Install vcpkg dependencies:

   cd C:\vcpkg
   git pull
   .\vcpkg.exe install zstd:x64-windows lz4:x64-windows snappy:x64-windows pthreads:x64-windows

3. Install TidesDB library:

   git clone --depth 1 https://github.com/tidesdb/tidesdb.git tidesdb-lib
   cd tidesdb-lib
   mkdir build && cd build
   cmake .. -G "Visual Studio 17 2022" -A x64 ^
     -DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake ^
     -DCMAKE_INSTALL_PREFIX=C:/tidesdb
   cmake --build . --config Release
   cmake --install . --config Release

4. Clone MariaDB and copy TidesDB storage engine:

   git clone --depth 1 --branch 12.1 https://github.com/MariaDB/server.git mariadb-server
   cd mariadb-server
   git submodule update --init --recursive
   xcopy /E /I path\to\tidesql\tidesdb storage\tidesdb
   xcopy /E /I path\to\tidesql\mysql-test\suite\tidesdb mysql-test\suite\tidesdb

5. Build MariaDB:

   mkdir build && cd build
   cmake .. -G "Visual Studio 17 2022" -A x64 ^
     -DCMAKE_PREFIX_PATH="C:/tidesdb;C:/vcpkg/installed/x64-windows" ^
     -DCONNECT_DYNAMIC=NO
   cmake --build . --config RelWithDebInfo --parallel

   Note: CONNECT plugin is disabled (-DCONNECT_DYNAMIC=NO) because it
   requires libxml2 headers that may conflict with vcpkg installations.

6. Run tests:

   cd mysql-test
   perl mtr --suite=tidesdb --parallel=4 --force


ENABLE PLUGIN
-------------

After building, enable the plugin in MariaDB:

   INSTALL SONAME 'ha_tidesdb';  -- works on all platforms



FEATURES
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

Core:
- Per-statement transactions with snapshot isolation
- Lock-free concurrency (no THR_LOCK, TidesDB handles isolation internally)
- LSM-tree storage with optional B+tree SSTable format
- Compression (NONE, LZ4, LZ4_FAST, ZSTD, Snappy)
- Bloom filters for fast key lookups
- Block cache for frequently accessed data
- Primary key and secondary index support
- TTL (time-to-live) per-row and per-table expiration
- Virtual/generated columns
- Online backup (SET GLOBAL tidesdb_backup_dir = '/path')
- Partitioning (RANGE, LIST, HASH, KEY)
- Data-at-rest encryption (MariaDB encryption plugin integration)
- Online DDL (instant metadata, inplace add/drop index, copy for columns)
- TRUNCATE TABLE as O(1) drop+recreate (instant regardless of table size)
- ANALYZE TABLE with detailed CF statistics (levels, keys, sizes, cache hit rate)
- Cached optimizer statistics (refreshed every 2 seconds from TidesDB)


CONFIGURATION
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

TidesDB stores its data as a sibling of the MariaDB data directory:
  <parent_of_datadir>/tidesdb_data


SYSTEM VARIABLES (SET GLOBAL tidesdb_...)
-----------------------------------------

All are read-only (set at startup) unless noted otherwise.

  flush_threads             Background flush threads (default: 2)
  compaction_threads        Background compaction threads (default: 2)
  log_level                 DEBUG/INFO/WARN/ERROR/FATAL/NONE (default: WARN)
  block_cache_size          Global block cache in bytes (default: 256MB)
  max_open_sstables         Max cached SSTable structures (default: 512)
  backup_dir                [dynamic] Set to a path to trigger online backup
  debug_trace               [dynamic] Per-operation trace logging (default: OFF)


TABLE OPTIONS (CREATE TABLE ... ENGINE=TidesDB <option>=<value>)
----------------------------------------------------------------

These are per-table options baked into the column family at creation time.
Changing them via ALTER TABLE only updates the .frm; it does not reconfigure
the live column family.

Storage:
  WRITE_BUFFER_SIZE         Memtable size before flush (default: 32MB)
  MIN_DISK_SPACE            Minimum free disk space (default: 100MB)
  KLOG_VALUE_THRESHOLD      Values larger than this go to vlog (default: 512)

Compression:
  COMPRESSION               NONE/SNAPPY/LZ4/ZSTD/LZ4_FAST (default: LZ4)

Bloom Filters:
  BLOOM_FILTER              Enable bloom filters (default: ON)
  BLOOM_FPR                 FPR in parts per 10000; 100 = 1% (default: 100)

Durability:
  SYNC_MODE                 NONE/INTERVAL/FULL (default: FULL)
  SYNC_INTERVAL_US          Sync interval in microseconds (default: 128000)

Isolation:
  ISOLATION_LEVEL           READ_UNCOMMITTED/READ_COMMITTED/REPEATABLE_READ/
                            SNAPSHOT/SERIALIZABLE (default: REPEATABLE_READ)

LSM Tree:
  USE_BTREE                 Use B+tree SSTable format (default: OFF)
  LEVEL_SIZE_RATIO          Level size multiplier (default: 10)
  MIN_LEVELS                Minimum LSM levels (default: 5)
  DIVIDING_LEVEL_OFFSET     Compaction dividing level offset (default: 2)
  L1_FILE_COUNT_TRIGGER     L1 file count trigger for compaction (default: 4)
  L0_QUEUE_STALL_THRESHOLD  L0 queue stall threshold (default: 20)

Skip List:
  SKIP_LIST_MAX_LEVEL       Max skip list level (default: 12)
  SKIP_LIST_PROBABILITY     Percentage; 25 = 0.25 (default: 25)

Block Indexes:
  BLOCK_INDEXES             Enable block indexes (default: ON)
  INDEX_SAMPLE_RATIO        Sample ratio for block index (default: 1)
  BLOCK_INDEX_PREFIX_LEN    Block index prefix length (default: 16)

TTL:
  TTL                       Default TTL in seconds, 0 = none (default: 0)

Encryption:
  ENCRYPTED                 Enable data-at-rest encryption (default: OFF)
  ENCRYPTION_KEY_ID         Encryption key ID (default: 1)


FIELD OPTIONS (per-column)
--------------------------

  TTL                       Marks column as per-row TTL source (seconds)

Example:
  CREATE TABLE t (
    id INT PRIMARY KEY,
    data VARCHAR(100),
    expires INT TTL=1
  ) ENGINE=TIDESDB TTL=3600 SYNC_MODE='NONE' COMPRESSION='ZSTD';


TESTING
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

Run all TidesDB tests (from the build directory):

  cd mysql-test
  perl mtr --suite=tidesdb

Run specific test:

  perl mtr --suite=tidesdb tidesdb_crud

Available tests:
  tidesdb_crud          Basic CRUD operations
  tidesdb_pk_index      Primary key and secondary index scans
  tidesdb_options       Table and field options
  tidesdb_ttl           Time-to-live expiration
  tidesdb_vcol          Virtual/generated columns
  tidesdb_encryption    Data-at-rest encryption
  tidesdb_backup        Online backup
  tidesdb_partition     RANGE/LIST/HASH/KEY partitioning
  tidesdb_online_ddl    Online DDL (instant, inplace, copy)
  tidesdb_analyze       ANALYZE TABLE with CF statistics output
  tidesdb_rename        Table rename

Run with verbose output:

  perl mtr --suite=tidesdb --verbose


QUICK TEST
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

  CREATE TABLE t (id INT PRIMARY KEY, data VARCHAR(100)) ENGINE=TidesDB;
  INSERT INTO t VALUES (1, 'hello'), (2, 'world');
  SELECT * FROM t;
  DROP TABLE t;


SANITIZER BUILDS
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

The TidesDB plugin supports plugin-level UBSAN/ASAN without rebuilding the
entire MariaDB server:

  cmake .. -DTIDESDB_WITH_UBSAN=ON   # UBSAN only
  cmake .. -DTIDESDB_WITH_ASAN=ON    # ASAN only
  cmake .. -DTIDESDB_WITH_ASAN=ON -DTIDESDB_WITH_UBSAN=ON  # both
  make -j$(nproc) tidesdb

  # Run tests with sanitizer error logging:
  UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=0 \
    perl ./mtr --suite=tidesdb --parallel=4

For full ASAN coverage (requires full server rebuild):

  cmake .. -DWITH_ASAN=ON -DWITH_UBSAN=ON -DCMAKE_BUILD_TYPE=Debug
  make -j$(nproc)


LICENSE
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

GNU General Public License v2