Skip to content

Conversation

@peter-quix
Copy link
Contributor

This PR updates all samples to use QuixStreams 3.23.1 and implements a comprehensive test infrastructure for sources, transformations, and destinations.

Key Changes

QuixStreams Updates (41 samples)

  • Updated all requirements.txt files from various versions (2.9.0, 3.x) to 3.23.1
  • Modernized API usage: app.run() instead of app.run(sdf) or app.run(dataframe=sdf)
  • Updated imports: from quixstreams import Application instead of from quixstreams.app import Application
  • Fixed QuixEnvironmentSource to use new API with app_config parameter

Sample Improvements

  • TDengine: Added support for both token (Cloud) and username/password (On-Premise) authentication; fixed subtable naming
  • SQL CDC: Complete rewrite with new cdc_helper.py abstraction, improved error handling, better snapshot and CDC logic
  • S3 Source: Refactored for better error handling and file processing
  • MQTT/HiveMQ Destinations: Added configurable TLS support, configurable consumer groups, message retention
  • HiveMQ Source: Added CA certificate support for TLS
  • Demo Data Source: Added MAX_ROWS environment variable for limiting output
  • Environment Source: Updated to use new QuixEnvironmentSource API
  • Confluent Kafka: Added configurable consumer groups
  • Redis/Slack/S3-Iceberg: Added configurable consumer groups and AWS session token support

Test Infrastructure (30+ test suites)

  • Added Docker Compose test configurations for automated testing
  • Implemented produce_test_data.py scripts for sources
  • Implemented verify_output.py scripts for validating results
  • Added setup scripts for complex dependencies (TDengine, SQL Server, PostgreSQL, InfluxDB, etc.)
  • Comprehensive tests for: MQTT, HiveMQ, TDengine, Elasticsearch, InfluxDB (1 & 3), MongoDB, PostgreSQL, Redis, WebSocket, OPC UA, S3, Segment, SQL CDC, and more

Cleanup

  • Removed duplicate http_api_sample (kept http_api_source)

Testing

All samples have been updated with test infrastructure to validate functionality against actual service dependencies using Docker Compose.

Peter Nagy added 7 commits October 2, 2025 14:31
http_api_sample is near identical to http_api_source, therefore consolidating
smaller changes in some of the samples to allow for testing
Some refactoring was necessary to make it work
return empty string instead of None for subtable names to prevent invalid line protocol
@gitguardian
Copy link

gitguardian bot commented Oct 2, 2025

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
21348440 Triggered Generic Private Key 68ed698 tests/sources/hivemq/certs/ca.key View secret
21328423 Triggered Generic Password 6447f76 tests/destinations/TDengine/docker-compose.test.yml View secret
21348441 Triggered Generic Private Key 68ed698 tests/sources/hivemq/certs/server.key View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Copy link
Contributor

@PatrickMiraP PatrickMiraP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the reasons HTTP sample is duplicated is because one was living in Connectors option and the other one in Templates.

I don't know if there is still some value on having something like an HTTP template on Templates option rather than ask the user to use the one in Connectors.

@peter-quix peter-quix merged commit 2cc816b into main Oct 2, 2025
2 checks passed
@peter-quix peter-quix deleted the task/update-quixsterams-re branch October 2, 2025 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants