Open
Description
Background
Some clients encode arbitrary precision numerical types as raw bytes when stored in Kafka. While QuestDB doesn't yet have native support for arbitrary precision Numeric types, many of these values can be effectively represented using QuestDB's Long256 type. Currently, this conversion requires implementing third-party transforms.
Current Workarounds
Users currently rely on external transforms to handle this conversion. Open source examples:
Proposed Enhancement
Integrate a native Bytes-To-Long256 trancoder directly into the QuestDB Kafka connector to:
- Simplify the data ingestion pipeline for users working with arbitrary precision numbers
- Eliminate the need for external dependencies
- Provide a consistent, well-tested implementation that matches QuestDB's behavior
- Improve overall user experience for financial and blockchain use cases where Long256 is commonly used
Notes
This is an interim solution until QuestDB implements full Numeric type support, but would immediately benefit users working with large integers, cryptocurrency hashes, and similar use cases.