PD-257019 Changes for emodb stabilization, kafka integration and stepfunction firing

[rishuyadavbv]

Pull Request: Integrate changes for emodb stabilization, kafka integration and stepfunction firing

Description:

As part of the Emo Stabilization Phase 1, this PR integrates the Kafka Producer and AdminClient into the Queue Module to ensure robust messaging and topic management. The goal is sending messages to Kafka, handle automatic Kafka topic creation, firing the stepfunction execution, and store the incoming throttled messages from kafka to cassandra

---

Key Changes:

1. Kafka Producer & AdminClient Integration:

   • Implemented the Kafka Producer and AdminClient to support messaging operations in the Queue Module.
   • Messages are now sent to Kafka using the sendAll function, which handles the topic creation and stepfn firing as well

2. Automatic Topic Creation:

   • Added logic to automatically create Kafka topics if they don't already exist.
   • Fixed the topic creation mechanism by checking for topic existence before sending messages, ensuring smooth operation.

3. Step Function Execution:

   • Introduced functionality to trigger a Step Function execution when a new Kafka topic is created for the first time.
   • Step Function executions are triggered with a well-defined name and appropriate input payload.

4. AWS Parameter Store Integration:

   • Integrated AWS Parameter Store to securely fetch configurations for both Kafka and Step Function services.
   • This improves the maintainability and security of managing sensitive configurations.

5. New Endpoint for Cassandra Queue Service:

   • Created a new API endpoint sendBatch1 to allow direct sending of messages to the Cassandra Queue Service.
   • This endpoint will be called by the Throttler Service in a rate-limited manner

---

Additional Notes:

• Refactored the Queue Module for better separation of concerns, ensuring that Kafka and Cassandra interactions are modular and maintainable.
• Tested the topic creation, message sending, and Step Function executions in a dev environment.

---

Checklist:

• Kafka Producer and AdminClient integrated
• Automatic Kafka topic creation implemented and verified
• Step Function execution for new topics tested
• AWS Parameter Store integrated for config management
• Code refactored for better separation of concerns
• New endpoint sendBatch1 for Cassandra created and tested