independently configurable logging for different modules

[ashwinidulams]

Summary

Today we have one global logger for all of the numaflow. As the complexity increases and we add more features, debugging issues via logs will get harder. for example, if one wants to debug Kafka source issues and enables DEBUG level, there will be a flood of DEBUG messages from all the modules in the pod (vertex).

What would be ideal is if we could selectively enable levels for different features in the pipeline.

Use Cases

When would you use this?

debugging watermark issues does not need logs from all the modules. similarly debugging GRPC Interface between numa container and UDF does not need any watermark logs.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.