Enhanced Rest Data Sink Component with ability to add headers to request, retry the request on failure, and upgrade to current streampipes interfaces

[heisenbergs-uncertainty]

Purpose

This enhances the currently implemented RestPublisher Data Sink which has very little functionality outside of making a request. Many implementations of REST functionality are going to be utilizing the sink to trigger a webhook which should at bare minimum use authorization to confirm the legitimacy of the request origin. This enhancement allows for the addition of custom headers so that more advanced REST scenarios can be addressed with the RestPublisher. I have also added retry functionality on failure, timeouts, and better logging for more advanced debugging.

This also upgrades the interfaces on from older streampipes models to the newer "IDataSinkInterface" as demonstrated by the Streampipes tutorial documentation.

Here is a picture of what the new sink looks like in the UI:

Here is an example of this same request hitting RequestBin:

Remarks

No remarks.

PR introduces (a) breaking change(s): no

PR introduces (a) deprecation(s): no

[tenthe]

Hi @heisenbergs-uncertainty,

thanks a lot for your PR! The changes are highly welcome. I really appreciate the update to the new API and the additional functionality for the sink.

Required Changes

There are some changes required to ensure that this PR works and that existing instances of the RestSink are migrated correclty.

• Checkstyle Issues: There are some checkstyle problems—please check the GitHub Actions logs for details.
• Version Update: Update the sink version to 1 (see line 74).
• Migration Script: A migration script is required to update existing sinks in pipelines. You can refer to DataLakeSinkMigrationV2 for guidance.
• Displayed Labels: Move static property texts to string.en in resource directory org.apache.streampipes.sinks.brokers.jvm.rest. Use the Labels.withId API—see the URL_KEY parameter of the RestSink before the changes for reference.

Please let us know if you need any assistance or if some of the changes I described are not clear.

Suggestion

What do you think about making MAX_RETRIES and RETRY_DELAY_MS user-configurable with default values? This would allow users to adjust them as needed.

Thank you again for your PR and we look forward to working with you.

[heisenbergs-uncertainty]

Hi @heisenbergs-uncertainty,

thanks a lot for your PR! The changes are highly welcome. I really appreciate the update to the new API and the additional functionality for the sink.

Required Changes

There are some changes required to ensure that this PR works and that existing instances of the RestSink are migrated correclty.

* **Checkstyle Issues**: There are some checkstyle problems—please check the GitHub Actions logs for details.

* **Version Update**: Update the sink version to **1** (see line 74).

* **Migration Script**: A migration script is required to update existing sinks in pipelines. You can refer to [`DataLakeSinkMigrationV2`](https://github.com/apache/streampipes/blob/43573850423475a429f517db6502fbe30d1a25b8/streampipes-extensions/streampipes-sinks-internal-jvm/src/main/java/org/apache/streampipes/sinks/internal/jvm/datalake/migrations/DataLakeSinkMigrationV2.java#L33) for guidance.

* **Displayed Labels**: Move static property texts to `string.en` in resource directory `org.apache.streampipes.sinks.brokers.jvm.rest`. Use the `Labels.withId` API—see the `URL_KEY` parameter of the `RestSink` before the changes for reference.

Please let us know if you need any assistance or if some of the changes I described are not clear.

Suggestion

What do you think about making MAX_RETRIES and RETRY_DELAY_MS user-configurable with default values? This would allow users to adjust them as needed.

Thank you again for your PR and we look forward to working with you.

Thank you for the suggestions! This is helpful to figure out how to clean this up and get the PR passing. I'll get the version update and migration script done asap.

The displayed labels comment makes a lot of sense and answers some of my confusion. I spent a long time looking for why so many components were using "withId" instead of "from". It would be helpful to include this in the "Extending Streampipes" tutorial section on the website. I could have also just missed it while skimming but I did look for a decent period of time and couldn't understand its usage.

In regards to the retry delay, I had this same thought and was planning on adding it later on but I will go ahead and include it now! Makes sense to include this in case anybody doesn't want the retry functionality up front.

[heisenbergs-uncertainty]

@tenthe I could use help with conditional ui elements and how to add them.

For example, the user input should be required for RETRY_DELAY_MS and NUM_RETRY integers if and only if the user has enabled them via the IS_RETRY_ENABLED boolean (or single selection... whatever suits better)

[dominikriemer]

Hi @heisenbergs-uncertainty thanks for your contribution.
We currently don't really have a concept for conditional ui elements, but since we've discussed this feature many times in the past, I'll be working on adding such a UI element which you can then also use in the REST sink.
I think it will be a combination of a slider and an element group where arbitrary other ui elements can be included.

[heisenbergs-uncertainty]

Hi @heisenbergs-uncertainty thanks for your contribution.

We currently don't really have a concept for conditional ui elements, but since we've discussed this feature many times in the past, I'll be working on adding such a UI element which you can then also use in the REST sink.

I think it will be a combination of a slider and an element group where arbitrary other ui elements can be included.

Sounds good. I'll just add documentation for it and maybe submit an issue as a reminder to me to update it to the conditional component once that's complete. Functionality should be done at this point and I just need to update the migration scripts.

[heisenbergs-uncertainty]

@tenthe @dominikriemer I might need help on the checkstyle issues. I don't typically work in Java and I am not totally certain of IDEA IDE and I am having trouble getting the formatting to work correctly with the checkstyle.xml. It seems like all of my checkstyle errors are currently related to indentations.