[FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source #119

RocMarshal · 2024-05-10T14:52:26Z

Introduce getLatestOptionalState & setOptionalState for JdbcParameterValuesProvider to process provider state.
Introduce SlideTimingParamameterProvider for supporting continuous splits
Introduce streaming semantic based on ResultSet.TYPE_SCROLL_INSENSITIVE
Introduce JdbcSourceStreamRelatedITCase for testing.

RocMarshal · 2024-05-22T03:56:21Z

Hi, @eskabetxe @leonardBang @Jiabao-Sun Could you help have a look if you had the free time ?
Thank you.

...-jdbc/src/main/java/org/apache/flink/connector/jdbc/utils/ContinuousEnumerationSettings.java

...jdbc/src/test/java/org/apache/flink/connector/jdbc/source/JdbcSourceStreamRelatedITCase.java

eskabetxe

LGTM

...r-jdbc/src/main/java/org/apache/flink/connector/jdbc/utils/ContinuousUnBoundingSettings.java

… the new JDBC source

1996fanrui · 2024-06-06T06:46:29Z

docs/content/docs/connectors/datastream/jdbc.md


-## `JdbcSink.sink`
+## Source of JDBC Connector(Experimental)


Suggested change

## Source of JDBC Connector(Experimental)

## Source of JDBC Connector(Experimental)

This part is not needed.

1996fanrui · 2024-06-06T06:50:32Z

docs/content/docs/connectors/datastream/jdbc.md

+Configuration goes as follow (see also {{< javadoc file="org/apache/flink/connector/jdbc/source/JdbcSource.html" name="JdbcSource javadoc" >}}
+and {{< javadoc file="org/apache/flink/connector/jdbc/source/JdbcSourceBuilder.html" name="JdbcSourceBuilder javadoc" >}}).
+
+### `JdbcSource.builder`


Suggested change

### `JdbcSource.builder`

## Jdbc Source

Jdbc Source can be as the second level instead of the third level. You can refer Kafka connector doc[1]. Kafka Source, Kafka SourceFunction and Kafka Producer as the second level. We don't need to distinguish the source or sink. (Current level is a little fine-grained.)

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#kafka-source

1996fanrui · 2024-06-06T06:51:28Z

docs/content/docs/connectors/datastream/jdbc.md

+```
+
+#### Minimalist Streaming Semantic and ContinuousUnBoundingSettings
+If you want to generate continuous milliseconds parameters based on sliding-window,


Suggested change

If you want to generate continuous milliseconds parameters based on sliding-window,

If you want to generate continuous milliseconds parameters based on sliding-window,

Add the empty line.

RocMarshal

Thanks @1996fanrui for the review.
I updated based on your comments.
PTAL if you had the free time~
CC @eskabetxe

…w Jdbc source.

1996fanrui

Thanks @RocMarshal for the hard work, and thanks @eskabetxe for the review!

LGTM, I will merge it next Tuesday(in 4 days) if there are no objections.

RocMarshal · 2024-06-14T03:06:24Z

👍 Thank you @1996fanrui @eskabetxe very much for your review and attention for the feature !

boring-cyborg bot added the component=Connectors/JDBC label May 10, 2024

RocMarshal force-pushed the FLINK-33461 branch 2 times, most recently from a2f6633 to a2fb896 Compare May 10, 2024 15:51

RocMarshal closed this May 10, 2024

RocMarshal reopened this May 10, 2024

RocMarshal changed the title ~~Flink 33461~~ [FLINK-33461][Connector/JDBC] Support streaming related semantics for the new jdbc source May 11, 2024

RocMarshal force-pushed the FLINK-33461 branch from a2fb896 to 6d5c69d Compare May 11, 2024 01:47

RocMarshal changed the title ~~[FLINK-33461][Connector/JDBC] Support streaming related semantics for the new jdbc source~~ [FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source May 11, 2024

RocMarshal force-pushed the FLINK-33461 branch from 6d5c69d to 2a54048 Compare May 17, 2024 11:04

RocMarshal marked this pull request as ready for review May 17, 2024 11:10

RocMarshal force-pushed the FLINK-33461 branch 6 times, most recently from 8f661db to e14152d Compare May 21, 2024 15:27

eskabetxe reviewed May 27, 2024

View reviewed changes

...-jdbc/src/main/java/org/apache/flink/connector/jdbc/utils/ContinuousEnumerationSettings.java Outdated Show resolved Hide resolved

...jdbc/src/test/java/org/apache/flink/connector/jdbc/source/JdbcSourceStreamRelatedITCase.java Outdated Show resolved Hide resolved

RocMarshal force-pushed the FLINK-33461 branch 5 times, most recently from 437372f to 2361ebb Compare May 28, 2024 05:13

RocMarshal requested a review from eskabetxe May 28, 2024 06:02

RocMarshal force-pushed the FLINK-33461 branch 2 times, most recently from 2fb94f6 to e00fe68 Compare May 28, 2024 06:59

RocMarshal mentioned this pull request May 30, 2024

[FLINK-33462][Connector/JDBC] Sort out the document page about the new Jdbc source. #122

Closed

1 task

RocMarshal force-pushed the FLINK-33461 branch from 42274c7 to 08a863c Compare May 30, 2024 01:49

boring-cyborg bot added the component=Documentation label May 30, 2024

eskabetxe approved these changes May 30, 2024

View reviewed changes

1996fanrui reviewed May 30, 2024

View reviewed changes

...r-jdbc/src/main/java/org/apache/flink/connector/jdbc/utils/ContinuousUnBoundingSettings.java Show resolved Hide resolved

RocMarshal force-pushed the FLINK-33461 branch 3 times, most recently from 5fd7b4a to 83365e3 Compare June 2, 2024 11:43

1996fanrui self-assigned this Jun 3, 2024

RocMarshal force-pushed the FLINK-33461 branch 5 times, most recently from 1f6a037 to 4b056a6 Compare June 6, 2024 03:08

[FLINK-33461][Connector/JDBC] Support streaming related semantics for…

b5b0caf

… the new JDBC source

RocMarshal force-pushed the FLINK-33461 branch from 4b056a6 to c551dcb Compare June 6, 2024 04:51

1996fanrui reviewed Jun 6, 2024

View reviewed changes

RocMarshal force-pushed the FLINK-33461 branch 2 times, most recently from 3508075 to 78efce0 Compare June 6, 2024 08:23

RocMarshal commented Jun 7, 2024

View reviewed changes

[FLINK-33462][Connector/JDBC] Sort out the document page about the ne…

a865eb0

…w Jdbc source.

RocMarshal force-pushed the FLINK-33461 branch from 78efce0 to a865eb0 Compare June 7, 2024 02:46

1996fanrui approved these changes Jun 7, 2024

View reviewed changes

1996fanrui merged commit 1bab533 into apache:main Jun 14, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source #119

[FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source #119

RocMarshal commented May 10, 2024 •

edited

Loading

RocMarshal commented May 22, 2024 •

edited

Loading

eskabetxe left a comment

1996fanrui Jun 6, 2024

1996fanrui Jun 6, 2024

1996fanrui Jun 6, 2024

RocMarshal left a comment

1996fanrui left a comment

RocMarshal commented Jun 14, 2024

	## Source of JDBC Connector(Experimental)
	## Source of JDBC Connector(Experimental)

	If you want to generate continuous milliseconds parameters based on sliding-window,

	If you want to generate continuous milliseconds parameters based on sliding-window,

[FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source #119

[FLINK-33461][Connector/JDBC] Support streaming related semantics for the new JDBC source #119

Conversation

RocMarshal commented May 10, 2024 • edited Loading

RocMarshal commented May 22, 2024 • edited Loading

eskabetxe left a comment

Choose a reason for hiding this comment

1996fanrui Jun 6, 2024

Choose a reason for hiding this comment

1996fanrui Jun 6, 2024

Choose a reason for hiding this comment

1996fanrui Jun 6, 2024

Choose a reason for hiding this comment

RocMarshal left a comment

Choose a reason for hiding this comment

1996fanrui left a comment

Choose a reason for hiding this comment

RocMarshal commented Jun 14, 2024

RocMarshal commented May 10, 2024 •

edited

Loading

RocMarshal commented May 22, 2024 •

edited

Loading