Try fixed https://github.com/sutugin/spark-streaming-jdbc-source/issu…

[sutugin]

Try fixed #5 using sql min/max functions&

[roychen11232357]

hi bro,

I took a look at this commit, and it is indeed optimized for the performance of offset fetching.

The key of this commit should be this line

val minMaxQuery = s"(select max($offsetColumn) as max_$offsetColumn, min($offsetColumn) as min_$offsetColumn from $dbTable) minMaxTable"

Can this commit be merged back to master? Or are there other unresolved issues?

---

By the way,

I have tested this commit, I use postgresql to test, I found that every trigger will run these two query

SELECT * FROM (select max(number) as max_number, min(number) as min_number from public.test3) minMaxTable WHERE 1=0
SELECT "max_number","min_number" FROM (select max(number) as max_number, min(number) as min_number from public.test3) minMaxTable

The first query (xxxx where 1=0), which should be generated by spark jdbc, is to get the schema, right?

If we really don't want to call the first query (where 1=0 ) every time, does getOffsetValues ​not use spark jdbc to get the data, but directly use other jdbc libs?

Because I don't think there is a need to get the schema every time the trigger, I don't know if there are other ways to avoid this situation?

[arouel]

@sutugin @roychen11232357 I'm curious about the outcome. Any suggestion to go forward?