Update kafka version to include client side gzip leak fix

[mridulm]

This is the same pr which I submitted against @jerryshao's older repo.
I have also bumped up the artifact version due to dependency version change.

[mridulm]

+CC @bikassaha @jerryshao

[mridulm]

Two tests failed again:

- basic stream receiving with multiple topics and smallest starting offset *** FAILED *** (1 minute, 42 seconds)
  The code passed to eventually never returned normally. Attempted 110 times over 1.6734079014833332 minutes. Last failure message: 0 did not equal 46 didn't get expected number of messages, messages:
  . (DirectKafkaStreamSuite.scala:152)
- pattern based subscription *** FAILED *** (1 minute, 41 seconds)
  The code passed to eventually never returned normally. Attempted 110 times over 1.672780918083333 minutes. Last failure message: 0 did not equal 41 didn't get expected number of messages, messages:
  . (DirectKafkaStreamSuite.scala:218)

Is this a kafka incompatibility ? Or flakey tests ?

[jerryshao]

We might hit this issue (https://issues.apache.org/jira/browse/SPARK-18057?focusedCommentId=15905716&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15905716) in the test code.

[mridulm]

Looks like it is not a straightforward case of upgrading dependency.
Given that skc is a convenience package to transition for older users, I am thinking of closing this PR and avoiding hassle with having to fix/make increasing code changes.
Any thoughts @jerryshao @bikassaha ?

[jerryshao]

If we ignore the UT failure, then I think it is OK to upgrade the Kafka version. This is because test code uses several internal APIs which doesn't guarantee to be compatible across version, whereas source code only depends on kafka-client APIs, which guarantees to be compatible, so it is OK to upgrade the Kafka version if we ignore the UT failures.

[mridulm]

@jerryshao Ignoring UT failures, which used to work with earlier versions, is a bit worrying; we potentially miss out of issues (particularly when evolving code).
If you believe it is non trivial to fix UT's (I really dont have much experience with streaming - but I can take a shot if it is something simple), I would rather abandon this change than make codebase unstable.