Gfodor/add timestamp to producer #44
Conversation
|
Hi @gfodor, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution! The agreement was validated by Microsoft and real humans are currently evaluating your PR. TTYL, MSBOT; |
| @@ -307,26 +410,30 @@ internal static Message ParseFrom(KafkaBinaryReader reader, long offset, int siz | |||
| /// <summary> | |||
| /// Clean up attributes for message, otherwise there is double decompress at kafka broker side. | |||
| /// </summary> | |||
| internal void CleanMagicAndAttributesBeforeCompress() | |||
| internal void CleanAttributesBeforeCompress() | |||
| { | |||
| this.Attributes = 0; | |||
There was a problem hiding this comment.
I wasn't completely sure about this change. It looks like based on the comments that someone was experiencing a magic value of 2, and hence was zeroing things out here, but magic value of 2 is an invalid value. It's now important that the packed messages retain their magic value of either 0 or 1 otherwise the timestamp will not be getting encoded properly.
|
It seems like there are at least two orthogonal[?] things contained in this Pull Request - support for different protocol versions, and the timestamp change itself. Could these be split into separate Pull Requests? It is very much easier for others to assess the contents and impact of a change(s) that way. Also, @gfodor don't be afraid to rewrite / rebase your PR commits to clean up the change history -- you can do interactive rebase on your local branch and then force push to the remote PR branch and GitHub will do the right thing and keep everything associated with this PR discussion thread. |
|
I think the way to do it would be 0.9.0 support and then 0.10.0 support. The protocol changes for 0.9.0 are adding the throttle time fields (FetchRequest message format v1, ProducerResponse format v1), and 0.10.0 is adding timestamp fields support (Message format v1, ProducerResponse format v2.) I can pull out the 0.9.0 stuff separately if that would be useful. |
consumedOffset=fetchOffset which could lead to data loss. Don't commit offset if it already commited modified: src/KafkaNET.Library/Consumers/ZookeeperConsumerConnector.cs modified: src/KafkaNET.Library/ZooKeeperIntegration/Listeners/ZKRebalancerListener.cs
Pretty self explanatory
- NuGet package cache will be restored by Visual Studio during build.
- Visual Studio will automatically create these files during build when needed.
- The Reference paths in project file `Kafka.Client.Tests.csproj` and NuGet `packages.config` file in that project specify different versions of the `FluentAssertions` and `Moq` packages. - Fix is to change `packages.config` file to specify same version as the `.csproj` file.
- The NuGet `packages.config` files are missing from projects `KafkaNET.Library` and `KafkaNETLibraryConsole`
fixed spelling
- Adding some missing package dependencies that Visual Studio 2013 complained were missing. - Ran the migrateToAutomaticPackageRestore.ps1 script to fix problems with nuget package auto-restore when building from inside Visual Studio. https://github.com/owen2/AutomaticPackageRestoreMigrationScript
This reverts commit 84b459b.
…ed to itself The kind people at Viva64 gave an evaluation license for PVS-Studio to try out on some OSS projects, and running it on Kafkanet source code found this problem. http://www.viva64.com/en/pvs-studio/ Error V3005. The 'ConsumeGroupFindNewLeaderSleepIntervalMs' variable is assigned to itself http://www.viva64.com/en/d/0403/print/ Fixes issue #20
…w mocking for unit tests
…terface * return list of IKafkaMessageStream from ZookeeperConsumerConnector.CreateMessageStreams * change IKafkaMessageStream to return IKafkaMessageStream from GetCancellable instead of IEnumerable
…new consumer group
…ad of 1, 3) Some minor performance improvement and bug fixes
… as connectionTimeout
|
This branch got hosed when trying to do my rebasing, going to just close it and create another one. |
This PR updates my previous PR #31 --