Flaky tests

[petedannemann]

Describe the bug

Many of the reader and consumer group tests sporadically fail. Here are a few recent examples. Some of these are "unsafe" tests, which we only recently started testing against in CI so I think that explains the recent increase in test failures.

TestReaderOnNonZeroPartition/topic_and_partition_should_now_be_included_in_header (Kafka 2.7.1)

ReaderTest (kafka 2.3.1)

CompressTest (kafka 2.1.1)

TestReaderConsumerGroup/rebalance_across_many_partitions_and_consumers (kafka 2.7.1)

Expected Behavior

Tests pass every time they are ran

[rhansen2]

Most of these failures seem to be timeout related. May be worth holding off until we migrate off of CircleCI.

Once outlier is

--- FAIL: TestCompressedMessages (2.78s)
    --- FAIL: TestCompressedMessages/gzip (0.32s)
        compress_test.go:208: wrong message key at loop 0...expected 1 but got 15

Which probably deserves it's own investigation.

[petedannemann]

Most of these seem to be fixed via #1349