[FLINK-3154][API] Upgrade from Kryo 2.x to Kryo 5.x. Removed twitter …

[kurtostfeld]

…chill companion library which is abandoned.

What is the purpose of the change

Update the Kryo serializiation library dependency. Remove the Twitter Chill library which is a Scala companion library for Kryo that has been abandoned.

This change is not binary compatible with saved state from previous versions of Flink 1.x. Since Flink 2.0 is not providing binary compatibility with saved state from Flink 1.x applications, it is possible to make this change.

Brief change log

• Updated Kryo dependency from 2.24.0 to the current version 5.6.2
• Updated objenesis from 2.1 to 3.4. The new Kryo uses 3.4. The mockito test library version in use uses 3.3. The project had this pinned to 2.1 which was what the old Kryo library used.
• Removed the Twitter Chill companion library for Kryo. This library is abandoned. Even the most recent version of Chill depends on Kryo 4.0,x. Removing Chill simplifies dependencies.
• Replaced classes to serialize Avro GenericData.Arrays. Replaced the older SpecificInstanceCollectionSerializerForArrayList and SpecificInstanceCollectionSerializer with a simple new serializer AvroGenericDataArraySerializer. The older classes were serializing to an ArrayList which wasn't working with the new version of Kryo.
• Various other changes to accommodate the above.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changed class annotated with @Public(Evolving): yes
• The serializers: yes
• The runtime per-record code paths (performance sensitive): yes
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
• The S3 file system connector: don't know

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 9d2c6d0 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[snuyanzin]

regarding table error

Jan 05 06:56:38 06:56:38.023 [ERROR] AdaptiveJoinTest.testWithShuffleHashJoin:47 ast ==> expected: <

it was fixed at https://issues.apache.org/jira/browse/FLINK-37006

rebase to master should help

[davidradl]

@kurtostfeld I see your motivation for doing this, but you mention that there will not be binary compatibility with v1 state. Do you have an idea on how v1 Kyro state would migrate to v2 state after this fix? We are trying to encourage v2 migration - and this could be a big inhibitor.

[kurtostfeld]

@davidradl great question. That is best answered by the project leaders + managers.