Upgrade guide for releases

[cemo]

Thanks for everything. I would like to upgrade our cluster for 3.0 to 3.1 but I have no idea how should I start. Any help would be awesome.

[solsson]

Our goal is always to support kubectl apply but obviously I can't guarantee anything. The usual principles with backup and testing apply. In this case zookeeper didn't change. For Kafka it's essential that you apply the ConfigMap before the StatefulSet, as the upgrade will move config from container command to properties. kubectl apply on the folder does so. Then, because updateStrategy is OnDelete you manually kill one kafka pod at a time.

[solsson]

@clintfred raised this topic in #174 (comment)

we sometimes end up wanting to completely start with a new cluster (zookeeper, brokers, registry), but retain the data

This is an important use case. We're facing the same situation sometimes.

I haven't understood enough of automatic volume provisioning to know when existing volumes will be reused or not. After #159 it is quite possible that doing delete of everyting followed by create/apply of everyting will reuse the volumes, because the kafka namespace isn't deleted and there are no manifests for the PVCs, which means we won't delete them.

I think that in some cases apply will produce a different result than replace. It might merge stuff, based on merge keys such as names. https://github.com/Yolean/kubernetes-kafka/tree/master/prometheus uses patch, but could possibly use apply to add the extra container.

I'd appreciate contributions of actual insights and facts :)