Fix orphaned golden actors by using deterministic actor IDs in ActorTemplateReconciler

[Troy Chiu (troychiu)]

This PR fixes an issue where reconciliation retries in PhaseInitial left orphaned actors in ateapi when CreateActor succeeded but status update failed.

Fixes #525

• Tests pass
• Appropriate changes to documentation are included in the PR

[Troy Chiu (troychiu)]

Kubernetes UIDs (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids) should be compatible with resource names, which are enforced by standard RFC-1123 DNS label rules.

[Eitan Yarmush (EItanya)]

Great catch!

[Zoe Zhao (zoez7)]

This changes the behaviour of Reconcile - if a previous attempt created a actor with at.UID and failed to produce a golden snapshot, this will reuse the same actor and may never produce a usable golden snapshot.

Also, now updates to that same actor template doesn't create a new snapshot. I think this bug is part of the complexity mentioned in #536.

[Eitan Yarmush (EItanya)]

This changes the behaviour of Reconcile - if a previous attempt created a actor with at.UID and failed to produce a golden snapshot, this will reuse the same actor and may never produce a usable golden snapshot.

Functionally I think that's correct, otherwise like mentioned above we may have a bunch of orphaned actors. Do you think a more correct way would be to leave them orphaned and have a secondary GC process pick them up?

Also, now updates to that same actor template doesn't create a new snapshot. I think this bug is part of the complexity mentioned in #536.

I thought the ActorTemplate was immutable today anyway?

[Troy Chiu (troychiu)]

This changes the behaviour of Reconcile - if a previous attempt created a actor with at.UID and failed to produce a golden snapshot, this will reuse the same actor and may never produce a usable golden snapshot.

Even before this PR, once at.Status.GoldenActorID was saved, all subsequent reconciliation phases already reused that single actor ID upon retries. The orphan leak happened specifically when CreateActor succeeded in ateapi but the kubernetes status update failed.

I thought the ActorTemplate was immutable today anyway?

Yes the ActorTemplate spec is immutable

substrate/pkg/api/v1alpha1/actortemplate_types.go

Lines 387 to 388 in 1afaaf5

	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Spec is immutable"
	Spec ActorTemplateSpec `json:"spec"`