[8.17](backport #6623) [k8s] Fix logical race conditions in kubernetes_secrets provider

[mergify]

What does this PR do?

This PR refactors the implementation to eliminate logical race conditions of the kubernetes_secrets provider alongside with brand new unit-tests.

Initially, the issue seemed to stem from misuse or lack of synchronisation primitives, but after deeper analysis, it became evident that the "race" conditions were logical rather than concurrency-related. The existing implementation was structured in a way that led to inconsistencies due to overlapping responsibilities of different actors managing the secret lifecycle.

To address this, I restructured the logic while keeping in mind the constraints of the existing provider, specifically:

• Using a Kubernetes reflector (watch-based mechanism) is not an option because it would require listing and watching all secrets, which is often a non-starter for users.
• Instead, we must maintain our own caching mechanism that periodically refreshes only the referenced Kubernetes secrets.

With this in mind, the provider behaviour is now as follows:

Cache Disabled Mode:

• When caching is disabled, the provider simply reads secrets directly from the Kubernetes API server.

Cache Enabled Mode:

• When caching is enabled, the provider stores secrets in a cache where entries expire based on the configured TTL (time-to-live) and a lastAccess field of each cache entry.
• The provider has two primary actors: cache actor and fetch actor, each with well-defined responsibilities.

Cache Actor Responsibilities:

1. Signal expiration of items: When a secret expires, the cache actor signals that a fetch should occur to reinsert the key into the cache, ensuring continued refreshing.
2. Detect secret updates and signal changes: When the cache actor detects a secret value change, it signals the ContextProviderComm.
3. Conditionally update lastAccess:
   • If the secret has changed, update lastAccess to prevent premature expiration and give the fetch actor time to pick up the new value.
   • In any other case, do not update lastAccess and let the entry "age" as it should.

Fetch Actor Responsibilities:

1. Retrieve secrets from the cache:
   • If present, return the value.
   • If missing, fetch from the Kubernetes API.
2. Insert fetched secrets into the cache if there isn't a more recent version of the secret already in it (can happen by the cache actor or a parallel fetch actor).
3. Always update lastAccess when an entry is accessed to prevent unintended expiration.

Considerations:

• No global locks: Store operations are the only part of the critical path, ensuring that neither cache nor fetch actors block each other.
• Conditional updates: Since cache state can change between the time an actor reads and writes, all updates use conditional store operations that are part of the critical path.
• Custom store implementation: The existing ExpirationCache from k8s.io/client-go/tools/cache does not suit our needs, as it lacks the aforementioned conditional insertion required to handle these interactions correctly.
• Optimized memory management: The prior implementation copied the cache map on every update to prevent Golang map bucket retention. However, I believe this was a misunderstanding of Golang internals and a premature optimisation. If needed in the future, this can be revisited in a more controlled manner.

PS: as the main changes of this PR are captured by the commit a549728, I consider this PR to be aligned with the Pull Requests policy

Why is it important?

This refactor significantly improves the correctness of the kubernetes_secrets provider by ensuring:

• Secrets do not expire prematurely due to logical race conditions.
• Updates are properly signaled to consuming components.
• Performance is optimised with minimal locking and unnecessary memory allocations.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding changes to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the change-log tool
• I have added an integration test or an E2E test

Disruptive User Impact

This change does not introduce breaking changes but ensures that the kubernetes_secrets provider operates correctly in cache-enabled mode. Users relying on cache behaviour may notice improved stability in secret retrieval.

How to test this PR locally

1. Run unit tests to validate the new caching behaviour:

   go test ./internal/pkg/composable/providers/kubernetessecrets/...

Related issues

• Closes Deletion race condition when fetching secret from the Kubernetes secrets provider cache #6340

---

This is an automatic backport of pull request #6623 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 6d4b91c has failed:

On branch mergify/bp/8.17/pr-6623
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit 6d4b91ccb.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1738139927-Fix-logical-race-conditions-in-kubernetes_secrets-provider.yaml
	modified:   internal/pkg/composable/providers/kubernetessecrets/config.go
	new file:   internal/pkg/composable/providers/kubernetessecrets/expiration_cache.go
	modified:   internal/pkg/composable/providers/kubernetessecrets/kubernetes_secrets_test.go
	new file:   internal/pkg/composable/providers/kubernetessecrets/store_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/composable/providers/kubernetessecrets/kubernetes_secrets.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube