Document the trust model and panics in the codebase.

[PiotrSikora]

No description provided.

[alexsnaps]

This issue with these panic is that currently envoy doesn't play nicely with them... Tho things are getting better, it can still result in unexpected behavior. So while I understand the rational behind wanting to panic!, given this is wasm and panic are aborting execution, this leaves the caller with no way to gracefully handle the error. Which by itself is problematic in my perspective, but together with some environment then failing further (like starving workers), is really not acceptable, as I imagine we can all agree on.

So, should there be a way to provide the user with means to be "fully responsible"? it could be "as simple" as allowing the user to call into functions that would expose all failures, while leaving the trait et al default implementation panic? Or maybe add a "SDK panic error handler" that defaults to the current behavior, but would let the user override that behavior? Thinking out loud here, but while I understand ideally "it would all always work according to spec", its hard to guarantee it will, more so with an on-going specification.

This has come up in the past when discussing one of my PRs with @PiotrSikora . We are now running into issues in production that don't allow us to tolerate the SDK panicking. Looking into a viable path forward that would check everyone's goals, if such a thing is possible obviously. Thanks!