ref(service): Classify service errors by kind

[lcian]

This is a different take on giving us more control over HTTP status codes when erroring out from the service layer that doesn't make callers more verbose, but actually less so in some cases.

1. ServiceError variants gain a kind that's used to determine log level and HTTP status code when converting to ApiErrorResponse. The kind is implemented via https://crates.io/crates/derive-error-kind which lets us do it concisely via a error_kind attribute on each variant. This is a very simple proc macro that we could also vendor in.

2. Existing constructors for specific Error variants remain, and map to ErrorKind::Internal.

3. For errors that could map to different kinds depending on the context, we also introduce additional intermediate wrapper types (needed to support different kinds) and constructors on ServiceError.
   Namely: reqwest_transparent (alternative to the existing reqwest helper), serde_client (alternative to the existing serde helper), and the metadata/metadata_client pair.

4. As a special case, we implement From<std::io::Error> for Error so that callers can keep using ? to propagate io errors. That's because so far we always want io::Error to map to ErrorKind::internal, so there's no reason to make callers more verbose in this case.
   This has the added benefit that the impl can encapsulate the unpack_client_error logic, to avoid repeating it in every caller.

5. Also renames ClientError to ClientStreamError as that's a less confusing name.

Refs FS-358

[linear-code]

FS-358

[lcian]

Improvement: here we have a valid Metadata, it's therefore our fault if we fail to serialize it, and now we can correctly map this to an internal error as opposed to always blaming the client.

[jan-auer]

I like the direction this is taking. Especially, it makes a lot of sense to introduce one flat kind that the server can match on to determine the status code.

The kinds are a bit broad and seem directly tied to the status codes that the server returns. Let's still keep more fine grained kinds (e.g. AtCapacity) and let the server choose how to map which of those, and which to group together. This decouples writing logic from the HTTP semantics a bit better.

#489 seems to take this one step further and replaces the typed error enum with a boxed error. This is ultimately what we should get to, although we could do that in multiple steps. What #493 (this PR) does better is that there are more convenient conversions that avoid .map_err(...) in all places.

One more thing to look into, possibly in a follow up, is an extension trait that we can call directly on Result. With that, it would get highly ergonomic to wrap an error and supply either the kind or kind + context.

[sentry]

Bug: Backend authentication or authorization errors (401/403) are incorrectly classified as client InvalidInput, resulting in a misleading 400 BAD_REQUEST response to the client.
_{Severity: HIGH}

Suggested Fix

Modify the kind_for_status function to handle specific backend 4xx status codes, such as 401 UNAUTHORIZED and 403 FORBIDDEN, separately. Instead of mapping all client errors to ErrorKind::InvalidInput, these specific codes should be mapped to a more appropriate server-side error kind, like ErrorKind::Internal, to avoid misrepresenting server-side issues as client errors.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: objectstore-service/src/error.rs#L374

Potential issue: The `kind_for_status` function incorrectly maps all 4xx HTTP status
codes from backend services to `ErrorKind::InvalidInput`. This causes server-side
issues, such as a `401 UNAUTHORIZED` from a storage backend due to misconfigured
credentials, to be re-mapped and returned to the end client as a `400 BAD_REQUEST`. This
misclassification masks the true source of the error, misleading API consumers into
believing their request was malformed when the problem is actually a server-side
configuration or authentication failure.

Also affects:

• objectstore-server/src/endpoints/common.rs:92~114

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 79ab712. Configure here.}

[cursor]

Backend forbidden classified client input

Medium Severity

kind_for_status labels every storage-backend HTTP 4xx (except a few status-specific cases) as ErrorKind::InvalidInput, and the API layer maps that kind to HTTP 400. Backend responses such as 403 Forbidden or 401 Unauthorized reflect service-to-backend auth or configuration problems, not malformed API client input, so callers get a misleading Bad Request instead of an server-side failure status.

Additional Locations (1)

• objectstore-server/src/endpoints/common.rs#L94-L98

 

^{Reviewed by Cursor Bugbot for commit 79ab712. Configure here.}