Document security risks of fixed multi-byte encodings in HTTP header encoding selectors

[Copilot]

Fixed multi-byte encodings (Encoding.Unicode, Encoding.UTF32) cause request smuggling vulnerabilities when used with HTTP header encoding selectors—their binary representation can inject protocol control characters (e.g., ✊ contains bytes interpreted as \n).

Changes

Added detailed remarks to RequestHeaderEncodingSelector and ResponseHeaderEncodingSelector documenting:

• Safe encodings: Encoding.ASCII, Encoding.Latin1, or Encoding.UTF8
• CAUTION callout: Fixed multi-byte encodings must never be used due to binary misinterpretation breaking HTTP protocol
• Concrete example: How emoji bytes can be interpreted as newlines terminating header values
• Caller responsibility: Ensuring encoding compatibility with server to prevent silent corruption

Example

// ❌ DANGEROUS - Unicode bytes can break protocol
handler.RequestHeaderEncodingSelector = (name, request) => Encoding.Unicode;

// ✅ SAFE - Use ASCII, Latin1, or UTF8
handler.RequestHeaderEncodingSelector = (name, request) => Encoding.UTF8;

The documentation warns that mismatched encodings between client and server can cause silent data corruption (e.g., Latin1 client with UTF-8 server).

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Document that fixed multi-byte encodings aren't appropriate for HTTP header encoding selectors</issue_title>
<issue_description>Users are generally expected to use one of the following encoding schemes: Encoding.ASCII, Encoding.Latin1, or Encoding.UTF8.
Other encoding schemes may be susceptible to request smuggling attacks. Fixed multi-byte schemes in particular (e.g. Encoding.Unicode, Encoding.UTF32) should never be used as their binary representation is likely to be misinterpreted and break the protocol (e.g. part of ✊ is interpreted as a new line \n that terminates the header value).

When specifying an encoding, it is the caller's responsibility to ensure that the values are representable via said encoding, and that the server agrees on the encoding used. Otherwise, values will become corrupted.
For example, since any set of bytes is considered a valid Latin1 sequence, a server replying with UTF-8 encoded header values may see its responses silently corrupted.</issue_description>

<agent_instructions>Document that such encodings must not be used with SocketsHttpHandler.RequestHeaderEncodingSelector and SocketsHttpHandler.ResponseHeaderEncodingSelector.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Document that fixed multi-byte encodings aren't appropriate for HTTP header encoding selectors runtime#116859

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Copilot]

Pull Request Overview

This PR adds comprehensive documentation for the RequestHeaderEncodingSelector and ResponseHeaderEncodingSelector properties in the SocketsHttpHandler class. The documentation provides guidance on safe encoding schemes and warns against security vulnerabilities related to encoding choices.

Key Changes:

• Added detailed remarks sections for both request and response header encoding selector properties
• Documented recommended encoding schemes (ASCII, Latin1, UTF8) and security implications of other schemes
• Included caution about fixed multi-byte encoding schemes that could break protocol integrity