Fix/template error status code - #17
Open
dengkliu92 wants to merge 788 commits into
Open
Conversation
…ai-dynamo#11993) Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Signed-off-by: Hannah Zhang <hannahz@nvidia.com>
…mo#11958) Signed-off-by: Zhuangcheng(Jesse) Gu <zcgu@connect.hku.hk>
Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com> Signed-off-by: PeaBrane <yanrpei@gmail.com> Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
…#11974) Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…ds (ai-dynamo#11786) Signed-off-by: Tushar Sharma <tusharma@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… scaling) (ai-dynamo#10785) Signed-off-by: Monishwaran Maheswaran <monishwaran@berkeley.edu> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…#11394) Signed-off-by: AmeenP <ameenp360@gmail.com> Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
…o#12084) Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
…d drt (ai-dynamo#12061) Signed-off-by: Sachal Malick <s@chal.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: furionw <qiwa@nvidia.com>
…mo#12070) Signed-off-by: Roman Ageev <rageev@nvidia.com>
Signed-off-by: Erez Zarum <erezz@amazon.com>
Signed-off-by: Olga Andreeva <oandreeva@nvidia.com>
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
…SEGV (OPS-7753) (ai-dynamo#11853) Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
HTTP --tls-cert-path/--tls-key-path no longer auto-set DYN_TCP_TLS_* env vars. TCP TLS on the frontend is now controlled independently via --tcp-tls-cert-path, --tcp-tls-key-path, and --tcp-tls-ca-cert-path. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit e86c36d) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 73db19f) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 3edbc96) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Replace hand-rolled `v == "1" || v == "true"` and
`eq_ignore_ascii_case("true")` with `crate::config::env_is_truthy()`
to pass the no_bool_parse_forks_outside_canonical_crate test.
Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
(cherry picked from commit e9ff9e0)
Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Add TLS page to Fern index.yml so it appears in the docs site. Fix Kubernetes section to show DynamoGraphDeployment podTemplate instead of unsupported InfrastructureConfiguration. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 3706336) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Extend TLS support to the shared TCP request plane that carries inference requests from frontend to worker. This closes the security gap where user prompts were transmitted in plaintext. Server side (shared_tcp_endpoint.rs): - Build TlsAcceptor from DYN_TCP_TLS_CERT_PATH/KEY_PATH at startup - Per-connection TLS handshake with configurable timeout - BoxRead/BoxWrite unify TLS and plaintext through read/write loops Client side (egress/tcp_client.rs): - OnceLock-cached TlsConnector from DYN_TCP_TLS_CA_CERT_PATH/INSECURE - TLS handshake with timeout on each pooled connection - BoxRead/BoxWrite through writer_task and reader_task Reuses the same DYN_TCP_TLS_* env vars as the call-home transport. No new env vars needed. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit bce0da3) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Enable opt-in TLS for the NATS client connection. A custom rustls TLS config is applied when NATS_TLS_CA_CERT_PATH is set or NATS_TLS_INSECURE is truthy. When only a tls:// URL is used without explicit TLS env vars, async-nats handles TLS natively with system roots. New env vars: NATS_TLS_CA_CERT_PATH, NATS_TLS_INSECURE. The NATS server URL validator now accepts tls:// (case-insensitive) in addition to nats://. No process-wide crypto provider is installed to avoid racing with the HTTP path's aws_lc_rs provider. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 111e16e) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Enable opt-in TLS for the NATS client connection. A custom rustls TLS config is applied when NATS_TLS_CA_CERT_PATH is set or NATS_TLS_INSECURE is truthy. When only a tls:// URL is used without explicit TLS env vars, async-nats handles TLS natively with system roots. New env vars: NATS_TLS_CA_CERT_PATH, NATS_TLS_INSECURE. The NATS server URL validator now accepts tls:// (case-insensitive) in addition to nats://. No process-wide crypto provider is installed to avoid racing with the HTTP path's aws_lc_rs provider. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 2107c06) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Extend TLS support with mutual authentication: TCP server (server.rs): - Read DYN_TCP_TLS_CLIENT_CA_CERT_PATH to enable client certificate verification via WebPkiClientVerifier - Log whether TLS or mTLS is active at startup TCP client (client.rs): - Read DYN_TCP_TLS_CLIENT_CERT_PATH and DYN_TCP_TLS_CLIENT_KEY_PATH to present a client certificate to the server NATS client (nats.rs): - Read NATS_TLS_CLIENT_CERT_PATH and NATS_TLS_CLIENT_KEY_PATH - Validate both are set or both unset tls_utils.rs: - server_tls_config() accepts optional client_ca_cert_path for mTLS - client_tls_config() accepts optional client_cert_path/client_key_path - Shared load_client_cert() helper for PEM loading New env vars: DYN_TCP_TLS_CLIENT_CERT_PATH, DYN_TCP_TLS_CLIENT_KEY_PATH, DYN_TCP_TLS_CLIENT_CA_CERT_PATH, NATS_TLS_CLIENT_CERT_PATH, NATS_TLS_CLIENT_KEY_PATH. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 4da67a3) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
…ration Add 9 TLS certificate path fields to InfrastructureConfiguration so the operator injects DYN_TCP_TLS_* and NATS_TLS_* env vars into all DGD pods automatically. This eliminates per-component env var setup for TLS. Fields: natsTLSCAPath, natsTLSClientCertPath, natsTLSClientKeyPath, tcpTLSCertPath, tcpTLSKeyPath, tcpTLSCAPath, tcpTLSClientCertPath, tcpTLSClientKeyPath, tcpTLSClientCAPath. (cherry picked from commit ada9857) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
NATS TLS support means the page now covers more than TCP transports, so retitle to TLS and update the subtitle to reflect TCP + NATS. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 13665a8) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
… page The operator-level TLS/mTLS auto-injection via InfrastructureConfiguration is Kubernetes/operator-specific, so move it out of the transport-agnostic TLS reference into a dedicated Operator TLS page under Kubernetes Operator. Cross-link the two pages. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 3677461) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
The request-plane page documents all other DYN_TCP_* environment variables, so add a pointer to the TLS reference for the encryption options as well (per review feedback). Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com> (cherry picked from commit 78ba919) Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
…ane TLS config SharedTcpServer::new was the only TLS entry point that panicked on bad/partial cert config; every other path (tcp/server.rs build_tls_acceptor, tcp/client.rs, egress/tcp_client.rs, nats.rs) returns a fallible Result. Make new() return anyhow::Result<Arc<Self>> and propagate the error, matching the response-stream server, so misconfiguration fails startup cleanly instead of aborting the process. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>
Pass the configured client CA to the TCP request-plane server so it rejects clients without a trusted certificate. Signed-off-by: dengke.liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Present the configured client certificate, fail closed on incomplete TLS configuration, and document request-plane coverage. Signed-off-by: dengke.liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Add focused client and server configuration coverage and warn when a TLS server is paired with a plaintext request-plane client. Signed-off-by: dengke.liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Flush batched requests through rustls and add request-plane mTLS handshake coverage. Signed-off-by: dengke.liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Server-side rejection is the enforcement signal; rustls may still return Ok from client connect() before the fatal alert is observed. Signed-off-by: dengke.liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Keep long-lived workers and frontends on valid mTLS credentials after Emissary rotates certificate symlinks. Signed-off-by: dengke.liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid waiting a full 30s after a failed inspect/load during certificate rotation, fingerprint identities after reading file contents, and add regression coverage for short retry and atomic symlink swaps. Signed-off-by: Dengke Liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: dengkliu92 <dengke.liu@datadoghq.com>
Always read due identities so metadata-stable rewrites are detected and the loaded bytes cannot be suppressed by a mismatched metadata fingerprint. Signed-off-by: dengke.liu <dengke.liu@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Template rendering errors (e.g., 'Unexpected message role', malformed chat history) are client validation errors and should return HTTP 400 Bad Request, not 500 Internal Server Error. Previously, these errors were propagated as generic anyhow::Error, which ErrorMessage::from_anyhow() mapped to 500. This fix explicitly converts template errors to DynamoError::InvalidArgument, which the HTTP layer correctly maps to 400. Fixes incorrect status codes for errors like: - 'Unexpected message role' (in tool_use context) - Invalid chat message structure - Template parameter validation failures Before: HTTP 500 'Internal server error: Failed to generate completions...' After: HTTP 400 'Failed to apply prompt template: ...'
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: dengkliu92 <dengke.liu@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:
Details:
Where should the reviewer start?
Related Issues
🔗 This PR is linked to an issue:
🚫 This PR is NOT linked to an issue: