Http Stress Status Report #42211

ManickaP · 2020-09-14T18:17:06Z

Http Stress Status Report

What we've run so far:

OS	HTTP 1.1	HTTP 2.0	Notes
Windows	4+ h (Mana) -> 2 errors	12 h (JJ) -> 11 errors
	6 h (Miha) -> 8 errors	6 h (Miha) -> 7 errors
	10 h (Miha) -> 53 errors	7 h (Miha) -> 139 errors	7 h = 1 + 6 h
Linux	17 h (JJ) -> 0 errors	12 h (JJ) -> 608 errors	we should rerun as this may be environmental problem
	12 h (Furtik) -> 0 errors	12 h (Furtik) -> 104 errors	ran on released runtime (not on master)

HTTP 2.0 Error Statistics

Error Type	Linux	Windows
Success	135,727,755	105,273,552
Errors	712	157
System.Threading.Tasks.TaskCanceledException: The operation was canceled.	37	8
System.Threading.Tasks.TaskCanceledException: A task was canceled.	13	4
System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.	464	18
System.Net.Sockets.SocketException (32): Broken pipe	198	0
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.	0	118
System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.	0	9

HTTP 1.1 Error Statistics

Error Type	Linux	Windows
Success	171,673,491	128,658,822
Errors	0	61
System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.	0	61

What we need to run:

More HTTP 1.1 Linux runs to confirm that we're clear. (easy, hi pri)
More HTTP 2.0 Linux runs to confirm that we have all error types captured. (easy, hi pri)
HTTP 2.0 tests against HTTPSys to eliminate/confirm server as the culprit. (mid, mid pri)
Run the matrix against 3.1 and compare. (hard, mid pri)

Existing issues, root caused:

HTTP 1.1 connect cancellation deadlock: Cancellation of Socket.ConnectAsync intermitently hangs #42198
HTTP 2.0 sending RESET before END_STREAM: HTTP/2 stress server intermitently fails in duplex scenarios #42200
HTTP 2.0 throwing TaskCancelledException as a reaction on GO_AWAY: HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
NEW HTTP 2.0 infinite loop when connecting H2C to HTTP 1.0 localhost server: H2C connection to HTTP.sys on localhost causes infinite loop #42259

Discovered exceptions, not-investigated:

HTTP 2.0 System.Net.Sockets.SocketException (32): Broken pipe. (Linux only)
HTTP 2.0 System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
~~HTTP 2.0 System.Threading.Tasks.TaskCanceledException: The operation was canceled.~~ HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
HTTP 2.0 System.Threading.Tasks.TaskCanceledException: A task was canceled. --> might be covered by HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
HTTP 2.0 System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host. (Windows only)
HTTP 2.0 System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. (Windows only)
NEW HTTP 1.1 System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (Windows only)

The discovered exceptions confirm what we've collected so far from the pipelines: #40388.

Distributable tasks by priority:

More HTTP 1.1 Linux runs: http11run
More HTTP 2.0 Linux runs: http20run
Investigate Windows HTTP 1.1 errors: winErr3
Investigate Windows HTTP 2.0 errors: winErr1, winErr2
Provide fix for #42200
Provide fix for #42198
Help with HTTPSys client connection errors: httpSys: put on back-burner
Run the tests against .NET Core 3.1: net31: put on back-burner

Tips and Tricks for investigations:

Clear up docker containers and images after a product code change (docker container prune && docker image prune -a)
Once container is built, switch -b might be omitted for subsequent re-runs (skips the runtime build)
Don't use containers for investigations, they're slow and rebuild takes too long
- To run the app against locally built runtime, swap artifacts/bin/testhost/net5.0-Linux-Debug-x64/shared/Microsoft.NETCore.App/6.0.0/ with the globally installed runtime (/usr/share/dotnet/shared/Microsoft.NETCore.App/your-latest-5.0-version)
  - Make a backup of the global runtime!
  - Using testhost's corerun didn't work for me since the app depends on ASP .NET Core SDK
  - If you change the product code, rebuild just System.Net.Http and copy System.Net.Http.dll to the global runtime again
- Build System.Net.Http/tests/StressTests/HttpStress
- Open 2 terminals and run:
  - server: dotnet run -runMode server -aspnetlog
    - -aspnetlog: console logging of server errors
    - -serverUri https://localhost:5002: bind to a different port (when running multiple tests in parallel)
  - client: dotnet run -runMode client
    - -serverUri https://localhost:5002: connect to a different port (when running multiple tests in parallel)
    - -ops 1 2 3: run only operation 1, 2 and 3 (GET, PUT Slow, etc...)
    - -trace: saves internal client/server traces in a log file, very verbose, useable only for very short runs
  - more options in source code: https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/tests/StressTests/HttpStress/Program.cs#L37-L62

If you have any improvements to the stress app or the containers, please create a PR and don't keep it just for yourself.
If you have more tips and tricks for running the tests, please share them.

The text was updated successfully, but these errors were encountered:

ghost · 2020-09-14T18:17:08Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Fixes: stress client double read of content fixed fixed stress client hangs at start and stop leveraged HttpVersionPolicy increased pipeline timeout since we doubled the runs fixed base docker images to avoid missing IO.Pipelines Kestrel exception. Re-hauled tracing: added server file logging added log file rotation Minor renames. Contributes to: #42211 and #42198

antonfirsov · 2021-03-22T16:59:28Z

The description of #49999 has some useful links to stack traces according how failures occur today. The "new" HTTP 1.1 connection timeout seems to be a dominant issue on CI, failing on almost every Windows run.

antonfirsov · 2021-05-10T11:18:56Z

In the last 20 CI runs, there were 3 occurrences of the "System.Net.Sockets.SocketException (10054)" Windows + HTTP 2.0 case. I won't be surprised, if it's also a firewall issue being eventually solved by #52381. If not we should look into it.

We decided that we need to rule out the Windows Firewall as a cause for further failure types documented in #42211.

antonfirsov · 2021-06-28T10:36:02Z

I browsed through the failures of the last 25 days in the CI, and it seems that we are facing the same errors as las September. The only thing worth mentioning is that I've noticed a TaskCanceledException failure with a stack trace that comes from PUT Slow, instead of POST ExpectContinue I can't decide if it's covered by #42472 or not, we may want to check it:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
-> System.TimeoutException: A task was canceled.
-> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, Boolean finalFlush, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1482
at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1275
at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 623
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 291
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1941
at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 833
at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 897
at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 910
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 920
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 548

Full CI run

karelz · 2021-07-29T08:03:13Z

@antonfirsov can you please add note about your analysis from last week?

antonfirsov · 2021-07-29T14:40:36Z

Report 7/29

with manual observations after a brief look through the CI output of between 7/13-7/29, excluding the HTTP3 failures:

There is a new failure type now tracked under Regression in HTTP/1.1 stress tests: ObjectDisposedException ('SslStream') #56159
Other failures I've seen are known and described in the opening post
There are 2 cases where the Job run is unfinished. Likely an AZP anomaly, but we should keep monitoring if it happens again

Going back a bit more I see signs of #55261 (see here), but it doesn't seem to happen since 7/8 (EDIT: Because the test was disabled on 7/8).

antonfirsov · 2021-08-05T15:14:19Z

Report 8/5

brief look through the CI output between 7/29-8/5:

No new failure types in the test runs
Add SslApplicationProtocol.Http3 #56775 broke the last build, which is being fixed in Fix HttpStress build #56912

antonfirsov · 2021-08-13T09:51:06Z

Report 8/13

No new failure types in HTTP 1.1 & 2.0 in the last 1 week.

antonfirsov · 2021-08-24T15:19:42Z

Report 8/24

No new failure types in HTTP 1.1 & 2.0 since last report
There is one ~~occurrence~~ occurence of an unexpected ObjectDisposedException that happened after merging Do not drain HttpContentReadStream if the connection is disposed #57287. I'm reluctant to reopen Regression in HTTP/1.1 stress tests: ObjectDisposedException ('SslStream') #56159 right away, will do local runs and keep watching the CI, and open an issue if it happens again.

ManickaP · 2021-08-24T15:35:44Z

The occurrence is on 13.8., isn't that before the fix got merged?

antonfirsov · 2021-08-24T16:06:07Z

Sorry, linked wrong build, this one happened on the 18th:
https://dev.azure.com/dnceng/public/_build/results?buildId=1301334&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=45b31e99-b18c-506f-46cf-7d3a4d5140a9

karelz · 2021-08-25T12:15:23Z

@Vijay-Nirmal please file a separate issue with additional information. This issue is tracking general stress test runs, which may or may not be related to what you see. It is fine to link this issue. Thanks!

antonfirsov · 2024-10-29T16:48:04Z

Report 2024-10-29

Looking at runs since 2024-10-22.

`main` branch

HttpStress on `main`

Some issues are still there, regardless of the aspnet fix

1 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-11-14T14:44:27Z

Report 2024-11-14

Looking at runs since 2024-10-29.

`main` branch

HttpStress on `main`

There was one case of asan failure curing coreclr startup, opened One time mass coreclr asan failure in HTTP stress tests #109824

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-11-19T16:09:59Z

Report 2024-11-19

Looking at runs since 2024-11-14.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-11-26T14:21:00Z

Report 2024-11-26

Looking at runs since 2024-11-19.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline is broken!

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-12-05T14:37:37Z

Report 2024-12-05

Looking at runs since 2024-11-26.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline seems to be healed since 2024-12-03.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-12-10T15:55:11Z

Report 2024-12-10

Looking at runs since 2024-12-05.

`main` branch

HttpStress on `main`

2 x ObjectDisposedException in HTTP/1.1 Stress #110577

SslStress on `main`

1 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044 (Not reopening the issue yet.)

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

The pipeline seems to be healed since 2024-12-03.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=781731&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a

SslStress on `release/8.0`

No failures

antonfirsov · 2024-12-17T14:35:10Z

Report 2024-12-17

Looking at runs since 2024-12-10.

`main` branch

HttpStress on `main`

1 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-01-07T14:49:30Z

Report 2025-01-07

Looking at runs since 2024-12-17.

`main` branch

HttpStress on `main`

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-01-14T16:01:35Z

Report 2025-01-14

Looking at runs since 2024-01-07.

`main` branch

HttpStress on `main`

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-01-21T14:18:08Z

Report 2025-01-21

Looking at runs since 2024-01-14.

`main` branch

HttpStress on `main`

The builds are broken since 2025-01-17, see #111660.

1 x ObjectDisposedException in HTTP/1.1 Stress #110577

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-01-28T16:17:26Z

Report 2025-01-28

Looking at runs since 2024-01-21.

`main` branch

HttpStress on `main`

#111660 has been fixed, the builds are running again.

2 x ObjectDisposedException in HTTP/1.1 Stress #110577

SslStress on `main`

1 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-02-04T16:23:47Z

Report 2025-02-04

Looking at runs since 2025-01-28.

`main` branch

HttpStress on `main`

1 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121 with stack trace:

---> System.Net.Quic.QuicException: Stream aborted by peer (267).
  at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
  at System.Net.Quic.QuicStream.HandleEventPeerReceiveAborted(_PEER_RECEIVE_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 612
  at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 663
  at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 694

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-02-11T16:19:23Z

Report 2025-02-11

Looking at runs since 2025-02-04.

`main` branch

HttpStress on `main`

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-02-18T13:53:12Z

Report 2025-02-18

Looking at runs since 2025-02-11.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-03-04T14:20:27Z

Report 2025-03-04

Looking at runs since 2025-02-18.

`main` branch

HttpStress on `main`

1 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `main`

2 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

`release/9.0` branch

HttpStress on `release/9.0`

No failures

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-03-11T14:00:32Z

Report 2025-03-11

Looking at runs since 2025-03-04.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0` branch

HttpStress on `release/9.0`

Linux builds are broken: [HttpStress] Linux image is running out of disk space #111660
No failures on Windows

SslStress on `release/9.0`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2025-03-18T14:15:43Z

Report 2025-03-18

Looking at runs since 2025-03-11.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

Linux builds were broken, just fixed
No failures on Windows

SslStress on `release/9.0-staging`

1 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

No failures

SslStress on `release/8.0-staging`

No failures

antonfirsov · 2025-03-25T13:57:46Z

Report 2025-03-25

Looking at runs since 2025-03-18.

`main` branch

HttpStress on `main`

2 x build failures, failing to run Invoke-WebRequest -Uri https://dot.net/v1/dotnet-install.ps1

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

No failures

SslStress on `release/9.0-staging`

1 x build failures, failing to run Invoke-WebRequest -Uri https://dot.net/v1/dotnet-install.ps1

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

No failures

SslStress on `release/8.0-staging`

No failures

antonfirsov · 2025-04-01T14:21:45Z

Report 2025-04-01

Looking at runs since 2025-03-25.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

No failures

SslStress on `release/9.0-staging`

1 x SslStress: 1 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

2 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `release/8.0-staging`

No failures

antonfirsov · 2025-04-08T13:15:51Z

Report 2025-04-08

Looking at runs since 2025-04-01.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

1 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `release/9.0-staging`

No failures

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

No failures

SslStress on `release/8.0-staging`

No failures

antonfirsov · 2025-04-15T15:11:27Z

Report 2025-04-15

Looking at runs since 2025-04-08.

`main` branch

HttpStress on `main`

2 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

2 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `release/9.0-staging`

No failures

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

1 x weird failure in POST ExpectContinue: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1010748&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

Failure Type 1/1:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpIOException: An error occurred while sending the request. (Unknown)
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.OperationCanceledException: The operation was canceled.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 169
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 806
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 806
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 806
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 89
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 179
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 352
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3477
   at System.Threading.Tasks.Task.TwoTaskWhenAnyPromise`1.Invoke(Task completingTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 6487
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3495
   at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 5737
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 683
   at System.Threading.TimerQueue.FireNextTimers() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 331
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1010
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 102
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 247
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 311
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 815
   at System.Net.Http.Http3RequestStream.ReadNextDataFrameAsync(HttpResponseMessage response, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1296
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1187
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1280
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1240
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_10>d.MoveNext() in C:\app\ClientOperations.cs:line 413
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on `release/8.0-staging`

No failures

antonfirsov · 2025-04-22T12:54:19Z

Report 2025-04-22

Looking at runs since 2025-04-15.

`main` branch

HttpStress on `main`

1 x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

No failures

SslStress on `release/9.0-staging`

No failures

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

No failures

SslStress on `release/8.0-staging`

No failures

MihaZupan · 2025-05-07T00:20:33Z

Report 2025-05-07

Looking at runs since 2025-04-22.

`main` branch

HttpStress on `main`

No failures until May 2nd, now all builds are failing.

SslStress on `main`

No failures until May 2nd, now all builds are failing.

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

No failures

SslStress on `release/9.0-staging`

No failures

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

No failures

SslStress on `release/8.0-staging`

No failures

MihaZupan · 2025-05-07T00:33:50Z

@jkoritzinsky Looks like #114285 broke our http/ssl stress test builds (we're using a weird configuration).

Repro:

.\build.cmd clr+libs -rc Release -c Release
cd src/libraries/System.Net.Http/tests/StressTests/HttpStress
.\build-local.ps1

error MSB3030: Could not copy the file "D:\runtime3\src\libraries\System.Net.Http\tests\StressT
ests\HttpStress\obj\Release\net10.0\apphost.exe" because it was not found.

Looking at binlogs, this might be because UpdateLocalAppHostPack is now overwriting AppHostSourcePath, which would previously point to
D:\runtime\src\libraries\System.Net.Http\tests\StressTests\HttpStress\.dotnet-daily\packs\Microsoft.NETCore.App.Host.win-x64\10.0.0-preview.5.25230.108\runtimes\win-x64\native\apphost.exe
, which is the local daily build of runtime build-local.ps1 downloads.

Any ideas about how we should go about fixing this?
Note that our stress projects are semi-standalone projects (e.g. custom build props).

jkoritzinsky · 2025-05-07T01:11:33Z

You can set <UseLocalAppHostPack> to false to pull from the package instead. That should work I believe.

MihaZupan · 2025-05-07T01:43:10Z

That seems to have done it, thanks!
(you may want to unsubscribe from this issue now since we keep posting reports here :))

MihaZupan · 2025-05-15T12:51:42Z

Report 2025-05-15

Looking at runs since 2025-05-08.

Build issues on main from last week were resolved by #115352.

`main` branch

HttpStress on `main`

1x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121 on Windows

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

No failures

SslStress on `release/9.0-staging`

No failures

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

1x [HttpStress] H3_REQUEST_REJECTED in Get / Get Slow #109121 on Windows

SslStress on `release/8.0-staging`

No failures

MihaZupan · 2025-05-19T19:02:23Z

Report 2025-05-19

Looking at runs since 2025-05-15.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

1x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044 on Windows

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

No failures

SslStress on `release/9.0-staging`

No failures

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

1x Build failure on Linux (not the most recent run, appears environmental)
- Failed during initial apt setup Failed to fetch http://deb.debian.org/debian/pool/main/l/llvm-toolchain-11/libclang-common-11-dev_11.0.1-2_amd64.deb Error reading from server - read (104: Connection reset by peer)

SslStress on `release/8.0-staging`

No failures

antonfirsov · 2025-05-27T14:57:11Z

Report 2025-05-27

Looking at runs since 2025-05-19. Exluding intermittent build failures.

`main` branch

HttpStress on `main`

No failures

SslStress on `main`

No failures

`release/9.0-staging` branch

HttpStress on `release/9.0-staging`

No failures

SslStress on `release/9.0-staging`

No failures

`release/8.0-staging` branch

HttpStress on `release/8.0-staging`

No failures

SslStress on `release/8.0-staging`

No failures

ManickaP added the area-System.Net.Http label Sep 14, 2020

ManickaP added this to the 6.0.0 milestone Sep 14, 2020

ManickaP self-assigned this Sep 14, 2020

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Sep 14, 2020

ManickaP removed the untriaged New issue has not been triaged by the area owner label Sep 14, 2020

ManickaP mentioned this issue Sep 16, 2020

HTTP stress test improvements #42313

Merged

karelz mentioned this issue Sep 21, 2020

HttpStress tests failing in difference scenarios on sending data to server #40388

Closed

karelz unassigned ManickaP Sep 21, 2020

karelz added the bug label Sep 21, 2020

ManickaP mentioned this issue Mar 22, 2021

Various HTTP Stress test failures on the CI #49999

Closed

antonfirsov mentioned this issue Apr 7, 2021

HttpStress: connection failures with HTTP 1.1 caused by Windows Firewall #50854

Closed

antonfirsov mentioned this issue May 6, 2021

HttpStress: Disable firewall in Windows runs #52381

Merged

antonfirsov mentioned this issue May 12, 2021

HttpStress: also disable firewall for HTTP2 runs #52643

Merged

antonfirsov added a commit that referenced this issue May 18, 2021

HttpStress: also disable firewall for HTTP2 (#52643)

8fb2eb5

We decided that we need to rule out the Windows Firewall as a cause for further failure types documented in #42211.

karelz modified the milestones: 6.0.0, 7.0.0 Jul 29, 2021

This comment has been minimized.

Sign in to view

MihaZupan mentioned this issue May 7, 2025

Fix HttpStress and SslStress missing apphost build error #115352

Merged

Http Stress Status Report #42211

Http Stress Status Report #42211

Comments

ManickaP commented Sep 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Http Stress Status Report

What we've run so far:

HTTP 2.0 Error Statistics

HTTP 1.1 Error Statistics

What we need to run:

Existing issues, root caused:

Discovered exceptions, not-investigated:

Distributable tasks by priority:

Tips and Tricks for investigations:

ghost commented Sep 14, 2020

Uh oh!

antonfirsov commented Mar 22, 2021

Uh oh!

antonfirsov commented May 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antonfirsov commented Jun 28, 2021

Uh oh!

karelz commented Jul 29, 2021

Uh oh!

antonfirsov commented Jul 29, 2021 • edited by karelz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Report 7/29

Uh oh!

antonfirsov commented Aug 5, 2021

Report 8/5

Uh oh!

antonfirsov commented Aug 13, 2021

Report 8/13

Uh oh!

antonfirsov commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Report 8/24

Uh oh!

ManickaP commented Aug 24, 2021

Uh oh!

antonfirsov commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

karelz commented Aug 25, 2021

Uh oh!

antonfirsov commented Oct 29, 2024

Report 2024-10-29

main branch

release/9.0 branch

release/8.0 branch

Uh oh!

antonfirsov commented Nov 14, 2024

Report 2024-11-14

main branch

release/9.0 branch

release/8.0 branch

Uh oh!

antonfirsov commented Nov 19, 2024

Report 2024-11-19

main branch

release/9.0 branch

release/8.0 branch

Uh oh!

antonfirsov commented Nov 26, 2024

Report 2024-11-26

main branch

release/9.0 branch

release/8.0 branch

Uh oh!

antonfirsov commented Dec 5, 2024

Report 2024-12-05

main branch

release/9.0 branch

release/8.0 branch

Uh oh!

antonfirsov commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Report 2024-12-10

ManickaP commented Sep 14, 2020 •

edited

Loading

antonfirsov commented May 10, 2021 •

edited

Loading

antonfirsov commented Jul 29, 2021 •

edited by karelz

Loading

antonfirsov commented Aug 24, 2021 •

edited

Loading

antonfirsov commented Aug 24, 2021 •

edited

Loading

`main` branch

`release/9.0` branch

`release/8.0` branch

`main` branch

`release/9.0` branch

`release/8.0` branch

`main` branch

`release/9.0` branch

`release/8.0` branch

`main` branch

`release/9.0` branch

`release/8.0` branch

`main` branch

`release/9.0` branch

`release/8.0` branch

antonfirsov commented Dec 10, 2024 •

edited

Loading

`main` branch

`release/9.0` branch

`release/8.0` branch

`main` branch

`release/9.0` branch

`release/8.0` branch

`main` branch

`release/9.0` branch

`release/8.0` branch