-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Closed
Labels
area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
Milestone
Description
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.18362 |
Baseline | 4da62e4341215905907bc93587eb44c8abb56d3f |
Compare | 46cb4eddbb1f4bebdea970964d7449bacde8df1a |
Diff | Diff |
Regressions in System.Collections.Tests.Add_Remove_SteadyState<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
ConcurrentStack - Duration of single invocation | 22.74 ns | 25.60 ns | 1.13 | 0.08 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState<Int32>*'
Payloads
Histogram
System.Collections.Tests.Add_Remove_SteadyState<Int32>.ConcurrentStack(Count: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.595127681490542 > 23.95560846666927.
IsChangePoint: Marked as a change because one of 12/1/2022 11:48:13 PM, 2/9/2023 4:10:34 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsRegressionStdDev: Marked as regression because -6.596255372280926 (T) = (0 -25.502845122630248) / Math.Sqrt((2.371620314086687 / (40)) + (1.258006927530987 / (20))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (20) - 2, .025) and -0.09939937751099344 = (23.197070731809863 - 25.502845122630248) / 23.197070731809863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.18362 |
Baseline | 4da62e4341215905907bc93587eb44c8abb56d3f |
Compare | 46cb4eddbb1f4bebdea970964d7449bacde8df1a |
Diff | Diff |
Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Int
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
GetHashCodeBenchmark - Duration of single invocation | 12.05 ns | 13.94 ns | 1.16 | 0.04 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*'
Payloads
Histogram
System.Runtime.Intrinsics.Tests.Perf_Vector128Int.GetHashCodeBenchmark
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.937877174207754 > 12.655306191910952.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 4:10:34 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsRegressionStdDev: Marked as regression because -208.95059889922348 (T) = (0 -14.024128670234585) / Math.Sqrt((0.0007295337975537152 / (40)) + (0.001386262659928889 / (20))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (20) - 2, .025) and -0.16199591997452498 = (12.06899992432163 - 14.024128670234585) / 12.06899992432163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.18362 |
Baseline | 4da62e4341215905907bc93587eb44c8abb56d3f |
Compare | 46cb4eddbb1f4bebdea970964d7449bacde8df1a |
Diff | Diff |
Regressions in Benchstone.BenchI.IniArray
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Test - Duration of single invocation | 58.24 ms | 109.01 ms | 1.87 | 0.04 | False | Trace | Trace |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.IniArray*'
Payloads
Histogram
Benchstone.BenchI.IniArray.Test
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.0112425 > 61.063725187500005.
IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsRegressionStdDev: Marked as regression because -141.4125909953871 (T) = (0 -108990158.125) / Math.Sqrt((1017796800282.5603 / (40)) + (1971298511332.8125 / (20))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (20) - 2, .025) and -0.8413090760895805 = (59191669.4162309 - 108990158.125) / 59191669.4162309 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```### Baseline Jit Disasm
```assembly
; Benchstone.BenchI.IniArray.Test()
sub rsp,28
mov rcx,offset MT_System.Char[]
mov edx,10
call CORINFO_HELP_NEWARR_1_VC
xor edx,edx
M00_L00:
xor ecx,ecx
nop dword ptr [rax]
M00_L01:
mov r8d,ecx
mov word ptr [rax+r8*2+10],20
inc ecx
cmp ecx,10
jl short M00_L01
inc edx
cmp edx,989680
jl short M00_L00
add rsp,28
ret
; Total bytes of code 65
Compare Jit Disasm
; Benchstone.BenchI.IniArray.Test()
sub rsp,28
mov rcx,offset MT_System.Char[]
mov edx,10
call CORINFO_HELP_NEWARR_1_VC
xor edx,edx
M00_L00:
xor ecx,ecx
nop dword ptr [rax]
M00_L01:
mov r8d,ecx
mov word ptr [rax+r8*2+10],20
inc ecx
cmp ecx,10
jl short M00_L01
inc edx
cmp edx,989680
jl short M00_L00
add rsp,28
ret
; Total bytes of code 65
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Metadata
Metadata
Assignees
Labels
area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark