Skip to content

Regressions in Benchstone.BenchI.IniArray #82103

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 4da62e4341215905907bc93587eb44c8abb56d3f
Compare 46cb4eddbb1f4bebdea970964d7449bacde8df1a
Diff Diff

Regressions in System.Collections.Tests.Add_Remove_SteadyState<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConcurrentStack - Duration of single invocation 22.74 ns 25.60 ns 1.13 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Add_Remove_SteadyState<Int32>.ConcurrentStack(Count: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.595127681490542 > 23.95560846666927.
IsChangePoint: Marked as a change because one of 12/1/2022 11:48:13 PM, 2/9/2023 4:10:34 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsRegressionStdDev: Marked as regression because -6.596255372280926 (T) = (0 -25.502845122630248) / Math.Sqrt((2.371620314086687 / (40)) + (1.258006927530987 / (20))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (20) - 2, .025) and -0.09939937751099344 = (23.197070731809863 - 25.502845122630248) / 23.197070731809863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 4da62e4341215905907bc93587eb44c8abb56d3f
Compare 46cb4eddbb1f4bebdea970964d7449bacde8df1a
Diff Diff

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Int

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 12.05 ns 13.94 ns 1.16 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*'

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.937877174207754 > 12.655306191910952.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 4:10:34 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsRegressionStdDev: Marked as regression because -208.95059889922348 (T) = (0 -14.024128670234585) / Math.Sqrt((0.0007295337975537152 / (40)) + (0.001386262659928889 / (20))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (20) - 2, .025) and -0.16199591997452498 = (12.06899992432163 - 14.024128670234585) / 12.06899992432163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 4da62e4341215905907bc93587eb44c8abb56d3f
Compare 46cb4eddbb1f4bebdea970964d7449bacde8df1a
Diff Diff

Regressions in Benchstone.BenchI.IniArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 58.24 ms 109.01 ms 1.87 0.04 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.IniArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.IniArray.Test


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.0112425 > 61.063725187500005.
IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsRegressionStdDev: Marked as regression because -141.4125909953871 (T) = (0 -108990158.125) / Math.Sqrt((1017796800282.5603 / (40)) + (1971298511332.8125 / (20))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (20) - 2, .025) and -0.8413090760895805 = (59191669.4162309 - 108990158.125) / 59191669.4162309 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; Benchstone.BenchI.IniArray.Test()
       sub       rsp,28
       mov       rcx,offset MT_System.Char[]
       mov       edx,10
       call      CORINFO_HELP_NEWARR_1_VC
       xor       edx,edx
M00_L00:
       xor       ecx,ecx
       nop       dword ptr [rax]
M00_L01:
       mov       r8d,ecx
       mov       word ptr [rax+r8*2+10],20
       inc       ecx
       cmp       ecx,10
       jl        short M00_L01
       inc       edx
       cmp       edx,989680
       jl        short M00_L00
       add       rsp,28
       ret
; Total bytes of code 65

Compare Jit Disasm

; Benchstone.BenchI.IniArray.Test()
       sub       rsp,28
       mov       rcx,offset MT_System.Char[]
       mov       edx,10
       call      CORINFO_HELP_NEWARR_1_VC
       xor       edx,edx
M00_L00:
       xor       ecx,ecx
       nop       dword ptr [rax]
M00_L01:
       mov       r8d,ecx
       mov       word ptr [rax+r8*2+10],20
       inc       ecx
       cmp       ecx,10
       jl        short M00_L01
       inc       edx
       cmp       edx,989680
       jl        short M00_L00
       add       rsp,28
       ret
; Total bytes of code 65

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions