-
Notifications
You must be signed in to change notification settings - Fork 4
Closed
Description
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 8c58fc2347820ce48e09605d8adddb993df9ebb5 |
Compare | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
Diff | Diff |
Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
EncodeUtf8 - Duration of single invocation | 24.25 ns | 27.38 ns | 1.13 | 0.06 | False | |||||
EncodeUtf8 - Duration of single invocation | 86.99 ns | 98.66 ns | 1.13 | 0.03 | False | |||||
EncodeUtf8 - Duration of single invocation | 56.38 ns | 60.28 ns | 1.07 | 0.04 | False | |||||
EncodeUtf8 - Duration of single invocation | 68.70 ns | 75.82 ns | 1.10 | 0.04 | False | |||||
EncodeUtf8 - Duration of single invocation | 54.40 ns | 66.48 ns | 1.22 | 0.04 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'
Payloads
Histogram
System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,hello "there",16)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.38339939908008 > 25.32808934198907.
IsChangePoint: Marked as a change because one of 10/25/2022 2:34:53 PM, 11/7/2022 4:10:19 PM, 11/10/2022 9:13:35 AM, 11/18/2022 8:48:12 PM, 12/1/2022 7:29:59 PM, 12/17/2022 10:44:50 PM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -18.058001012376725 (T) = (0 -27.46168470771202) / Math.Sqrt((0.8326968207866183 / (29)) + (0.013254001035322508 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.12771116366632132 = (24.351700676998586 - 27.46168470771202) / 24.351700676998586 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,&lorem ipsum=dolor sit amet,512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 98.66323554506643 > 91.24919194338443.
IsChangePoint: Marked as a change because one of 10/31/2022 10:23:25 PM, 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -19.80649573019915 (T) = (0 -98.59663915861687) / Math.Sqrt((8.486130171467286 / (29)) + (0.0009985656060789638 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.12193298874261922 = (87.88104115658173 - 98.59663915861687) / 87.88104115658173 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,hello "there",512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.27949161899527 > 59.25339992121415.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 11/7/2022 4:10:19 PM, 11/10/2022 9:13:35 AM, 12/1/2022 7:29:59 PM, 12/3/2022 7:24:45 AM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -16.995514013921436 (T) = (0 -59.752643845121426) / Math.Sqrt((0.4719556899203906 / (29)) + (0.2326512657276228 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.054390844842166344 = (56.67029843574363 - 59.752643845121426) / 56.67029843574363 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: JavaScript,&Hello+<World>!,512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 75.82173553085313 > 72.94025391330943.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 11/7/2022 4:10:19 PM, 11/10/2022 9:13:35 AM, 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -18.63914200299813 (T) = (0 -75.77183580670749) / Math.Sqrt((2.936184503467602 / (29)) + (0.15829679708616626 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.0899511845616439 = (69.51855906939664 - 75.77183580670749) / 69.51855906939664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,&lorem ipsum=dolor sit amet,16)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 66.48226516794898 > 57.10550034286644.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 11/7/2022 4:10:19 PM, 11/10/2022 9:13:35 AM, 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -18.74017308510893 (T) = (0 -66.51670872735204) / Math.Sqrt((10.325223086645083 / (28)) + (0.004032309356583842 / (14))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (14) - 2, .025) and -0.2064944704777867 = (55.13221183766437 - 66.51670872735204) / 55.13221183766437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 8c58fc2347820ce48e09605d8adddb993df9ebb5 |
Compare | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
Diff | Diff |
Regressions in System.Collections.TryAddGiventSize<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Dictionary - Duration of single invocation | 4.11 μs | 4.63 μs | 1.13 | 0.04 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddGiventSize<Int32>*'
Payloads
Histogram
System.Collections.TryAddGiventSize<Int32>.Dictionary(Count: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.62625227239137 > 4.331658172567962.
IsChangePoint: Marked as a change because one of 10/17/2022 8:10:07 PM, 10/19/2022 7:27:15 PM, 10/25/2022 2:34:53 PM, 11/3/2022 3:03:31 PM, 11/18/2022 8:48:12 PM, 12/7/2022 8:33:47 PM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -29.883741928896654 (T) = (0 -4640.507786964781) / Math.Sqrt((7330.286335941202 / (29)) + (415.0575826401282 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.12135473611799896 = (4138.304889164409 - 4640.507786964781) / 4138.304889164409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 8c58fc2347820ce48e09605d8adddb993df9ebb5 |
Compare | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
Diff | Diff |
Regressions in Burgers
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Test3 - Duration of single invocation | 130.63 ms | 185.03 ms | 1.42 | 0.02 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Burgers*'
Payloads
Histogram
Burgers.Test3
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 185.02650623076923 > 137.19255053269234.
IsChangePoint: Marked as a change because one of 12/16/2022 6:44:06 PM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -1112.7765880908323 (T) = (0 -185122771.44463634) / Math.Sqrt((13240655093.700161 / (29)) + (27129338344.55862 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.41669925035144206 = (130671892.00439878 - 185122771.44463634) / 130671892.00439878 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 8c58fc2347820ce48e09605d8adddb993df9ebb5 |
Compare | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
Diff | Diff |
Regressions in System.Collections.ContainsTrue<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
HashSet - Duration of single invocation | 2.71 μs | 3.89 μs | 1.44 | 0.12 | False | |||||
Stack - Duration of single invocation | 11.18 μs | 14.28 μs | 1.28 | 0.15 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*'
Payloads
Histogram
System.Collections.ContainsTrue<Int32>.HashSet(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.8907520246405722 > 2.8081831122808456.
IsChangePoint: Marked as a change because one of 10/10/2022 3:36:43 PM, 11/7/2022 4:10:19 PM, 11/16/2022 6:51:59 PM, 12/6/2022 2:12:43 AM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -20.382220831488578 (T) = (0 -3887.1581556971505) / Math.Sqrt((89606.66516106357 / (29)) + (39.38660082669739 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.41163258525157526 = (2753.6613962509214 - 3887.1581556971505) / 2753.6613962509214 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsTrue<Int32>.Stack(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.279416201891715 > 11.746494870154343.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 10/17/2022 12:47:34 PM, 10/25/2022 12:27:43 PM, 12/17/2022 10:44:50 PM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -8.432930007529418 (T) = (0 -13658.81911015653) / Math.Sqrt((301396.32639466535 / (29)) + (415571.30843547493 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.14102872541050823 = (11970.61809749136 - 13658.81911015653) / 11970.61809749136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 8c58fc2347820ce48e09605d8adddb993df9ebb5 |
Compare | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
Diff | Diff |
Regressions in System.Collections.ContainsFalse<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
List - Duration of single invocation | 17.56 μs | 21.17 μs | 1.21 | 0.38 | False | |||||
Stack - Duration of single invocation | 16.14 μs | 21.66 μs | 1.34 | 0.21 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<Int32>*'
Payloads
Histogram
System.Collections.ContainsFalse<Int32>.List(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.173356838880416 > 18.43591835033121.
IsChangePoint: Marked as a change because one of 10/19/2022 7:27:15 PM, 10/25/2022 2:34:53 PM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -10.460864814769154 (T) = (0 -20488.20646486722) / Math.Sqrt((1031305.6060377982 / (29)) + (791933.3435009287 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.18339694871477663 = (17313.046553921195 - 20488.20646486722) / 17313.046553921195 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsFalse<Int32>.Stack(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.662782607192256 > 18.488819196403437.
IsChangePoint: Marked as a change because one of 11/3/2022 3:03:31 PM, 11/7/2022 4:10:19 PM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -13.008394141928974 (T) = (0 -20825.254297429554) / Math.Sqrt((330277.48204226414 / (29)) + (818953.8689956036 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.19779192267618312 = (17386.37062345557 - 20825.254297429554) / 17386.37062345557 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 8c58fc2347820ce48e09605d8adddb993df9ebb5 |
Compare | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
Diff | Diff |
Regressions in System.Numerics.Tests.Perf_Plane
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
CreateFromVerticesBenchmark - Duration of single invocation | 4.12 ns | 5.16 ns | 1.25 | 0.13 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Plane*'
Payloads
Histogram
System.Numerics.Tests.Perf_Plane.CreateFromVerticesBenchmark
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.161945325220399 > 4.32516936202548.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/15/2022 1:59:35 AM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -39.59038821839044 (T) = (0 -5.1753305683102795) / Math.Sqrt((0.017429968384272555 / (29)) + (0.0008638337964258109 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.24522849534533744 = (4.156129246684973 - 5.1753305683102795) / 4.156129246684973 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 8c58fc2347820ce48e09605d8adddb993df9ebb5 |
Compare | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
Diff | Diff |
Regressions in System.Collections.ContainsTrueComparer<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
HashSet - Duration of single invocation | 4.28 μs | 4.63 μs | 1.08 | 0.13 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer<Int32>*'
Payloads
Histogram
System.Collections.ContainsTrueComparer<Int32>.HashSet(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.634883012398908 > 4.494167167146445.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/3/2022 7:24:45 AM, 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -8.383402882087836 (T) = (0 -4590.878715862526) / Math.Sqrt((2824.678779357185 / (29)) + (13342.592213360631 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.06290815065136753 = (4319.167853825526 - 4590.878715862526) / 4319.167853825526 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository