bdna analysis is always showing passed

[ravimainali]

Description

we have two run in aggregate data set where the latest run Meantime is 200, whereas the other run Meantime is 100 but while running the analysis it is showing passed, but actually it should fail with 100 percent increase in Meantime tolerance .

Configuration

{
"creation": "2024-03-26T17:10:34+05:32",
"buildNumber": "1.0.0.11",
"pinned": false,
"buildUri": null,
"branchName": null,
"commitSha": null,
"tags": null,
"runs": [
{
"creation": "2024-03-26T17:10:34+05:32",
"benchmarkDotNetVersion": "0.13.12",
"results": [
{
"fullName": "BenchmarkDotNetInDocker.MyBenchmark.Sum",
"namespace": "BenchmarkDotNetInDocker",
"type": "MyBenchmark",
"method": "Sum",
"parameters": "",
"meanTime": 200.0257419322928,
"minTime": 500.9189009666443,
"q1Time": 52.7650326490402,
"medianTime": 54.8924565315247,
"q3Time": 59.4338625669479,
"maxTime": 67.7644848823547
}
]
}
]
},
{
"creation": "2024-03-26T17:10:34+05:30",
"buildNumber": "1.0.0.10",
"pinned": true,
"buildUri": null,
"branchName": null,
"commitSha": null,
"tags": null,
"runs": [
{
"creation": "2024-03-26T17:10:34+05:30",
"benchmarkDotNetVersion": "0.13.12",
"results": [
{
"fullName": "BenchmarkDotNetInDocker.MyBenchmark.Sum",
"namespace": "BenchmarkDotNetInDocker",
"type": "MyBenchmark",
"method": "Sum",
"parameters": "",
"meanTime": 100.0257419322928,
"minTime": 100.9189009666443,
"q1Time": 100.7650326490402,
"medianTime": 100.8924565315247,
"q3Time": 50.4338625669479,
"maxTime": 100.7644848823547
}
]
}
]

Other information

[Tony Knight]

Thank you, I will have a look.

[Tony Knight]

Hi @ravimainali,

Thanks for the sample text. It looks very much that the data you've provided has been changed in some way - quite probably a manual change.

The first run object (that has a mean time of 200.0257419322928) has a creation time of 2024-03-26T17:10:34+05:32, whereas the second run (with mean time 100.0257419322928) has a creation time of 2024-03-26T17:10:34+05:30. These time stamps are significant: they're used to determine the order of execution and the sequencing of trend analysis.

The only difference between 2024-03-26T17:10:34+05:30 and 2024-03-26T17:10:34+05:32 is the time zone offset: +5:30 and +5:32 is off by 2 minutes. The date time component (2024-03-26T17:10:34) is exactly the same in both runs.

The additional 2 seconds to the time zone offset puts the first run chronologically before the second run, and so the trend computation sees a first run of 200.0257419322928 followed by a run of 100.0257419322928. That means bdna believes that there's a 50% improvement in run time and so reports success, as expected.

If you change the 200 msec time to a reasonable time (that is after 2024-03-26T17:10:34+05:30) with a tolerance of 1, you'll see bdna returns failure, as expected.

I can only assume someone's manually changed the timestamps, because Benchmark dotnet runs do not take zero time, the only difference between the two runs' time is the 2 second time zone offset, and a search through UTC standards show no standard time zone offset of +5:32.

Hope this helps,
T