TDVT failures with Actual being equal to Expected

Question

TDVT failures with Actual being equal to Expected

GumpacG opened this issue a year ago · comments

About You:
Name: Guian Gumpac
Company: Improving

Your question:
How come I am getting test failures for tests when my actual results are the same as the expected results?

Affected test cases:

Query.Filter_SliceAggQUnaggQUnaggDT_AxisQOYear
Query.Filter_SliceAggQUnaggQ_AxisQOYear
Filter.Add_to_context
Query.Dates_MDY
Query.Unaggregated_Filter

For example, Query.Filter_SliceAggQUnaggQUnaggDT_AxisQOYear's expected and my actual is:
Expected:

"EAST"
"Barbara Fisher"
62180.0
491.0
2002
"EAST"
"Barbara Fisher"
257629.0
1135.0
2001
"EAST"
"Dario Medina"
88698.0
680.0
2002
"EAST"
"Dario Medina"
108758.0
1755.0
2001
"EAST"
"Dave Hallsten"
112045.0
431.0
2002
"EAST"
"Dave Hallsten"
216261.0
1239.0
2001
"EAST"
"David Flashing"
129300.0
903.0
2002
"EAST"
"David Flashing"
290536.0
1991.0
2001
"EAST"
"David Wiener"
88361.0
401.0
2002
"EAST"
"David Wiener"
466789.0
2453.0
2001
"EAST"
"Deanra Eno"
187930.0
680.0
2002
"EAST"
"Deanra Eno"
309042.0
2019.0
2001
"EAST"
"Debra Catini-Entman"
49439.0
500.0
2002
"EAST"
"Debra Catini-Entman"
248659.0
1576.0
2001
"EAST"
"Elizabeth Moffitt"
50749.0
512.0
2002
"EAST"
"Elizabeth Moffitt"
315255.0
2303.0
2001
"EAST"
"Ellis Ballard"
105488.0
564.0
2002
"EAST"
"Ellis Ballard"
389843.0
2186.0
2001
"EAST"
"Frank Olsen"
263523.0
779.0
2002
"EAST"
"Frank Olsen"
347670.0
1408.0
2001

Actual:

"EAST"
"Barbara Fisher"
62180.0
491.0
2002
"EAST"
"Barbara Fisher"
257629.0
1135.0
2001
"EAST"
"Dario Medina"
88698.0
680.0
2002
"EAST"
"Dario Medina"
108758.0
1755.0
2001
"EAST"
"Dave Hallsten"
112045.0
431.0
2002
"EAST"
"Dave Hallsten"
216261.0
1239.0
2001
"EAST"
"David Flashing"
129300.0
903.0
2002
"EAST"
"David Flashing"
290536.0
1991.0
2001
"EAST"
"David Wiener"
88361.0
401.0
2002
"EAST"
"David Wiener"
466789.0
2453.0
2001
"EAST"
"Deanra Eno"
187930.0
680.0
2002
"EAST"
"Deanra Eno"
309042.0
2019.0
2001
"EAST"
"Debra Catini-Entman"
49439.0
500.0
2002
"EAST"
"Debra Catini-Entman"
248659.0
1576.0
2001
"EAST"
"Elizabeth Moffitt"
50749.0
512.0
2002
"EAST"
"Elizabeth Moffitt"
315255.0
2303.0
2001
"EAST"
"Ellis Ballard"
105488.0
564.0
2002
"EAST"
"Ellis Ballard"
389843.0
2186.0
2001
"EAST"
"Frank Olsen"
263523.0
779.0
2002
"EAST"
"Frank Olsen"
347670.0
1408.0
2001

Luke Petschauer · Answer 1 · Fri Sep 15 2023 22:15:50 GMT+0800 (China Standard Time)

Internal tracking: W-14128514

Ross Brown · Answer 2 · Wed Sep 27 2023 02:49:35 GMT+0800 (China Standard Time)

Hi, can you attach one of the actual files generated to this issue to help us diagnose the issue. Thanks.

Guian Gumpac · Answer 3 · Wed Sep 27 2023 04:09:55 GMT+0800 (China Standard Time)

Hi @rosswbrown, here are the generated files.
tdvt_actuals_combined.zip
tdvt_log_combined.txt
test_results_combined.csv
tabquery_logs.zip

Ross Brown · Answer 4 · Thu Sep 28 2023 01:07:49 GMT+0800 (China Standard Time)

Reviewing the tdvt_actuals_combined.zip for the following tests I'm seeing each of them report a slightly more informative error message in the actual.setup.<testname>_diff.xml files: Different number of tuples.

Query.Filter_SliceAggQUnaggQUnaggDT_AxisQOYear
Filter.Add_to_context
Query.Dates_MDY
Query.Unaggregated_Filter

I'm not seeing any failure of Query.Filter_SliceAggQUnaggQ_AxisQOYear.

Comparing the individual actual.setup.<testname>.xml to the expected.setup.<testname>.xml files in the repo in a diffing tool I can see many of the tuples missing, so the error looks correct.

Guian Gumpac · Answer 5 · Thu Sep 28 2023 01:46:36 GMT+0800 (China Standard Time)

Awesome, wasn't aware of more in depth result files. Thanks so much!