Output of failing property is confused by comments between signature and definition
brprice opened this issue · comments
With certain comment placement, the output of a failing test can be rather confusing, with failures and annotations ascribed to the wrong source lines.
The bug
Consider the file
import Hedgehog
import Hedgehog.Main
prop_bug :: Property
-- This comment moves all the test output (annotations and diffs and failure indication) down by a line.
-- (Well, two now.) The line numbers in the output are attached to the wrong code, but the right output.
-- (Ok, three.) Note that this comment does not appear in the source printout on error
prop_bug = property $ do
-- Comments here (Line 9) are not problematic, and show up in the source printout on error
annotate "Line 10"
-- Here (Line 11) is also fine
i <- forAll $ pure 0
i === 1
success
success
main :: IO ()
main = defaultMain [check prop_bug]
which has comments between prop_bug :: Property
and prop_bug = ...
.
The output of this failing test attaches annotations and failures to the wrong source lines. It also attaches the line numbers incorrectly (the annotations etc are attached to the correct line number, but the wrong source code is printed).
The output is (commentary below)
✗ <interactive> failed at Main.hs:13:3
after 1 test.
┏━━ Main.hs ━━━
4 ┃ prop_bug :: Property
5 ┃ prop_bug = property $ do
6 ┃ -- Comments here (Line 9) are not problematic, and show up in the source printout on error
7 ┃ annotate "Line 10"
8 ┃ -- Here (Line 11) is also fine
9 ┃ i <- forAll $ pure 0
10 ┃ i === 1
┃ │ Line 10
11 ┃ success
12 ┃ success
┃ │ 0
━━━ Failed (- lhs) (+ rhs) ━━━
- 0
+ 1
This failure can be reproduced by running:
> recheck (Size 0) (Seed 13577958291790835144 10236192217858061691) <property>
Commentary
In the output we see the source code of the failing property, with attached line numbers.
It starts fine, with the first line correctly numbered
✗ <interactive> failed at Main.hs:13:3
after 1 test.
┏━━ Main.hs ━━━
4 ┃ prop_bug :: Property
But after that the line numbers do not correspond to the source code printed:
5 ┃ prop_bug = property $ do
is the next line of output. We have skipped the three comment lines, but not updated the line numbering correctly. So far this is a minor annoyance, but things are made worse because only the source code is shifted - other output such an annotations are not. Thus we continue
6 ┃ -- Comments here (Line 9) are not problematic, and show up in the source printout on error
7 ┃ annotate "Line 10"
8 ┃ -- Here (Line 11) is also fine
Notice that hedgehog has not (yet) displayed anything for this annotation on line (output number)7/(source line)10, as it is attached to output line number 10: 3 lines further down!
Continuing, we see the same shifted output problem for the forAll
9 ┃ i <- forAll $ pure 0
10 ┃ i === 1
┃ │ Line 10
Now the annotate
output shows up, confusingly delayed. The test i === 1
fails, but this has not been reported yet.
11 ┃ success
12 ┃ success
┃ │ 0
━━━ Failed (- lhs) (+ rhs) ━━━
- 0
+ 1
Finally, we drop off the end and report the failure, which is not ascribed to any particular line (because it has been "shifted off the end").
For more confusion, if we had an extra success
line at the end, we would see
13 ┃ success
┃ ^^^^^^^
┃ │ ━━━ Failed (- lhs) (+ rhs) ━━━
┃ │ - 0
┃ │ + 1
which apparently blames a trivial test for our earlier failure.
Workaround
Since this bug is triggered by comments between signatures and definitions:
prop_bug :: Property
-- I.e. comments here
prop_bug = property $ do
it is easy to avoid by writing such comments elsewhere
-- Here is fine
prop_bug :: Property
prop_bug = property $ do
-- Here is also fine