Comparing sequences fails
abelbraaksma opened this issue · comments
Description
The following fails:
Seq.init 100 ((+) 1)
|> should equal (Seq.init 100 ((+) 1))
Repro steps
See above.
Expected behavior
Should succeed.
Actual behavior
Gives test error:
FsUnit.Xunit+MatchException : Exception of type 'FsUnit.Xunit+MatchException' was thrown.
Expected: Equals seq [1; 2; 3; 4; ...]
Actual: seq [1; 2; 3; 4; ...]
Which is rather confusing as it doesn't show where the two sequences diverge (spoiler alert: they don't ;) ).
Known workarounds
A workaround is to not use sequences in tests
I.e.: turning them into eager collections works fine:
Seq.init 100 ((+) 1)
|> Array.ofSeq
|> should equal (Seq.init 100 ((+) 1) |> Array.ofSeq)
Related information
Version 5.0.5
Using: FsUnit.Xunit
Hi @abelbraaksma,
there is an operator (equivalent) to compare lists, arrays and sequences that is not implemented for xUnit.
MsTest and NUnit have collection asserts for that case. xUnit doesn't, that's because it isn't implemented in FsUnit.Xunit.
xUnit is not idiomatic for that.
Full list of operators: http://fsprojects.github.io/FsUnit/operators.html
Yes, so basically this doesn’t work. Also, equivalent I believe doesn’t test sequence order, right @CaptnCodr? My guess is that this could work by adding a type test in the equals code that just pairwise iterates over any given sequence.
Yes, equivalent only tests if the elements are equal, order doesn't matter.
I don't know why e.g. seq {1;2;3} = seq {1;2;3} results in false; and seq [1;2;3] = seq [1;2;3] results in true.
That's basically what happens in equal: actual = expected.
I think that's by design.
I don't know if that's a fix worth to just "fix" the equality of seq. What do you think, @sergey-tihon?
I guess that it is does not work for sequences because they can be infinite.
It would be weird to get in infinite loop inside equal
It would be weird to get in infinite loop inside equal
Since this is a test framework, if someone requests if A=B, then we should do our best to honour that request. If someone wants to compare infinite sequences this way, they’ll simply get a test time out. And these are small sequences, but that’s really up to the programmer.
Yes and no,
when there is no sequence built-in sequence equality comparison in F# then it gets harder than you think.
Consider that you can have e.g. nested sequences or sequences with objects those cannot be compared, at this point it gets complicated and we don't want to reinvent the wheel with complicated implementations. Simple implementations or constraints directly from NUnit, xUnit and MsTest are ok and this is not simple.
I recommend your mentioned workaround in the beginning to cast it to an array that has equality comparison built-in.
Simple implementations or constraints directly from NUnit, xUnit and MsTest are ok and this is not simple.
But xUnit has a built-in sequence comparison on Assert.Equal<_>:
[<Fact>]
let ``Test collections`` () =
let x = Seq.init 100 ((+) 1)
let y = seq {
yield! Seq.init 99 ((+) 1)
yield 43
}
Assert.Equal<seq<_>>(x, y)
Which is results in a very useful message:
Assert.Equal() Failure
↓ (pos 99)
Expected: ···.., 96, 97, 98, 99, 100]
Actual: ···.., 96, 97, 98, 99, 43]
↑ (pos 99)
Some random thoughts below (with a suggestion at the bottom):
Tbh, F# behaves really weird here:
> let x = [1..3] |> Seq.ofList;;
val x: seq<int> = [1; 2; 3]
> let y = [1..3] |> Seq.ofList;;
val y: seq<int> = [1; 2; 3]
> x = y;;
val it: bool = true
And:
> let x = [1..100] |> Seq.ofList;;
val y: seq<int> =
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; ...]
> let y = [1..100] |> Seq.ofList;;
val y: seq<int> =
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; ...]
> x = y;;
val it: bool = true
And:
> let x = Seq.empty<int>;;
val x: seq<int>
> let y = Seq.empty<int>;;
val y: seq<int>
> x = y;;
val it: bool = true
But:
> let x = Seq.init 0 ((+) 1);;
val x: seq<int>
> let y = Seq.init 0 ((+) 1);;
val y: seq<int>
> x = y;;
val it: bool = false
But:
> let x = Seq.init 2 ((+) 1);;
val x: seq<int>
> let y = Seq.init 2 ((+) 1);;
val y: seq<int>
> x = y;;
val it: bool = false
Which to me means that equality in sequences is a mouse trap in F#. I can kinda understand that sequences are not directly comparable (them maybe having side effects and what not). And it appears to me that the only thing that is being compared is reference equality (though: why then are the first two examples true? They are not reference-equal).
You'd expect something like this though, which may be a candidate extension?
> let x = Seq.init 100 ((+) 1);;
val x: seq<int>
> let y = Seq.init 100 ((+) 1);;
val y: seq<int>
> (x, y) ||> Seq.forall2 (=);;
val it: bool = true
Thinking out loud: maybe we can add something like x |> should be (pairwiseEqual y)?
@abelbraaksma When you have an empty seq it does not succeed anymore:
> (Seq.init 100 ((+) 1), Seq.empty) ||> Seq.forall2 (=);;
val it: bool = true
I was thinking about to make an extra operator for FsUnit.xUnit that isn't quite far away.
equalSeq with just Assert.Equal<seq<_>>(x, y) as an implementation could work pretty well, isn't it @sergey-tihon?
e.g.: seq { 1; 2; 3 } |> should equalSeq (seq { 1; 2; 3 })
@abelbraaksma When you have an empty seq it does not succeed anymore:
Hmm, yeah. Related: dotnet/fsharp#14121, which I raised, because there's quite a bit of weird stuff going on when applying functions to two sequences. I.e., List.map2 and Seq.map2 do not behave the same.
Which basically means that two sequences are only Seq.forall2 -> true in cases where all items in both sequences until one is exhausted are true. But using List.forall2 or Array.forall2 it'll raise on unequal lengths.
This is not the same as Assert.Equals<seq<_>>, which will trigger when the sequences are of unequal length.
seq { 1; 2; 3 } |> should equalSeq (seq { 1; 2; 3 })
That's a good idea. For myself, I just added seq { 1; 2; 3 } |> should be (pairwiseEqual (seq { 1; 2; 3 }) or something along those lines, which should also call into Assert.Equals<seq<_>>, not Seq.forall2, apparently.
Arguably, equalSeq is better discoverable, as it will appear in the dropdown box when typing x |> should equal..., and users may be less inclined to use equal with sequences and falling into this trap.
Hey @abelbraaksma,
equalSeq is now available in Release v5.1.0.
This has been uploaded to Nuget just now. 🙂
Cheers!
@CaptnCodr that’s awesome, thanks! I’ll be updating my code, I found myself being very careful when testing sequences, this will certainly help (and with my current work on TaskSeq, all I do is comparing sequences 😆).
You are not the only one who needs/needed this. 😉
I was counting on that, otherwise why add it, right?