Is this a bug or am I using list_of wrong?

[John DeWyze]

With the following example:

property "last list_of doesn’t work" do
  check all list1 <- list_of(integer()),
            list2 <- list_of(integer()), initial_size: 5 do
    assert false
  end
end

list1 will have sizes ranging fro 0 to 5, but list2 is always empty. This seems to always be the case if list_of is the last clause. I.e. if I add a list3, then list2 is populated, but list3 is empty.

[Andrea Leopardi]

It seems to be a bug. I'll take a look, thanks for reporting. :)

[Andrea Leopardi]

Hey @dewyze, I am looking into this and I cannot reproduce it. This is the code I'm using:

defmodule RegressionTest do
  use ExUnit.Case
  use ExUnitProperties

  property "regression #128" do
    check all list1 <- list_of(integer()),
              list2 <- list_of(integer()),
              initial_size: 5,
              max_shrinking_steps: 0 do
      IO.inspect(list1, label: "list1")
      IO.inspect(list2, label: "list2")
      assert false
    end
  end
end

Using this code, the lists are always populated very randomly with list2 often being larger than an empty list. Can you provide the code you used to verify that list2 is always empty?

[John DeWyze]

I am using this and it never passes.

property "regression #128" do
  check all list1 <- list_of(integer()),
            list2 <- list_of(integer()),
            initial_size: 5,
            max_shrinking_steps: 0 do
    assert length(list2) > 0
  end
end

Here is some more system info:

# mix.exs
defp deps do                                     
  [                                              
    {:stream_data, "~> 0.1", only: [:dev, :test]}
  ]
end


→ cat mix.lock 
%{
  "stream_data": {:hex, :stream_data, "0.4.3", "62aafd870caff0849a5057a7ec270fad0eb86889f4d433b937d996de99e3db25", [:mix], [], "hexpm"},
}


→ elixir -v
Erlang/OTP 22 [erts-10.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.8.2 (compiled with Erlang/OTP 22)

[John DeWyze]

Here is a repo with my exact setup, fails every time I run mix test: https://github.com/dewyze/stream_data_test

[John DeWyze]

The example I pasted before I realize is not a good example, because sometimes the list will be empty, and often it would say "failed after 19 runs" or something. I pushed a better example to the repo:

property "regression #128" do
  check all list1 <- list_of(integer()),
            list2 <- list_of(integer()),
            initial_size: 100,
            max_shrinking_steps: 0 do
    assert false
  end
end

When that fails, at least on my machine, the results always look something like this:

1) property regression #128 (StreamDataTestTest)
   test/stream_data_test_test.exs:5
   Failed with generated values (after 0 successful runs):

       * Clause:    list1 <- list_of(integer())
         Generated: [-61, 91, -51, -66, 11, 0, -71, -5, -59, 27, 54, 62, 74, 1, 56, 79, 28, 18, 55, -10, 90, 32, -39, 58, 99, 14, -40, 29, -28, 66, 33, -12, -87, 91, -2, -15, -90, -11, -43, -68, -55, -92, -51, 97, 58, -5, 50, 64, 64, 84, ...]

       * Clause:    list2 <- list_of(integer())
         Generated: []

   Expected truthy, got false
   code: assert false
   stacktrace:
     test/stream_data_test_test.exs:10: anonymous fn/1 in StreamDataTestTest."property regression #128"/1
     (stream_data) lib/stream_data.ex:2114: StreamData.shrink_failure/6
     (stream_data) lib/stream_data.ex:2078: StreamData.check_all/7
     test/stream_data_test_test.exs:6: (test)

Is it possible that just on the first run, the last list_of always produced an empty list first?

[Andrea Leopardi]

Ok I know what was happening. There was a bug until earlier today where max_shrinking_steps: 0 wouldn't work (see ebf5b2a). This meant that when you did assert false and the test failed, we shrunk the generated inputs and stopped when list2 was completely shrunk to [] (whether this behaviour is correct is out of scope and very hard to fix anyways). The generated values that are printed are always shrunk, not the actually generated ones. Let me know if you have questions, but the morale is that it was never a bug and your reproducing repo only behaves like that because of the bug that's fixed in master.

[John DeWyze]

Thanks for the clarification!