Split out Arb/CoArb and Gen
garyb opened this issue · comments
After talking about core-tests with @hdgarrood in purescript/purescript#936 it got me thinking about how we might do things a little differently to help with testing etc.
If we could break out the Arbitrary and CoArbitrary classes into their own project with no dependencies we could then allow data-defining libraries like purescript-maybe and purescript-tuples provide the instances themselves, rather than them being defined in QC, which would help avoid some of the tangle of dependencies we currently have (although it certainly doesn't solve everything).
The thing that makes that difficult is Gen, as it has a bunch of dependencies on maybe, tuples, foldable, and so on... but it also seems that if we could somehow class-ify Gen then we could actually make the Arbitrary and CoArbitrary instances usable for both quickcheck and strongcheck, assuming they share enough in common to make a sensible Gen class.
Does that seem realistic, or even worth doing? /cc @paf31 @jdegoes
👍 I would also like to be able to provide an Arbitrary instance Arbitrary a => Arbitrary (Seq a) in https://github.com/hdgarrood/purescript-sequences without having to depend on all of quickcheck.
Does Gen need a typeclass? Maybe just parameterising the state would be sufficient?
data Gen s a = Gen (s -> GenOut s a)
data GenOut s a = { state :: s, value :: a }
What happened to the quickcheck-src directory idea? I quite liked that.
Oops, sorry, closed by accident.
Any abstraction over QuickCheck Gen and StrongCheck Gen would have to be at least as powerful as StrongCheck Gen. It's possible, of course.
Could the Gen module just implement its own local versions of these data structures and functions like Maybe, and use records instead of Tuple? There aren't any types with Maybe in them, so we don't need to worry about compatibility there. And the functions with Tuple could easily be converted to records. Then the dependencies could be dropped, and data libs could again depend on this lib.
I know this spits in the face of DRY, but having tests in entirely different repositories spits in some other face.
That sounds like a good way to do something about the testing situation.
Unfortunately it won't help with the reusable Arbitrary/Coarbitrary/Testable, but I did try working on that and it was pretty hard going trying to accommodate both SC and QC.
What would a quickcheck-types package look like? Just the types for Gen, and Arb and CoArb? If so, and if we can make that package have no dependencies, then my vote would be to use that, and to move the Arb/CoArb instances for Maybe, Tuple etc. into their quickcheck-src directories, adding purescript-quickcheck-types as a dev dependency for Bower.
That's pretty much what I was proposing here, but Gen is the problem.
Shared:
type GenState = { newSeed :: LCG, size :: Size }
type GenOut a = { state :: GenState, value :: a }QC:
data Gen a = Gen (GenState -> GenOut a)SC:
data GenT f a = GenT (Mealy.MealyT f GenState (GenOut a))
type Gen a = GenT Trampoline aOh, sorry, you're just talking about for QC aren't you?
For now, yes.
If the issue is just for testing the core libraries, another option is to just continue testing them without quickcheck, with asserts. They are very simple libraries after all.
If the issue is just for testing the core libraries, another option is to just continue testing them without quickcheck, with asserts.
I'm in favor of this approach. Core libraries should probably have no test dependencies and property verification packages should probably live in user-land. At some point the core libraries stop changing so the value of their test suites goes down with time (in contrast to constantly evolving libraries, whose value goes up over time).
I'm not sure I'd call some of the core libraries simple :) In particular transformers, free, maps.
Well I think the rule of thumb is: if quickcheck depends on it, you can't use quickcheck to test it. So I think those three are all ok.
I don't know if quickcheck would be a great way to test transformers or free though. What would your Arbitrary instances generate?
Also, parametricity in these cases ensures if it works for one concrete data type, it will work for all, so unit tests are as powerful as property tests.
quickcheck doesn't depend on those though (I don't think it really NEEDS to depend on much, aside from exceptions, math and random), so there doesn't seem to be a reason to only rely on unit tests. Actually, looks like maps uses quickcheck already.
As @jdegoes stated, parametricity helps quite a bit here, so we needn't worry about writing arbitrary instances for everything.
E.g. ReaderT could be monomorphized to Reader and just tested with already existing instances:
prop_FunctorId :: (String -> Number) -> String -> Boolean
prop_FunctorId r x = let r' = ReaderT $ Identity <<< r in
(runReader r' x) == runReader (id <$> r') x
prop_FunctorComp :: (String -> Number) -> (Number -> Boolean) -> (Boolean -> String) -> String -> Boolean
prop_FunctorComp r f g x = let r' = ReaderT $ Identity <<< r in
(runReader ((f >>> g) <$> r')) x == runReader (g <$> (f <$> r')) xMost all of transformers could be tested this way, and it might catch more bugs like: purescript/purescript-transformers#29 that went unnoticed for roughly 8 months. I assume similar things could work for the other core libs. But I don't have faith that unit tests are the end solution. Who knows how many bugs are in the Comonad transformers, or still lurking in a Monad transformer.