Lazy can't see through public -> private type aliases

[Ashley Mercer]

Java: OpenJDK 1.8.0_202
Scala: 2.12.4
shapeless: 2.3.3

Minimal reproduction: I believe all four cases shown below should compile, but only the first three do. The fourth fails because Lazy seems to lose the detailed knowledge of NonEmptyChain[String] (but works in the simple case where the first type parameter case is just String).

trait TC[F[_]]

type ValidatedString[A]    = cats.data.Validated[String, A]
type ValidatedNecString[A] = cats.data.Validated[NonEmptyChain[String], A]

{
  implicit def mkTypeClass[F[_]](implicit F: cats.Functor[F]): TC[F] = ???
  implicitly[TC[ValidatedString]]        // works
  implicitly[TC[ValidatedNecString]]     // works
}

{
  implicit def mkTypeClassLazy[F[_]](implicit F: Lazy[cats.Functor[F]]): TC[F] = ???
  implicitly[TC[ValidatedString]]        // works
  implicitly[TC[ValidatedNecString]]     // doesn't compile
}

I've had a look through related tickets but I don't think this is covered elsewhere.

[Ashley Mercer]

Having poked this a bit more, I wonder if it's some weird interaction between Lazy and the (slightly strange) way in which NonEmptyChain is declared as a no-overhead newtype.

Replacing NonEmptyChain with regular Chain allows everything to work normally, i.e. the following compiles as expected:

type ValidatedChainString[A] = cats.data.Validated[Chain[String], A]

{
  implicit def mkTypeClassLazy[F[_]](implicit F: Lazy[cats.Functor[F]]): TC[F] = ???
  implicitly[TC[ValidatedChainString]]
}

Is it possible that Lazy can't "see through" the new type to figure out that NonEmptyChain is really just an alias for Chain, and that it should look in the latter's companion object for instances?

EDIT: the required implicits are re-declared on the NonEmptyChainImpl object so this is incorrect.

[Georgi Krastev]

You could try this patch: #797

[Ashley Mercer]

You could try this patch: #797

No joy, unfortunately: I applied this patch on top of 2.3.3 and it still doesn't work. Taking a quick look at the patch, it seems that this is specific to shapeless' @@ type (where the problem here is the newts / newtype approach used by cats in, for example, NonEmptyChain)?

[Georgi Krastev]

Yes, but they are related (it's a similar encoding). So I hoped that the fix to dealias less would help in this case as well. But no luck...

[Ashley Mercer]

Okay I'm building a test case and I've found something interesting: it seems to be caused by the specific way in which the packages are laid out in cats. A minimal reproduction of this layout is as follows:

trait TC[F[_]]

object a {

  // 1. the Impl class is package-private here
  private[a] object FooImpl {
  
    // Zero-overhead newtype
    private[a] type Base
    private[a] trait Tag extends Any
    type Type[+A] <: Base with Tag

    implicit def mkTC: TC[Foo] = new TC[Foo] {}
  }

  // 2. we then re-export the "nice" type alias
  type Foo[+A] = FooImpl.Type[A]
}

// This works
implicitly[TC[a.Foo]]

// This doesn't
implicitly[Lazy[TC[a.Foo]]]

The Lazy implicit does work however if you remove the private[a] qualifier from FooImpl i.e. make it public again. It's as if, when the alias points to a type which is private (not visible from the call site) then the regular compiler can see through but Lazy cannot.

[Ashley Mercer]

And having discovered that I've just managed to solve my (original) problem: simply explicitly re-importing the implicits I need allows Lazy to see them again. From my original example:

{
  implicit def mkTypeClassLazy[F[_]](implicit F: Lazy[cats.Functor[F]]): TC[F] = ???
  implicitly[TC[ValidatedString]]        // works

  import cats.data.NonEmptyChain._
  implicitly[TC[ValidatedNecString]]     // now works again
}

[Miles Sabin]

It's as if, when the alias points to a type which is private (not visible from the call site) then the regular compiler can see through but Lazy cannot.

Thanks for the analysis ... very helpful.

If you're building on 2.13, would you mind trying to replace Lazy with a by-name implicit argument and see how that behaves?

[Ashley Mercer]

Compiling with scala 2.13.1, a by-name implicit works as expected. Assuming the definition of object a above:

def implicitByName[A[_]](implicit tc: => TC[A]) = println { tc }
def implicitByLazy[A[_]](implicit tc: Lazy[TC[A]]) = println { tc }

// This works as expected
implicitByName[a.Foo]

// This doesn't work without explicit re-import
import a.Foo._
implicitByLazy[a.Foo]

[Georgi Krastev]

Ok I think that's another instance of scala/bug#6794 - the implicit is accessible when inferred by the compiler but not when written down by the user (and having a macro generate it is morally equivalent to it being written down by the user).

So I don't think anything could be done in shapeless to fix it.

[Miles Sabin]

It's possible that a bit less dealiasing in the Lazy macro might help, but even if it did it would be very fragile.

I think it's a mistake to rely on constructions which depend on aliases having different properties from their alisees (IOW, I think this is a problem with the newtype implementation) ... we should have referential transparency at the type level as well as the term level.

I'll leave this open for a bit, but I'm inclined to close it.

[Ashley Mercer]

Thanks all for your input on this. Agreed, it seems like it's not something shapeless can reasonably fix, so I'm happy for you to close. The workaround (for now at least) is to explicitly re-import the necessary implicits.

It also seems like there are other issues cats with newtypes, so I will comment there and link back to this issue.