Testing support
cdegroot opened this issue · comments
I have a question about ExActor and how to support testing.
Here's a bit of an intro: after some toy services and problems ("toy" as in "relatively simple" - we do run some of this in production) I'm cutting my teeth on some things that are more complex, especially in how multiple modules/processes/... interact, to help guide my thinking of how to transfer all my knowledge and gut feelings from mostly the OO world (I'm a Smalltalker at heart, still 😉) to the Erlang model. To make a long story short, I have the feeling that with some minimal adjustments to a library like ExActor we could have a nice way of testing inter-process (well, inter-genserver) interactions without breaking encapsulation, etcetera. I have been jotting down my thoughts https://github.com/cdegroot/simpler/blob/master/notes.org mostly to help me structure my own thinking. To make a long story short, if we could change one thing in ExActor and add one thing, then it'd not only be a cool library to write shorthand GenServers, but also to actively support nicer testing approaches than what I'm seeing in the wild. Specifically, i'd like the message format to be exposed (through helper functions) so you could pattern match on it in tests. Quoting myself from my notes, I'd like to write:
test "foo" do
# ... do some testing ...
expect_called Person.print(self())
end
where expect_called would expand in an assert_received with the argument being converted in the message being created by print's defcall or defcast (where you probably want to have an expect_call for a cast). This way, it's possible to write down intent and not break encapsulation by keeping the message format that ExActor uses under wraps. For more precise matching, but I'm not sure whether that's needed or not, it'd be neat if ExActor sends the module along in its internal message format so some more convoluted test could say something like
expect_called Person.print
expect_called SomethingElse.print
Does that all make sense? Would you be willing to accept a PR that would add this to ExActor? (If I can figure out the code - macros in Elixir not yet my strongest point ;-)).
(note: I'm am still thinking here. Ideally I'd at the very least like to have ExActor spit out (matchers for) the messages it generates, but there's no need for immediate action until there's some example code by me that shows how it'd be used ;-))
Ok, I cobbled up an example: https://github.com/cdegroot/simpler/blob/master/test/expect_call_test.exs. Now, the small bit I'd like to see in ExActor, even though it's maybe nitpicking, is support for the code in the test where the expectation is formed. Currently it reads:
expect_call {:some_call, "you", "me"}, reply: :ok
which means that now my test has knowledge about the internal messages that flow "between" the interface method and the server implementation for some_call. I'd love to be able to write this as
expect_call Dependency.some_call(mock_dependency, "you", "me"), reply: ok
which is doable of course, except for the bit that ExActor should have a hook to convert the function invocation of the latter form into the actual message being sent in the former form - that way, ExActor encapsulates its message format and that would make me happy.
Am I going overboard here? ;-)
Thanks for the detailed writeup! I need to think about this a bit, before discussing further. I'm a bit busy ATM with some personal obligations & work, but I'll try to get back to you by the end of the week.
As I said, no hurry. And it might be we conclude that the documented message format is enough. I'll keep chipping away at my code samples in the meantime ;-)
I read through your proposal a bit more carefully, and I'm still not convinced that this is the right approach to solve the problem (at least the way I understand it).
It seems that the core challenge here boils down to testing that an operation in module under test does something with the dependency, without actually invoking the dependency.
In your example, you have this code:
defcall do_something_with_dependency, state: state do
Dependency.some_call(state, "you", "me")
reply(:ok)
endAnd you want to test that Dependency.some_call is invoked with proper arguments (without actually invoking it).
I think that instead of passing a test (mock) GenServer, it's better to pass a test module. I believe this is better because it doesn't assume anything internal about Dependency.some_call/3. In contrast, in your example tests, you assume that it's a GenServer implemented with ExActor, and that it has some particular internal message format. That's a lot of assumption about something we're not even testing here :-)
Here's a sketch of how I do it occasionally:
defmodule ModuleUnderTest do
use GenServer
def start_link(options \\ []), do:
GenServer.start_link(__MODULE__, Keyword.merge(default_options(), options))
def init(options), do:
{:ok, options}
def handle_call(do_something_with_dependency, _from, options) do
dependency_mod(options).some_call(...)
{:reply, :ok, options}
end
defp dependency_mod(options), do:
Keyword.fetch!(options, :dependency_mod)
defp default_options(), do:
[dependency_mod: Dependency]
endSo now, by default the dependency is Dependency, while in test you can provide your own explicit test mock (which I usually define under the test module).
This will require more typing, but I believe it's a better approach than mocking handle_* callbacks of a GenServer.
You could also think about creating some generic helpers to simplify creation of test modules. Here's an example inspired by your own test code:
test "some test" do
test_dependency_mod = Mock.create_mock_module()
{:ok, pid} = ModuleUnderTest.start_link(dependency_mod: test_dependency_mod)
Mock.expect(test_dependency_mod, :some_call, fn(x, y, z) -> ... end)
ModuleUnderTest.do_something_with_dependency(...)
Mock.verify(test_dependency_mod)
...
endThe core idea here is that Mock.create_mock_module/1 dynamically generates a module with a unique name (which is admittedly hacky and controversial). The unique name still keeps this mock private (in theory it's global but no one else will use it), so tests can be run with async: true. The downside is that the solution is admittedly hacky, and might not scale beyond some point (too many tests defining mock modules). Perhaps there are cleaner ways to do it, but I didn't give it a lot of thought :-)
Regardless of whether you have a generic helper, or just manually define your test modules (which is what we're currently doing at my company), I believe that such approach is cleaner, and it solves a wider class of problems than mocking handle_* functions of a GenServer.
That being said, I'm still open for discussion, so if you have some counter arguments, or other cases which I missed, let me know :-)
I actually have these helpers already and wrote a non-trivial bit of code with it :-). Coming from OO-land, my first knee-jerk reaction is that an "object" is state+behaviour, where state is the pid and behaviour is the module, so I wrote some code and patterns around it to encapsulate dependencies in {module, pid} tuples. I even went a bit overboard by having "interfaces" consisting of lists of defmethod macro invocations; the defmethod generates a @callback and a forwarding call, so that uses of the interface can call Interface.method(dependency, ...) and be oblivious of the fact that the pseudo-pid argument is in fact a module+pid combo that the generated forwarding call unpacks. An example says more than a thousand words, so https://github.com/cdegroot/amnesix/blob/master/lib/amnesix/partition_worker.ex for a sample interface, https://github.com/cdegroot/amnesix/blob/6580dd78f02ee355dfc33b00801f31cdcc9b3994/lib/amnesix/workers_supervisor.ex#L79 for a sample invocation, and https://github.com/cdegroot/simpler/blob/master/lib/interface.ex. I'm decidedly not happy with the code for two reasons - the invocation code is ugly (but that can be fixed, I didn't want to go off into tweaking mode until I had written some decent amount of code with it and could step back and take a look), and, more importantly, it feels very much like an OO dude trying to impose his preconceived notions onto a non-OO language. I've learned to be wary of that :-)
This is basically the point where I started the Org doc and writing down some ideas and reasoning from scratch, which leads me to conclude that the proper level of defining an interface is, for Erlang/Elixir, at the message passing level. Which is heresy as well because Elixir inherited the Erlang pattern of having a method-level API which hides the message passing ;-). So now I'm stuck, but then I realize that it is all ok if you keep the message passing hidden from view!
https://github.com/cdegroot/simpler/blob/b09a524a1b38add89006830248db597e35516b83/test/expect_call_test.exs#L49 does exactly that. It sweeps the fact that we're intercepting at the API level under the rug. It does require some cooperation from the dependency code, but I think that that's a fairly minor issue - in most of the cases, your GenServer interface method will do an immediate message send, for which this code works fine; for the exceptions you can re-work the code a bit to make it work this way, I think.
Anyway, the discussion indeed hinges on the view of what the proper interception point is. Injecting module+pid is indeed more flexible, but feels maybe "too OO"; injecting at the message level feels more correct but does indeed risk breaking encapsulation. In any case, thanks for your extensive reply - I'm a troubled person traveling through Elixir-land and need all the help I can get to come to a decision ;-).
Did I prematurely decide that {module,pid} wasn't the right answer because of my less-than ideal implementation and should I revisit my macros?
(I find dynamically generating a module in a test not really controversial - it is way better than the alternative that e.g. José proposes, which is globally swapping out implementations; however, it does show that intercepting at the message level, where you basically can dynamically decide what to do with it, is maybe the cleaner solution as it matches well with what happens in (late-binding/dynamic) OO-land, where dynamic message interception is a much-used tool, especially when defining mocks (Smalltalk's doesNotUnderstand invocation and similar stuff in Ruby and Python))
Coming from OO-land, my first knee-jerk reaction is that an "object" is state+behaviour, where state is the pid and behaviour is the module, so I wrote some code and patterns around it to encapsulate dependencies in {module, pid} tuples.
As a long time OO programmer myself, I think I roughly understand what you're trying to do here. I also had a similar line of thoughts in my initial Erlang/Elixir months, but in the end I found out that it doesn't really work well.
While processes, especially GenServers, do resemble objects (since they encapsulate state, and messages can be used to affect/query that state), you should not use them in the same way you use objects. The thing is that classes and objects are used to organize your code - i.e. to split different concerns across separate units of code, so its easier to analyze and extend the code. In Elixir/Erlang world for that purpose modules and functions are your primary tools to organize your code, together with Elixir additions of protocols and macros. Processes are in contrast used to organize your runtime. You run things in separate processes to improve fault-tolerance and scalability of your systems.
Therefore, many OO patterns, such as dependency injection, shouldn't be used in the same fashion as you'd use them in your OO programs. In particular, one GenServer usually shouldn't store and keep a pid to another GenServer (there are exceptions of course). The reason is that this other GenServer might crash and be restarted, and at this point you end up having a pid of a non-existing process. I covered this in more details in this talk. Some things in the talk are now obsolete. The gproc library I discuss doesn't need to be used, since now you have Registry module out of the box in Elixir 1.4+. But the ideas are still relevant - in most cases you want to have late discovery of the target process.
Anyway, the discussion indeed hinges on the view of what the proper interception point is.
That's precisely it! It is my opinion that the "interception" (i.e. varying) point should be function invocation. This also support the view I stated above - that modules and functions are tools for organizing your code. And, as I said earlier, this approach covers a wider area of cases, plus it doesn't assume anything about the implementation of the dependency. It's also simpler - instead of passing messages across process boundaries, you're just verifying that your dependency is invoked.
In other words, a dependency is a function, or a module with well-defined contract. To support varying dependency, your interface (some function) accepts a lambda, or a module, through function parameter.
Did I prematurely decide that {module,pid} wasn't the right answer because of my less-than ideal implementation and should I revisit my macros?
As someone who made similar mistakes in the past, I'm pretty certain that any solution that tries to simulate OO with processes is not a good solution. The same holds for extensive use of custom macros. My advice would be to go with plain GenServers and avoid any macro hackery to simulate OO.
I find dynamically generating a module in a test not really controversial
The controversial part is that this is an improvisational abuse of the platform. It's worth spending some time to search for a cleaner solution. I'm not dismissing that approach, after all I proposed it myself :-) But that proposal came after about 10sec of thinking, so it's not really thoroughly thought out :-)
it is way better than the alternative that e.g. José proposes, which is globally swapping out implementations
I don't think this is what José proposes. IMO, the central idea of his article is to explicitly distill the contract to dependency, rather than mocking low-level functions, such as HTTP client modules to simulate Twitter API. How exactly are you going to vary the implementation in tests depends on the particular case. In some cases global setting (e.g. through application configuration) might be the simplest approach, while in others you'll need to pass the dependency (module or function) as an argument.
Thanks for that. I'm aware of the "GenServer-is-not-an-object" thing, but when one needs another to do shit and you want to swap out implementations, one starts pulling from experience :). I do like the code-structure vs runtime-structure dichotomy - I'm gonna steal that and use it at work, thanks for that!
Anyway, I have tons to think about - I'm going to close this issue, as a design debate is incredibly interesting and very useful to me at least, but probably not the best use of your time. I'll go back to the drawing board and look at my code some more and see how I can work with ExActor to make testing super nice. Two closing remarks:
- You're right about what Valim wrote. And I was wrong. I must have mixed up two articles by different authors - in Mocks and explicit contracts he indeed says that pasting your test doubles into
config/test.exsis not a good approach. I thought he did - sorry for the confusion. For me,async: trueis sacrosanct - not only because it makes your test suite run faster, but also because it forces you to take care of all sorts of code smells - it's harder, it's more work, but in the long run, it pays off; - Extensive use of custom macros - bad indeed. I'm hoping to find and propose macros and a coding style/design patterns that will have wider usage. One of the drawbacks of Elixir code these days is that it's very verbose, with lots of duplication - methods in your GenServers get mentioned twice, three times if you also add callback signatures when you do want to do some type checking on stubs. I think a language should strive to make its basic bread-and-butter constructs as simple to use as possible with a very large signal/noise ratio, and Erlang and Elixir fall short here - in my opinion. Luckily it's an extensible language (one of the main reasons I looked into Elixir two years ago and not into Golang) so this problem is solvable for those who want it solved. If everybody tells me to sod off and crawl back under my stone and type out normal GenServers, I'll be happy to oblige but not before I tried it :-)
One of the drawbacks of Elixir code these days is that it's very verbose, with lots of duplication - methods in your GenServers get mentioned twice, three times if you also add callback signatures when you do want to do some type checking on stubs.
I had the similar train of thoughts at some point, and that's why I wrote ExActor (after trying some other approaches in plain Erlang as well). And you know what? These days I'm not using it anymore :-) It took me some time to realize that GenServer is not really burdened by duplication. The reason for this "duplication" is that your typical GenServer module bundles the server side with the client side. If you write an HTTP server and a client, you'd also need to implement the message format knowledge in two places.
While ExActor does reduce boilerplate in some cases, it becomes more cumbersome in cases where the interface function (i.e. client code) needs to do something else than just forwarding a message. Moreover, if you want to do pattern matching (client- or server- side), things also become tricky. So in such cases (which are not uncommon), ExActor powered code actually becomes more complex than plain GenServer.
So ultimately, I concluded that GenServer doesn't introduce needless boilerplate. It's more that there are some things that look like duplication, but are really client- and server- side views of the messaging protocol.
Of course, if you have some ideas you want to explore, definitely do that by all means. I'm just trying to warn against some misconceptions that I personally had.
You might also take a look at Jeeves by Dave Thomas. It's an exploration in its early stage, but it takes a bit different approach. The idea is that you write plain modules, and then you tag them as services by use-ing some module. Under the hood, this will turn them into GenServers.
I didn't look into the code, but I believe this idea is very interesting. Unlike ExActor, which aims to be just a plain boilerplate removal helper, Jeeves tries to distill the core concept of GenServer. And I do believe, that GenServer is all about (micro)services. So that's certainly one interesting venue to explore.
Thanks for the warning and the pointer to Jeeves!
Thank you for the interesting discussion! Also, in case you're not following, this topic at Elixir forum might have some useful insights. I also highly recommend the forum as a great place for discussing various aspects of the language and the ecosystem.