Development workflow re: recording and replaying
jennybc opened this issue · comments
Creating a home for a discussion started on a gh PR (r-lib/gh#51 (comment)).
I believe the main conclusion was to implement control via an environment variable:
Agreed. How about controlling the behavior via environment variables? Then we could have a default behavior, for the user, and developers could change it via a simple command that would just get/set an environment variable?
From my early bit of usage, I think it's not desirable to automatically start recording and replaying upon every instance of devtools::load_all(). I think I expect default state during development to be neither recording nor replaying. And I would expect to do something very deliberate to toggle the recording/replaying state, beyond a reflexive load_all().
Related: should there be a message whenever something gets replayed?
From my early bit of usage, I think it's not desirable to automatically start recording and replaying upon every instance of devtools::load_all(). I think I expect default state during development to be neither recording nor replaying. And I would expect to do something very deliberate to toggle the recording/replaying state, beyond a reflexive load_all().
Yes, the default is stop_recording(), stop_replaying(). The helper.R in gh changes this, and you are right, I'll change it there.
The toggle is currently start_recording() and/or start_replaying(), but as we discussed I am willing to add another layer, that makes it easier to choose.
Related: should there be a message whenever something gets replayed?
There kinda is. It uses the debugme package, so if you set DEBUGME=httrmock, then you'll see some debug messages, although maybe too many.
Btw. for DEBUGME, you need to reload the package, because it switches on debugging at load time.
OK, I removed the start of recording and replaying in helper.R: r-lib/gh@b6cac6d
Actually, for a dev default setting, maybe stop_recording(), start_replaying() is better, no? Just because I don't want to perform all already recorded requests when I am just running devtools::test().
so if you set DEBUGME=httrmock, then you'll see some debug messages, although maybe too many
This is quite nice.
OK, I am thinking about the workflow again, and basically my rather generic conclusion is that we need to be a lot more flexible. Here are some random points and ideas:
- we need to be able to specify that mocking should only be performed for a certain domain, or certain types of requests, etc.
- we need to be able to have exceptions, i.e. requests that are not recorded.
- we can have a
with_httrmockfunction, and mocking only happens within this. (It would just set an environment variable, which thehttrhooks support.) - we should be able to turn recording/replaying on/off just in a single test file, because the test file we are working on should be treated differently.
- maybe we could have a package level config file for
httrmock, in/tests, that specifies what should be mocked. - we could have a mode that errors on unknown requests. These are the ones that should have been recorded, but they were not.
- we should be able to specify which headers are matched and recorded.
I realize that this is not very coherent, and I'll make another 1-2 hours effort tonight, to come up with a more specific plan. As always, feedback is welcome!
Actually, for a dev default setting, maybe stop_recording(), start_replaying() is better, no?
Yes, agree on this. Locally, I have done this.
Configuring the tests, i.e. what to mock, what not to mock, and how exactly, seems relatively easy to implement. As I see the challenge is really to create a nice workflow for writing / updating tests.
I have an idea for this. I'll implement an interactive httrmock() function that can be used to select the current behavior. This is for the developer that is writing the tests. This will include a choice of setting a custom behavior for individual test files. I.e. for the current test file.
E.g. we can have the default behavior for all test files (which should be probably start_replaying(), stop_recording() and fail_for_unknown(), or sg like this).
But for the single test file you are currently writing you could choose a custom behavior:
- Initially you would probably want the requests to go through.
- Then, once the tests are done, you would want to record them, once.
- Finally you would switch off the special behavior of the current test file.
Does this make sense?
And we also need better logging via debugme, i.e. more specific messages. Anyway, there is a lot to improve, but personally I like the concept of the current test file.
That all sounds very good. I also know you are a ninja with the R prompt so I will tempt you with the notion of indicating current recording/replaying status there 😜.
HAha! No problem, it will be in the prompt, and we can use cwt() as an analogy to cd, to navigate it.
This is done.