Future of assert_cli
killercup opened this issue · comments
@epage asked in #40 (comment):
I'm curious, what do you see needing done before moving this to 1.0? While I should get back to my regularly schedule project, I would like to also help work towards more crates hitting 1.0.
Good question! I'd like to make the arg parsing a bit leaner (cf. #25 and #32) and maybe improve the error output (#17, #18, #30).
My general goal is to make a library that makes writing tests like these easy and fun. It'll probably involve generating some test boilerplate code, and I'm not entirely sure what I want that API to look like. I'm not even sure if that's in scope for assert_cli. What do you think?
EDIT
The building blocks for assert_cli 1.0 are
tl;dr After considering this for the last couple days (and digging into the issues), I feel like PR #43 is sufficient for releasing 1.0.
I feel the main categories of 1.0 blockers
- API changes
- Usable.
RE Waltz's tests
While I see a lot of value in what you're doing with waltz, I feel like that has a longer path to stabilization than the core of assert_cli.
assert_cli does a great job of doing one thing very well, testing short lived programs. If we take the approach of trying to have assert_cli cover additional use cases, whether its the side effect testing done in waltz or the orchestrating multiple programs with duct's piping, I feel like it might detract from its primary strengths.
Also, thinking about it some more, side effect testing might have value outside of running programs. Waltz's test harness could be really nice for also wrapping functions.
Pinging some people who used assert_cli in their projects (found using Github search):
We are thinking about doing a 1.0 release and would love to hear your opinion on the crate and ideas for its future!
- @livioribeiro who used it for cargo-readme integration tests
- @bjgill who updated cargo-edit to assert_cli 0.4
- @mattgathu who quite recently used assert_cli in rget
- @jxson who uses it in some tests of to-directory
Hello @killercup @epage thank you for building such an amazing crate!! It has been really helpful.
I have been using it to test a rust cli app I'm writing using the assert_cli::Assert::main_binary interface. This is part of my contribution for Increasing Rust's Reach
I was surprised to find assert_cli::Assert::main_binary missing in your docs examples considering how helpful it is!! I had to dig a little deeper into your docs to get it.
I'm planning to:
- Write a Pull Request to enhance the docs and add example for assert_cli::Assert::main_binary and other useful interfaces.
- Write a How to test Rust CLI apps blog post, with introduction to writing tests with
assert_cli
Let me know what you think!
About future ideas, I have been wondering how feasible it is to test for colored output?
If my CLI binary prints out colored text, can I test this?
What are your thoughts on this.
I was surprised to find assert_cli::Assert::main_binary missing in your docs examples considering how helpful it is!! I had to dig a little deeper into your docs to get it.
Yeah, I suspect thats from it being developed later and that it makes it harder to run the examples as tests. I agree that more attention should be drawn to it.
Let me know what you think!
Sounds great! The main thing to watch out for is not reducing coverage.
About future ideas, I have been wondering how feasible it is to test for colored output? If my CLI binary prints out colored text, can I test this?
Most programs detect if they are outputting to a console and don't print color if they are. Some have a "force color" option.
So either we'd need to figure out how to fake being a console or clients need to have a force color option and set it.
When forced, I assume color just shows up in stdout as weird characters. Can you just manually enter those characters in to test it? Do you have a proposal for what an API might look like for more advanced color testing?
Once #46 goes in, should we also need to wait for Environment to hit 1.0?
Good question. I'm okay with publishing this with a re-export of the 0.1. I also think that Environment could be 1.0 already.
I would like to have a way to remove variable from Environment before 1.0.
something like:
let env: Environment = vec![("foo", "bar"), ("bar", "baz")].into();
...
env.remove("foo");I would also like to have a has/check method:
let env = Environment::inherit();
if env.has("foo") && env.check_if("bar", "baz") {
...
}Having a macro style would be a good way to use it:
let e = inherit_env!(
"foo" -> "bar", // <--- don't know if I can do that
"bar" -> "baz",
);
let e = empty_env!();
assert_cli::Assert::command(&["printenv"])
.with_env(inherit_env!( "bar" -> "baz" ))
.stdout().contain("bar=baz")
.unwrap();let e = inherit_env!(
"foo" -> "bar", // <--- don't know if I can do that
"bar" -> "baz",
);I think I generally see people use the => syntax when they experiment with these type of things, for example https://github.com/kmcallister/literator
@epage yep that's looking better.
Another question is: Macro style must expend to compile version? or not?
inherit_env! { "bar" => "baz" }
Becomes
Environment::inherit().insert("bar", "baz").compile()
or just
Environment::inherit().insert("bar", "baz")But it's a discussion on Environment repository I think^^
@killercup just recently came across https://github.com/emk/subtitles-rs. Looks like some of its building blocks are similar to the type of thing you are trying to do with waltz.
I've not explored it enough to know all of the implications of TestDir, if that concept is even compatible with my projects. I could see the idea of splitting the file system tests out into is own crate so the two can be composed together. I could see using the file system stuff in my lower level integration tests as well.
I did a usability study of assert_cli at RustBeltRust
- Discoverabiliy: Searching for "test command line" and "assert command line" did not return results on crates.io but "test cli" and "assert cli" did
- stdout testing: The API looks weird when run through rustfmt because it puts each function on a separate line, so it clouds up what the predicate is operating on
- Keep it, tools are meant to work for us
- Go back to having a function per variant
- Follow
Environmentand take in a predicate parameter- Easier to do multiple predicates (option to take in an array)
- While it helps in places, I found it annoying in clap to access a bunch of different types
- More linking is needed to help people find all of the relevant predicates
One thing in favor of accepting predicates as parameters is that someone wanted to pass multiple predicates at once which is more of an issue with contains and negated contains/is and not is.
But! At that point me might as well do
.stderr(|x: OutputAssertThingy| -> bool { x.is("bar") || (x.contains("baz") || !x.is("bazinga")) })to not reinvent boolean expressions. Which I think was something someone was asking for in another issue anyway (cf. #57).
I think there is still value in specialized predicates for better error reporting. We could just use From to auto-convert a Fn into one of the predicates.
We've started the major shift to assert_cli 1.0. assert_cmd is the spiritual successor to what assert_cli today. We plan to turn assert_cli into a higher level CLI testing crate like waltz tests.
The building blocks for assert_cli 1.0 are
- https://github.com/assert-rs/assert_fs/
- https://github.com/assert-rs/assert_cmd/
- https://github.com/assert-rs/predicates-rs/
- https://github.com/Freyskeyd/environment ?
The bad: it will probably take longer to stablize assert_cli because we need to re-evaluate the design and play with ideas
The good: the smaller pieces should be a lot easier to stablize.
Started looking at how others test CLIs and for python, I only found https://bitheap.org/cram/