abseil / abseil-py

It'd be really sweet to have a glob pattern that specifies what tests to run, similar to googletest's --gtest_filter.

You can specify filters as positional arguments, or if bazel is used, via the --test_filter flag.

However, the filter only supports specific test class / names (e.g. MyTestCase, or MyTestCase.testA MyTestCase.testB), as it just delegate to what Python's unittest framework supports.

I'm keeping this open as I do see interests in support a glob pattern for filtering.

As far as I can tell, there is no way to select parameterized tests at all.

unittest's -k flag (3.7+ https://docs.python.org/3/library/unittest.html#cmdoption-unittest-k) should already work, e.g.

python -m unittest -k 'test_name' will run any tests that contains the case insensitive substring test_name and it should work for parameterized tests too.

FWIW, I'm seeing this confusing behavior:
This works:

bazel test --test_arg=-k=TestLexer.test_lex_iter

This does not (it silently ignores the filter and runs all the tests, it would be improved if it at least barfed):

bazel test --test_filter=TestLexer.test_lex_iter

unittest's -k flag (3.7+ https://docs.python.org/3/library/unittest.html#cmdoption-unittest-k) should already work, e.g.

python -m unittest -k 'test_name' will run any tests that contains the case insensitive substring test_name and it should work for parameterized tests too.

Hi，it doesn't work when i try to pass the '-k' to the unit test of tensorflow.keras, the excepition is:
absl.flags._exceptions.UnrecognizedFlagError: Unknown command line flag 'k'
I have used python3.7.10 and python3.8 according to https://docs.python.org/3.8/library/unittest.html#cmdoption-unittest-k
It seems that abseil-test needs to support the parameter 'k'.

How are you running the test and which test? The flag -k is provided by unittest and shouldn't be passed to absl's own flags parsing.

re #128 (comment):

This is likely that you are using unittest.main(), thus bazel test --test_arg=-k=TestLexer.test_lex_iter works.

bazel test --test_filter=TestLexer.test_lex_iter only works when using absltest.main(), since it requires special handling of TESTBRIDGE_TEST_ONLY environment variable passed by bazel when using --test_filter= (https://docs.bazel.build/versions/2.2.0/test-encyclopedia.html)

I cannot use test_filter with Keras as @LeoLau94

Could you please provide more details? What are you running and how to you expected it to work?

(This is still open as a feature request to support glob patterns).

I am running a command like this bazelbuild/bazel#13664 (comment)

This is unique to how tensorflow calls absltest.main here: https://github.com/tensorflow/tensorflow/blob/b743f04b1f1b0600ac5bb1c0b8bfc7e33896255a/tensorflow/python/platform/googletest.py#L62-L64

It always explicitly specifies a non-None argv= parameter.

The test filtering is supposed to be specified in this argv variable here in absltest:

abseil-py/absl/testing/absltest.py

Line 2539 in cde7550

kwargs.setdefault('argv', argv)

However, because tensorflow sets an argv in the kwargs dict, it's not passed on.

I currently think it makes sense for absltest to merge the updated argv with the user specified one from kwargs, and it would make tf.test.main() work. I'll try to make the change and see if any users are relying on the current behavior.

Ah, since kwargs['argv'] always overrides argv in _run_and_get_tests_result, we should add filtering flags to kwargs['argv'] if present.

@yilei Thanks, so is this a new ticket? Or do we want to keep this for the glob patterns?

/cc @haifeng-jin

Could you please file a new one? Thanks!

I want to ask you a clarification, what is now the use of --test_filter vs --test_arg without glob support? Is it exactly the same?

--test_filter is supposed to only perform test filtering (without glob support).

--test_arg is used to pass them as command line arguments to your test binary. When you specify e.g. MyTestCase.test_method, then this is non-flag arguments that are eventually passed to unitest.main(argv=), which has the same effect as test filtering. But you can also use it to specify other flags (e.g. --my_flag=value) if you test binary supports them. When using absltest.main(), it means flags defined by absl.flags.

Hope this helps.

Thanks, so we don't have so many solutions for a parametrized tests pattern other than waiting for glob.

What changes need to be implemented to support Glob patterns here?

As we really want to give a solution to contributors to filter a single parametrized test when they are preparing a PR.

You can already leverage Python unittest (3.7+)'s -k flag to do pattern matching today: https://docs.python.org/3/library/unittest.html#cmdoption-unittest-k

bazel test --test_arg=-k=<pattern or substring>

I'll try to make a change so test_filter matches -k's behavior for Python 3.7+ internally and see what happens (this is a behavior change since it now also does substring matching so it can run more tests than before)

But still any workaround for google test right?

But still any workaround for google test right?

I'm not exactly sure what this question means, but I'll try to expand:

--test_filter= is effectively a shortcut to --test_arg= with custom interpretations determined by the test framework that the underlying test uses.
--test_arg=-k=<pattern or substring> should work for tests using either absltest.main() or unittest.main(), since that's simply passing the argument to sys.argv and both of them should work.
--test_filter=<pattern or substring> only works if the test framework interprets the TESTBRIDGE_TEST_ONLY env var set by bazel (https://docs.bazel.build/versions/main/test-encyclopedia.html#initial-conditions), and absltest does that, but not if you are using plain unittest.

I meant something like Case 2, -k It is interpreted as an absl arg:

FATAL Flags parsing error: Unknown command line flag 'k' Pass --helpshort or --helpfull to see help on flags.

Oh you are right. Sorry about that.

The flag needs to be specified as -- -k=<pattern or substring> for it to work with absltest:

bazel test --test_arg=-- --test_arg=-k=<pattern or substring>

args after -- are treated as positional arguments to absl.flags and passed to unittest

Ok thanks, in the meantime we will suggest this workaround then when ready, --test_filter will be more linear.

This is now released. For Python 3.7+, the filter now has the same semantic meaning of Python unittest's -k flag (https://docs.python.org/3/library/unittest.html#command-line-options)

absltest feature request: run specified tests