Running all tests without requiring bazel? (test discovery)

[Andrea Tagliasacchi]

The introduction here creates an analogy between abseil-py and python native "unittest". From the readme it seems that to run tests bazel is required.

Does abseil provide a "test discovery" alike unittest? (see docs)

# When executed without arguments Test Discovery is started:
python -m unittest

[Yilei Yang]

abseil doesn't have special logic for test discovery. Since absl.testing.absltest is built on top of unittest, if you just use absl.testing.absltest.TestCase over unittest.TestCase in your code, python -m unittest would also work.

But if your test or your test dependencies also define flags with absl.flags, python -m unittest will likely fail because absltest.main() is not called (it parses flags and does some extra things).

We usually define a test module with the following main block:

from absl.testing import absltest
if __name__ == '__main__':
  absltest.main()  # Parses flags and does some extra things

and execute the modules separately as main programs, and it doesn't require bazel.

Does this answer your question?

[Andrea Tagliasacchi]

I think so, as far as I add the main block above, then I can recycle the unittest logic.
Let me test it (aha!) and I will close it if it works well.

@julienvalentin for your reference

This works

(python3) ~/dev/graphics/tensorflow_graphics/nn/metric: python -m unittest discover --pattern *_test.py --failfast -v
test_evaluate_preset((0, 1, 1, 1, 1), (1, 1, 0, 0, 0), 0.3333333333333333) (tests.fscore_test.FscoreTest)
[...]
----------------------------------------------------------------------
Ran 50 tests in 1.273s
OK (skipped=4)

And this one works too, as pytest is unittest compatible:

===================================================================================================== test session starts =====================================================================================================
platform linux -- Python 3.6.9, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/ataiya/dev/graphics
collected 50 items                                                                                                                                                                                                            

tests/fscore_test.py ........s                                                                                                                                                                                          [ 18%]
tests/intersection_over_union_test.py ............s                                                                                                                                                                     [ 44%]
tests/precision_test.py .............s                                                                                                                                                                                  [ 72%]
tests/recall_test.py .............s                                                                                                                                                                                     [100%]

====================================================================================================== warnings summary =======================================================================================================
/home/ataiya/Envs/python3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15
  /home/ataiya/Envs/python3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================================================================================== 46 passed, 4 skipped, 1 warning in 3.70s ===========================================================================================

[Andrea Tagliasacchi]

Worked like a charm:
tensorflow/graphics#296

Closing the issue.

[June Oh]

Re #138 (comment): as @yilei said --

But if your test or your test dependencies also define flags with absl.flags, python -m unittest will likely fail because absltest.main() is not called (it parses flags and does some extra things).

While python -m unittest and pytest may seem to work, they may have unexpected side affects as the absltest.main() is not called to parse the flags etc.

Instead, a better workaround would be to trigger the test discovery within the unittest.TestProgram that absltest.main() uses, simply by:

absltest.main(module=None)

And then call the above line e.g. via entry point script, or a __main__.py within the test directory.