Falky tests
nbraud opened this issue · comments
Found some more flakyness by running Hypothesis for 6h (max_examples
set to half-a-million):
=========================================== FAILURES ============================================
_____________________________________ test_trig_invariance ______________________________________
@given(angle=angles(), n=st.integers(min_value=0, max_value=1e5))
> def test_trig_invariance(angle: float, n: int):
tests/test_vector2_rotate.py:111:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../.venv/lib/python3.7/site-packages/hypothesis/core.py:593: in execute
% (test.__name__, text_repr[0])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <hypothesis.core.StateForActualGivenExecution object at 0x7f52d5e7ffd0>
message = 'Hypothesis test_trig_invariance(angle=82.80876812840495, n=44343) produces unreliable results: Falsified on the first call but did not on a subsequent one'
def __flaky(self, message):
if len(self.falsifying_examples) <= 1:
> raise Flaky(message)
E hypothesis.errors.Flaky: Hypothesis test_trig_invariance(angle=82.80876812840495, n=44343) produces unreliable results: Falsified on the first call but did not on a subsequent one
../.venv/lib/python3.7/site-packages/hypothesis/core.py:761: Flaky
------------------------------------------ Hypothesis -------------------------------------------
Falsifying example: test_trig_invariance(angle=82.80876812840495, n=44343)
δcos: -2.0268564604464245e-11
rel_max = 0.12518140611629808
diff = 2.0268564604464245e-11 = 1.619135399839974e-10 * rel_max
δsin: 2.557398737224048e-12
rel_max = 0.9921338697816041
diff = 2.557398737224048e-12 = 2.5776750649456224e-12 * rel_max
Unreliable test timings! On an initial run, this test took 643.91ms, which exceeded the deadline of 200.00ms, but on a subsequent run it took 0.08 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
______________________________________ test_scalar_linear _______________________________________
@given(scalar=floats(), x=vectors(), y=vectors())
> def test_scalar_linear(scalar: float, x: Vector2, y: Vector2):
tests/test_vector2_scalar_multiplication.py:24:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../.venv/lib/python3.7/site-packages/hypothesis/core.py:593: in execute
% (test.__name__, text_repr[0])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <hypothesis.core.StateForActualGivenExecution object at 0x7f52cee241d0>
message = 'Hypothesis test_scalar_linear(scalar=4.539659888108493e+74, x=Vector2(4.539659888108493e+74, 0.0), y=Vector2(3.272143794426157e+74, -0.0)) produces unreliable results: Falsified on the first call but did not on a subsequent one'
def __flaky(self, message):
if len(self.falsifying_examples) <= 1:
> raise Flaky(message)
E hypothesis.errors.Flaky: Hypothesis test_scalar_linear(scalar=4.539659888108493e+74, x=Vector2(4.539659888108493e+74, 0.0), y=Vector2(3.272143794426157e+74, -0.0)) produces unreliable results: Falsified on the first call but did not on a subsequent one
../.venv/lib/python3.7/site-packages/hypothesis/core.py:761: Flaky
------------------------------------------ Hypothesis -------------------------------------------
Falsifying example: test_scalar_linear(scalar=4.539659888108493e+74, x=Vector2(4.539659888108493e+74, 0.0), y=Vector2(3.272143794426157e+74, -0.0))
Unreliable test timings! On an initial run, this test took 784.57ms, which exceeded the deadline of 200.00ms, but on a subsequent run it took 0.04 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
=========================== 2 failed, 265 passed in 21859.03 seconds ============================
python3 -m pytest --hypothesis-profile ridiculous -v tests/test_vector2_* 21834.05s user 8.80s system 99% cpu 6:04:19.95 total
Need to investigate tomorrow.
I cannot reproduce the failure, even by giving the exact same inputs to Hypothesis (with @example
).
OK for closing the issue?