Some tests are dependant on Python's hashing randomization

Question

Some tests are dependant on Python's hashing randomization

multani opened this issue 10 years ago · comments

The problem basically is:

gourami:~/.../nevow$ trial nevow.test.test_flatsax.Basic.test_otherns                
nevow.test.test_flatsax
  Basic
    test_otherns ...                                                       [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 0.017s

PASSED (successes=1)


gourami:~/.../nevow$ PYTHONHASHSEED=42 trial nevow.test.test_flatsax.Basic.test_otherns
nevow.test.test_flatsax
  Basic
    test_otherns ...                                                     [FAIL]

===============================================================================
[FAIL]
Traceback (most recent call last):
  File ".../nevow/nevow/test/test_flatsax.py", line 184, in test_otherns
    self.failUnlessEqual(xml, flatten(parseString(xml)))
  File "/usr/lib/python2.7/dist-packages/twisted/trial/_synctest.py", line 356, in assertEqual
    % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = '<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"><p>in default namespace</p><xf:input><xf:label>in another namespace</xf:label></xf:input></html>'
b = '<html xmlns:xf="http://www.w3.org/2002/xforms" xmlns="http://www.w3.org/1999/xhtml"><p>in default namespace</p><xf:input><xf:label>in another namespace</xf:label></xf:input></html>'


nevow.test.test_flatsax.Basic.test_otherns
-------------------------------------------------------------------------------
Ran 1 tests in 0.017s

FAILED (failures=1)

This means that some tests will break some because they depend on some ordering which is not stable . This currently blocks #47 since Tox sets a different hash seed for each run and the probability of getting the "right" seed is probably very low.

I would propose to hunt down the places where such ordering is not guaranteed and use sorted() or something appropriate to fix the problem. (The fix for this test case is too wrap this iteritems() into sorted()). This will make the output more predictable obviously, ease the tests as the tester can expect to have a reliable output, but it has a slight cost performance though.

Other possibilities that I can think of:

change the tests so they are smarter: instead of doing a basic string equality (or a assert expected in result as I saw in other failing tests where expected content is not in the same order as the one in result), parse the result (they are XML-like strings each times) and assert that the structure is properly formed.
find the right hash :) (and set it somewhere once and for all, like this for Tox.