Demos and tests sometimes run with inconsistent results on travis

Question

Demos and tests sometimes run with inconsistent results on travis

mbonneau opened this issue 7 years ago · comments

As part of the tests on travis, RxPHP runs most of the demos to make sure they are still outputting the same expected results.

This is causing some flakiness in the tests as the demos use a real time clock to run and, given the asynchronous nature of Rx, sometimes the test environments get different results. (These are often cleared up by just rerunning the tests).

I opened this issue as a place to discuss possibilities for resolving this issue.

Example: #98

Alexander · Answer 1 · Sat Dec 24 2016 23:52:22 GMT+0800 (China Standard Time)

Do we know what other demos are affected? I'd just remove the .expect files for non deterministic demos if it's not easy to fix.

Matt Bonneau · Answer 2 · Sun Dec 25 2016 07:48:37 GMT+0800 (China Standard Time)

@asm89 - I don't have a record because travis does not keep the log when you rerun from what I can see. I will keep track so that we have some examples to look at before removing expect files.

Matt Bonneau · Answer 3 · Tue Jan 10 2017 00:39:32 GMT+0800 (China Standard Time)

I am going to close this issue as it does seem to be a case-by-case basis and can be corrected either by adjusting timing of the tests until they are deterministic or if it is truly non-deterministic, dropping the expect as suggested.