TestError: Screenshot 0 was different.

[Nathan Birrell]

One particular error keeps getting returned when I record and run the Huxley tests, on particular pages only. This error has been found before and the solution was unclear (#27).

Here's an example of the error returned on such pages:

NathanMBA:test nathan$ huxley  --record
--------------------------------------
Running Huxley file: HuxleyfileCurrent
--------------------------------------
[UTIL02-0_FAQs.html] Running test: UTIL02-0_FAQs.html
Begin record
Press enter to take a screenshot, or type Q+enter if you're done

1 screenshots taken
Press enter to take a screenshot, or type Q+enter if you're done
q

Up next, we'll re-run your actions to generate screenshots to ensure they are pixel-perfect when running automated. Press enter to start.

Begin rerecord
  Sleeping for 1395.0 ms
  Clicking [726, 285]
  Sleeping for 2284.0 ms
  Taking screenshot 0

Playing back to ensure the test is correct

Begin playback
  Sleeping for 1395.0 ms
  Clicking [726, 285]
  Sleeping for 2284.0 ms
  Taking screenshot 0
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "build/bdist.macosx-10.9-intel/egg/huxley/threadpool.py", line 32, in >thread
    func(*args, **kwargs)
  File "build/bdist.macosx-10.9-intel/egg/huxley/cmdline.py", line 69, in >run_test
    screensize=screensize
  File "build/bdist.macosx-10.9-intel/egg/huxley/main.py", line 110, in main
    TestRun.record(local_d, d, (url, postdata), screensize, filename, >diffcolor, sleepfactor, save_diff)
  File "build/bdist.macosx-10.9-intel/egg/huxley/run.py", line 152, in record
    cls.rerecord(test, path, url, remote_d, sleepfactor, diffcolor, save_diff)
  File "build/bdist.macosx-10.9-intel/egg/huxley/run.py", line 79, in rerecord
    cls.playback(test, path, url, d, sleepfactor, diffcolor, save_diff)
  File "build/bdist.macosx-10.9-intel/egg/huxley/run.py", line 85, in playback
    run._playback(sleepfactor)
  File "build/bdist.macosx-10.9-intel/egg/huxley/run.py", line 95, in _playback
    step.execute(self)
  File "build/bdist.macosx-10.9-intel/egg/huxley/steps.py", line 103, in >execute

   raise TestError('Screenshot %s was different.' % self.index)

TestError: Screenshot 0 was different.

The page has no animation or anything on it that would cause the screenshot to be different.

[Nathan Birrell]

@igreulich could you please explain how you reached a solution?

[Ian Greulich]

I have been trying to recreate the original bug I had. So far, no luck.

Off the top of my head I think the problem I had was that I misread the docs. I was expecting huxley to be the same as huxley --record.

What dos your Huxleyfile look like?

[Nathan Birrell]

Basically just this, but repeated for a lot of pages:

[filename.html]
url=http://0.0.0.0:4567/filename.html
sleepfactor:0.3

[Nathan Birrell]

And it only happens on a specific set of our pages which is odd. Does Huxley use any JavaScript that could potentially alter the look of the page?
It seems to be changing the length of the window between playback and verification.

[Nathan Birrell]

Okay, so we have solved the problem. It turns out it was an auto-generated lorem ipsum text from Middleman which was changing on every page load: lorem.paragraphs rand(10)+1.

Thanks anyway @igreulich.