add functional tests that run regularly

Question

add functional tests that run regularly

shyba opened this issue 8 years ago · comments

From @cz8s on January 26, 2016 14:22

we already have some functional tests that run after every build. But they are not in a very good shape and it would be nice to have tests that can be run regularly to make sure everything still works.

Definition of done

running leap test tries to login with phantomjs and checks if the inbox is visible.

Copied from original issue: pixelated/pixelated-platform#155

Victor Shyba · Answer 1 · Fri Feb 26 2016 05:48:34 GMT+0800 (China Standard Time)

From @cz8s on January 26, 2016 14:40

possible problems:

only phantomjs2 works with the leap webapp.
we have to download the latest phantomjs binaries during installation.

Victor Shyba · Answer 2 · Fri Feb 26 2016 05:48:35 GMT+0800 (China Standard Time)

From @cz8s on January 26, 2016 15:0

if we run the tests, we need valid credentials. leap already has a way to create temporary users.
Thats why it would be good to integrate our tests into leap.
See https://github.com/leapcode/leap_platform/blob/develop/tests/helpers/bonafide_helper.rb#L7

Victor Shyba · Answer 3 · Fri Feb 26 2016 05:48:35 GMT+0800 (China Standard Time)

From @cz8s on February 8, 2016 10:54

the tests are run by check_mk now. To qa, look at nagios. there should be a a Test called 'Job pixelated' with a history.

Victor Shyba · Answer 4 · Fri Feb 26 2016 05:48:35 GMT+0800 (China Standard Time)

From @mnandri on February 11, 2016 16:0

I have tried on staging:

Job pixelated is not appearing on Nagios's list of services... so even if it is notified, it won't do anything.
Manually running the functional tests, that is scheduled in cron, actually fails... while I can login and use the user agent...

Notify @cz8s

Victor Shyba · Answer 5 · Fri Feb 26 2016 05:48:36 GMT+0800 (China Standard Time)

From @cz8s on February 16, 2016 9:24

we have to run check_mk -II and check_mk -R before the test shows up in nagios.
@varac: do you know a good way to do this?

Victor Shyba · Answer 6 · Fri Feb 26 2016 05:48:36 GMT+0800 (China Standard Time)

From @cz8s on February 16, 2016 9:25

the tests fail because the new dispatcher now send a content-security header which breaks phantomjs

Victor Shyba · Answer 7 · Fri Feb 26 2016 05:48:36 GMT+0800 (China Standard Time)

From @cz8s on February 16, 2016 10:50

maybe we should tell the user-agent to not send the csp header if the client sent a header indicating
that it doesn't want to see it? like X-NO-CSP-PLEASE=true

Victor Shyba · Answer 8 · Fri Feb 26 2016 05:48:37 GMT+0800 (China Standard Time)

From @cz8s on February 22, 2016 10:46

we are running the tests in our pipeline directly on the server now. that works and its the same way nagios runs them later

Victor Shyba · Answer 9 · Fri Feb 26 2016 05:48:37 GMT+0800 (China Standard Time)

From @cz8s on February 23, 2016 9:6

all jobs are registered with nagios on deploy.
we need to run a fake job on deploy to make sure the job is registered with nagios.
when the cronjob then runs the job, it will already be configured in nagios and the result
can be processed.

Victor Shyba · Answer 10 · Fri Feb 26 2016 05:48:38 GMT+0800 (China Standard Time)

From @varac on February 25, 2016 15:34

Using the curl/sh onliner from https://github.com/pixelated/puppet-pixelated#testing-pixelated, i get this error:

   - [node1.example.org] Error: /Stage[main]/Pixelated::Tests/Exec[dummy_register_job]: Failed to call refresh: /usr/bin/mk-job: line 65: /usr/bin/time: No such file or directory
   - [node1.example.org] Error: /Stage[main]/Pixelated::Tests/Exec[dummy_register_job]: /usr/bin/mk-job: line 65: /usr/bin/time: No such file or directory

Victor Shyba · Answer 11 · Fri Feb 26 2016 05:48:38 GMT+0800 (China Standard Time)

From @varac on February 25, 2016 15:37

also, bzip2 is missing:

   - [node1.example.org] Notice: /Stage[main]/Pixelated::Tests/Exec[install_phantomjs]/returns: tar (grandchild): bzip2: Cannot exec: No such file or directory
   - [node1.example.org] Notice: /Stage[main]/Pixelated::Tests/Exec[install_phantomjs]/returns: tar (grandchild): Error is not recoverable: exiting now
   - [node1.example.org] Notice: /Stage[main]/Pixelated::Tests/Exec[install_phantomjs]/returns:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
   - [node1.example.org] Notice: /Stage[main]/Pixelated::Tests/Exec[install_phantomjs]/returns:                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0antomjs]/returns: 
  0 22.3M    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0tar: Child died with signal 13
   - [node1.example.org] Notice: /Stage[main]/Pixelated::Tests/Exec[install_phantomjs]/returns: tar: Error is not recoverable: exiting now
   - [node1.example.org] Notice: /Stage[main]/Pixelated::Tests/Exec[install_phantomjs]/returns: curl: (23) Failed writing body (3511 != 16384)
   - [node1.example.org] Error: /usr/bin/curl -L https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2 | tar xvj --strip-components=2 -C /usr/local/bin/ phantomjs-2.1.1-linux-x86_64/bin/phantomjs returned 2 instead of one of [0]

Victor Shyba · Answer 12 · Fri Feb 26 2016 05:48:39 GMT+0800 (China Standard Time)

From @cz8s on February 25, 2016 15:43

why does this only happen if we install on vagrant?

Victor Shyba · Answer 13 · Fri Feb 26 2016 05:48:39 GMT+0800 (China Standard Time)

From @varac on February 25, 2016 17:10

because the vagrant baseimage is stripped down to the bare minimums, and bzip2 is not installed. This is at least the explanation for the second issue (which i already fixed).

Victor Shyba · Answer 14 · Fri Feb 26 2016 05:48:40 GMT+0800 (China Standard Time)

From @varac on February 25, 2016 17:21

the first issue about the missing /usr/bin/time binary is an check-mk upstream bug, it depends on the time package. I fixed this in leap_platform.

Christoph · Answer 15 · Wed Mar 02 2016 16:51:52 GMT+0800 (China Standard Time)

current status:
the tests ca not be run be nagios directly because they take too long.
we now create a cron job that runs them daily and reports to nagios via mk-job
The tests are red at the moment because of bug in sqlcipher.
@fbernitt and me are looking into the bug and will see if the tests become green on the way

Christoph · Answer 16 · Tue Mar 08 2016 16:28:59 GMT+0800 (China Standard Time)

Finally. Tests are green in go and on staging.
The way it works:

tests are run during deploy on the go-agent against staging.
tests are run regularly on the platform and the result is reported to nagios

to qa: login to nagios on staging and see that the Job pixelated-functional-tests is green.
If you want to run the tests locally, you have to shut down the firewall on the target system
(e.g. staging): shorewall clear
Then you should be able to run the tests:
puppet-pixelated/files/functional-tests ᐅ TESTHOST=staging.pixelated-project.org behave --tags ~@wip --tags @staging
Be very happy :)

Varac · Answer 17 · Tue Mar 08 2016 20:28:12 GMT+0800 (China Standard Time)

i deployed to dev.pix but the tests fails with No module named enum

root@dev1:~# /usr/local/bin/behave --tags @staging --tags ~@wip --no-capture -k /srv/leap/tests_custom/functional-tests/
Traceback (most recent call last):
  File "/usr/local/bin/behave", line 7, in <module>
    from behave.__main__ import main
  File "/usr/local/lib/python2.7/dist-packages/behave/__init__.py", line 21, in <module>
    from behave.matchers import use_step_matcher, step_matcher, register_type
  File "/usr/local/lib/python2.7/dist-packages/behave/matchers.py", line 5, in <module>
    from parse_type import cfparse
  File "/usr/local/lib/python2.7/dist-packages/parse_type/__init__.py", line 3, in <module>
    from parse_type.cardinality import Cardinality
  File "/usr/local/lib/python2.7/dist-packages/parse_type/cardinality.py", line 8, in <module>
    from enum import Enum
ImportError: No module named enum

apt-get install python-enum

fixed it.