Giters

pixelated / puppet-pixelated

Puppet Module to setup the Pixelated User Agent

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Functional tests don't remove test users and fill up disk

varac opened this issue · comments

commented

Every hour the pixelated functional tests are run by cron. But the created testusers are not removed, and they keep filling the disk:

screenshot_2016-05-18_14-10-14

here's the disk usage graph indicating that every hour to grows:

screenshot_2016-05-18_14-14-08

(from https://unstable.pixelated-project.org/pnp4nagios/index.php/graph?host=unstable1.unstable.pixelated-project.i&srv=fs_/storage)

also, but off topic, it would be good to name the testusers starting with 'test-' or sth so you can easily spot them as testusers.

commented

There are four kinds of users:

  • regular users
  • admin users
  • tmp users: users with a username that match /^tmp_user_/
  • test users: users with a username that match /^test_user_/

for users that are created and destroyed frequently, it is highly preferable to use "tmp_user_xxx" because these special users are stored in a separate couchdb database (tmp_users instead of users). This allows the tmp_users db to be regularly destroyed so that tmp users don't bloat the users db over time by accruing lots of tombstone documents for the deleted users.

there is one problem with 'tmp' users: they don't have Identity documents. This means that you cannot deliver mail to these 'tmp' users. So, for mail integration tests, users with "test_user_xxx" are needed. These users are long lived, and should not be created and destroyed each time the test is run. These "test_user" accounts can be created and destroy special by authenticating with the API using the "monitor" authentication tokens (api_monitor_auth_token in secrets.json). The monitor token is not authorized to create/destroy/update regular or admin users, just "test" users.

This silly system can go away once we are off couchdb.

commented

behave-testuser gets deleted now.
For qa: If you want to see if the user is still in the database, use this:

/usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep --color=auto 'behave'

Or use the webapp if you have an admin account

commented

how often are these test users created and destroyed? unless the users start with 'tmp_user_', they will slowly bloat the users database with tombstone records.

commented

okay, what would you suggest @elijh .
We want to test sending mail, so tmp_test_XX can not be used.
We already can delete users because we can read the netrc file.
So does it make sense to change the username to test_user_XX ?
If there are tombstone records left that can not be deleted,
would it helpf if we ran these tests less often? maybe once a week?
How bloated will the database be then?
We do not want to keep the user around because we want to test a
defined state.
Maybe we should not run these tests on production then?

commented

for the leap test run tests, it creates a permanent TEST_EMAIL_USER (test_user_email) that is always used for testing email delivery. This user is never destroyed, once created, but the per-user mail storage db for this user is destroyed frequently, so no db bloat.

https://github.com/leapcode/leap_platform/blob/develop/tests/white-box/mx.rb#L128

yes, i think running the tests once a week would not create noticeable db bloat. i don't remember how big the tombstone records are, but they are small. nonetheless, small adds up quickly when you run a test every five minutes.

commented

cool. will change the cronjob to once a week.
Problem is, once they fail they will stay red for a week (or retriggered manually)
So we have to make sure they don't fail :)
Where get the mails fro the test_user_email user get deleted?
Maybe we should have done the same in the first place. We wouldn't have to touch the tests for that ...

commented

Where get the mails fro the test_user_email user get deleted?

it is not very elegant, but you can see it here https://github.com/leapcode/leap_platform/blob/develop/tests/white-box/couchdb.rb#L130

out of curiosity, what mail delivery tests are needed that can't use test_user_email?

commented

we could rename the user to test_user_email. We only need to know the password.
Our tests login to the webinterface, send an email to self, receive that email and check if the
labels (encrypted,signed) are correct.

commented

we only delete the user-database now. user does not get deleted:
d025c97

commented

looks like it spits out some passwords or uuids when i run it with

INVITE_CODE_ENABLED=true /usr/bin/mk-job pixelated-functional-tests /usr/local/bin/behave --tags @staging --tags ~@wip --no-capture -k /srv/leap/tests_custom/functional-tests/

I don't paste the output in case it's the pw :D

commented

What i did:

  • updated both puppet-pixelated submodules in pixelated-platform-environments repo
  • deployed to dev1
  • on dev the tests break:
root@dev1:/srv/leap/couchdb/scripts# INVITE_CODE_ENABLED=true /usr/bin/mk-job pixelated-functional-tests /usr/local/bin/behave --tags @staging --tags ~@wip --no-capture -k /srv/leap/tests_custom/functional-tests/
Traceback (most recent call last):
  File "/usr/local/bin/behave", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/behave/__main__.py", line 111, in main
    failed = runner.run()
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 659, in run
    return self.run_with_paths()
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 664, in run_with_paths
    self.load_hooks()
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 623, in load_hooks
    exec_file(hooks_path, self.hooks)
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 306, in exec_file
    execfile(filename, globals, locals)
  File "/srv/leap/tests_custom/functional-tests/environment.py", line 22, in <module>
    from page_objects import SignUpPage
  File "/srv/leap/tests_custom/functional-tests/page_objects/__init__.py", line 29, in <module>
    from leap_login_page import LeapLoginPage
ImportError: No module named leap_login_page
commented

deployed again to dev1, ran functional tests there, but the user-db of the behave-user is still there:

root@dev1:/srv/leap/couchdb/scripts# INVITE_CODE_ENABLED=true /usr/bin/mk-job pixelated-functional-tests /usr/local/bin/behave --tags @staging --tags ~@wip --no-capture -k /srv/leap/tests_custom/functional-tests/

@staging
Feature: account # ../../tests_custom/functional-tests/account.feature:17

  @register
  Scenario: user goes to signup-page  # ../../tests_custom/functional-tests/account.feature:20
    When I visit the signup-page      # ../../tests_custom/functional-tests/steps/account.py:61 0.218s
    Then I should see a signup button # ../../tests_custom/functional-tests/steps/account.py:66 0.007s

...
  @unencrypted
  Scenario: user receives an unencrypted email  # ../../tests_custom/functional-tests/send_mail.feature:29
    Given I send an unencrypted email           # ../../tests_custom/functional-tests/steps/send_mail.py:37 0.179s
    And I login as behave-testuser              # ../../tests_custom/functional-tests/steps/send_mail.py:29 17.797s
    When I open the unencrypted email           # ../../tests_custom/functional-tests/steps/send_mail.py:84 30.030s
    Then I see a unencrypted email flag         # ../../tests_custom/functional-tests/steps/send_mail.py:99 0.035s

2 features passed, 0 failed, 6 skipped
7 scenarios passed, 0 failed, 9 skipped
23 steps passed, 0 failed, 62 skipped, 0 undefined
Took 3m29.227s


root@dev1:/srv/leap/couchdb/scripts# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep behave
{"id":"85daeaa5e7d69c10d7f976f38c6e7ab7","key":"85daeaa5e7d69c10d7f976f38c6e7ab7","value":{"rev":"6-e4a8debeeaadb3901d23a9995cd826b0"},"doc":{"_id":"85daeaa5e7d69c10d7f976f38c6e7ab7","_rev":"6-e4a8debeeaadb3901d23a9995cd826b0","user_id":"85daeaa5e7d69c10d7f976f38c4e3378","address":"behave-testuser@dev.pixelated-project.org","destination":"behave-testuser@dev.pixelated-project.org","keys":{"pgp":"-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFcwPAEBEADIJ/TDv1SFy1jB+MsyTlE+vQjPrEUmJ61sMX5bRJ+YboQnfWbg\nN2JGOGhlzxwOrHtHI9Qwvxc70KyJJparOnY/ZH824o7SR6OBkDzUe5Mt3T1KQtJR\n11diwUWtzNoZ88DCmwUohng/a7mEJ3gha1uO4Ng2Kxr4JtGYmSBZKUerQfUHgISl\nGSIxshBVr6XEQ4frJojvp1eq9fmLf0Sgi3AotjSRMJ7yT6NaUZjouH892I84wkoB\npYbkOzHUxrINB7zEn5P5k1FOrVZY3Uyu2JbWl9ZJWcX70Vv3EkU8KwoTC1NKScXO\nFfw7WXJzwE9zDzW/OGszsID9DL42XP4p3FYO/y3L5LblzuYIUofOynKA0FfQEubR\nDQ+7cuLWImANVW+0bsJc/7MF+DdK99N4/5KSzDFqUeeDXmZ9JdkSBtr5/xPq4cDj\nPjg6nCCKfne6Vi+IvpoNy/VwChN50Bhkv95Q1ekyR5rgAycwFxhAVnw/v/6fTqmQ\nIThm5BDrwok3W6am795g0DiImO8zUybNKo34Nvj7ci5ndq4IaRngTHnYwzfVnFtx\nsPyFfXMjjxsyY6XbboXR+teFmePKm+SjZAmBkMKOngbjOaioJByp8bRmu5PM878Y\n+Q6X1V7BQ1CF8BH1SsZBL06X2+Y8XWNlXLLMQV+XC7JYauX8gEUX0t87+QARAQAB\ntFViZWhhdmUtdGVzdHVzZXJAZGV2LnBpeGVsYXRlZC1wcm9qZWN0Lm9yZyA8YmVo\nYXZlLXRlc3R1c2VyQGRldi5waXhlbGF0ZWQtcHJvamVjdC5vcmc+iQI+BBMBAgAo\nBQJXMDwBAhsvBQkB4K5fBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRA3CuKF\nFVrSie8tD/9nsnd0lnB25lDA0j+jez8Hl9jjS50/Cn8I0JMOmHqSJV0yzaROyhPS\nJdfG0hFOxiZwG6P7WTxmkHZjfihqMWPFdcLZ6b6ywNoKQQK+j7vBfNxj6RsH8m/M\nOgsSZpckWptCk19Qu+FzLCkS4BcHBpTCKhSfSIb3+Uxw0luBBStyilZoDPm2+ckF\ntrPk0cNHKMgs2mgVzTsk9/4pBcz0AuLLV5OYgUeoobte39mM3oGGR2ciO4P7Rb9h\nbyxsteZWmU/DmBz0gPy9XFbfeWqNX+8i2FqvihKgYl2O34TAh9gaa6dW9xh8/pcw\nCMCCSrBEUSbq2tWm4HyQt5EtDaKD/kxqmOCzkmxmwMnjLg1JKASPCW9+d98vhTFf\nL8qxpqifLTw9trx4+7fjbR3UusU8SlPNG44+moqj3tgdF2uz8I0ug9SzW3TnpsSS\nUZBo4+KPpdLfLP6EfPlxARL6yuUJNkNmyzwrsdjNtTHZpC0bA0J4euj+QHwBhLih\nzm+hpVX2Cwy3M9HXIRtPXTBWUtSqlY2mFrPqd7jQZ+GNP16Ku39pORFzRCALhDlW\nJsusM+sbwwlOAYoPaerR0az12n2Mw+1fp9KNwR0QIqsX84C+49fNqIbXd9PFVd5Y\nOw1KS84Zius+5N6TEToXfAzdd+dhhMZP18I23r/vJnv8rsp7YnWopw==\n=5lSl\n-----END PGP PUBLIC KEY BLOCK-----\n"},"cert_fingerprints":{"16:92:54:22:77:a9:71:12:8f:21:9c:4a:3e:66:d2:96:12:9f:5c:95":"2016-07-09","1e:6d:46:88:7d:6d:45:98:72:96:62:02:66:b5:6c:e5:af:95:88:5f":"2016-07-17","cb:4b:45:4d:c6:21:15:c5:8f:9c:ca:9f:57:04:34:62:54:cf:c7:9b":"2016-07-23","6b:8e:d9:30:c2:52:7e:18:64:b2:5a:1e:ca:21:ff:32:bf:66:29:f7":"2016-07-24"},"enabled":true,"type":"Identity"}},


root@dev1:/srv/leap/couchdb/scripts# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/_all_dbs' | json_pp | grep 85daeaa5e7d69c10d7f976f38c4e3378

   "user-85daeaa5e7d69c10d7f976f38c4e3378",
commented

any updates on this ?

commented

all envs are green and the db gets deleted.

commented

to qa:
login to any env and check which user_id the behave-testuser has:

/usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep behave

Then check if the database exists (it shouldn't):

 curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/_all_dbs' | json_pp | grep <USER_ID>
commented

on unstable the behave-userdb still exists:

root@unstable1:~# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep behave 
{"id":"ad8feb7e16b749039fe0eb3822fab002","key":"ad8feb7e16b749039fe0eb3822fab002","value":{"rev":"3-0dff17498767a69bd2e7d42a7edbca63"},"doc":{"_id":"ad8feb7e16b749039fe0eb3822fab002","_rev":"3-0dff17498767a69bd2e7d42a7edbca63","user_id":"ad8feb7e16b749039fe0eb3822dc2d27","address":"behave-testuser@unstable.pixelated-project.org","destination":"behave-testuser@unstable.pixelated-project.org","keys":{"pgp":"-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFdNZ0sBEACszIAV5CpX4CJUQ0pOnwMR6eE/y3FMiBjgXTcn6C1srlonobht\nxXDb6pjHlRx+8/N7peu+uCqUO2bp73SsqZiVV3iKIDDJYbu91tM6oOGGQEsCVVYM\ntNMlWYXt9cOzc9CnBiTYoHax9CTh1kSR6fjJORxxbHLaLmp/pAQNxf8BxWZyXqm/\nDyDlxFpAi7ujhitiLo5i+Bs3llFNOGNpGaKqUSXZ7k7cecvZBBLiuKADj3h+rTXP\n142TZyr8k9GHB3rL7iWXeQnNk2VTBTHAGFqhYkAer1XXZwZ0Kz1SC8GO7DJfLSod\nVWVXmCVz/zHdtxoZjY/NzJkgvaYqeGXhGZ5q8vyIfi7sXQM7dypcWAbnA2TFEEGG\ncusoIce5zR6hSmq3UGeb5U5gtByfwfKjLiSjh01nvYYEM5rhdAayzZXjVK5i5G+Z\nIMfzHkLx5lQCNOLMFx2ACQZXSwD2uIzA8e7XzaIjFL2Fdug8KJCGg4v56KqTl5C6\nOZHtWiZXQf8fpNz7IkGbQlXoIUw++uvhTVK/17BjUi5o/cfFVcMVJHQ9/9mPkDAQ\nBIExQr2P3/dr12a7b9Z6X7qdUh6GzANSRdMo47ET59i5CDBWEF/a78I/6tOOQH+x\nx8MZvRPQK3VqlHFi7LNFcRg4kvzvOgH75Dhvf7SOYRh6Rqm+J3tj1Fy5gQARAQAB\ntF9iZWhhdmUtdGVzdHVzZXJAdW5zdGFibGUucGl4ZWxhdGVkLXByb2plY3Qub3Jn\nIDxiZWhhdmUtdGVzdHVzZXJAdW5zdGFibGUucGl4ZWxhdGVkLXByb2plY3Qub3Jn\nPokCPgQTAQIAKAUCV01nSwIbLwUJAeCEFQYLCQgHAwIGFQgCCQoLBBYCAwECHgEC\nF4AACgkQia8HOtFppxCRIA/9GWJf7+b0yh7hDfJXHguTCvQ4MY+6M/67pVqTZe7D\n71BtkBF1PtRAVpE85t0WUTRXB5WBcBcblsX9OS53W6whY5rcbMHIrzgA7dHO5uoB\nhZPYxBEoCVoMjyItnTnaQfAQesCdy+Zeb3nq1WlwHs5Q8dHoFIuqY8GHV5hpVhr/\nZZz/Cz3mBqgkTv9Mq6jrRHEfXKTEsF68BH8E7KBnCTF2978CDMa7L8Ko8/DGFrzf\nalUB0xIwz5Joth3oXch5v8jtMcIJCG0fNGvgwknI+l3d/0s9l8eCo6/pNqGVIWWW\nZWj0kZoashuZFka+o/TwjZt3wRXwzbseC3Izbm9vsUk3JZoSp5LMWjacHJ+WqnRW\nvRPKSphYKs7Yd/HXH4EwLUur5Kl5umlK1UAfcQnvTjnxgKRxLwSQQKyMG4DeF4JV\nexUCnXC0YeeFrtcEXH+FNBno1z4DvbUTWPe/jF44kyz5jtz5fPVLPVBdBjjO0X86\npD5jI6fs9xI/XfqCvJdTHh3bSygtidAr2z4qo9dyyyziKVFFSd8/8vQDJf1uMdDI\n1i1xmVS9zi4xelUlYsJ9rYr/+i+uAFnOqhb9lxaPKzJo24Zj3nBZ+YPUTrnjs7QT\na8HVE6vIzyE5UK62a72jHEG7yNVzmjm0oRdHwy9jiNlZaqYSXTahFLPYxcv716RQ\nkao=\n=MJrH\n-----END PGP PUBLIC KEY BLOCK-----\n"},"cert_fingerprints":{"1c:1f:fe:50:d4:5f:97:ab:f4:9a:88:5e:45:07:89:89:2a:55:69:a2":"2016-07-31"},"enabled":true,"type":"Identity"}},

root@unstable1:~# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/user-ad8feb7e16b749039fe0eb3822dc2d27' 
{"db_name":"user-ad8feb7e16b749039fe0eb3822dc2d27","doc_count":431,"doc_del_count":0,"update_seq":2134,"purge_seq":0,"compact_running":false,"disk_size":20934768,"data_size":801296,"instance_start_time":"1465288286911492","disk_format_version":6,"committed_update_seq":2134}
commented

they were failing on unstable because user-agent was not able to login.
Restart helped. Functional tests work again

commented

@cz8s even with failing tests, the user database should be deleted too?
Because on dev worked fine. But on unstable (where tests are failing) the database was not deleted 😕

commented

the database can not be deleted if the login fails. so if the tests fail, the database does not get deleted.

commented

OK, só I'll try to discover why the testa are failing on unstable. For now we can launch this issue

commented

eventually, every soledad db without an associated user should get destroyed, but it might take a while. the script is https://leap.se/git/couchdb_scripts.git/blob_plain/HEAD:/cleanup-user-dbs

I have not confirmed that it is actually being run.

commented

I have not confirmed that it is actually being run.
It is run by cron daily, see root's crontab with crontab -l:

# Puppet Name: cleanup_stale_userdbs
7 4 * * * (/bin/date; /srv/leap/couchdb/scripts/cleanup-user-dbs) >> /var/log/leap/couchdb-cleanup.log