robertmuehsig / opengever.core

OneGov GEVER core package

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

opengever.core

Development installation

To get a basic development installation, make sure the dependencies listed below are satisfied and run the following steps:

$ git clone git@github.com:4teamwork/opengever.core.git
$ cd opengever.core
$ ln -s development.cfg buildout.cfg
$ python bootstrap.py
$ bin/buildout

Dependencies

Python 2.7

opengever.core requires at least Python 2.7, and using a 64-bit build of Python is highly recommended.

SQL Database

opengever.core requires a SQL database to store some configuration. Before you can configure your first client you need to set up a database.

Currently there are three SQL databases supported:

  • PostgreSQL
$ brew install postgresql --with-python
$ brew services start postgresql
$ brew services run postgresql
$ createdb opengever
  • MySQL
$ brew install mysql
$ mysql -u root
> CREATE DATABASE opengever CHARACTER SET utf8;
> GRANT ALL ON opengever.* TO opengever@localhost IDENTIFIED BY 'opengever';
> FLUSH PRIVILEGES;
  • Oracle

OpenLDAP 2.x

The Python ldap module requires the OpenLDAP 2.x client libraries.

Java

If fulltext indexing using ftw.tika is enabled, Java is required in order to run tika-server (at least JRE 1.6 is required for Tika).

LaTeX

A LaTeX distribution and the pdflatex binary are required for generating dossier covers, dossier details and dossier listing PDFs as well as open task reports and task listing PDFs.

For CentOS, the tetex-latex package contains the pdflatex binary. For local development on OS X we recommend the MacTeX distribution.

There is a 4teamwork internal devdocs LaTeX section on how to install pdflatex with our own fonts.

HAProxy

For a production installation you need to configure at least two Zope instances per AdminUnit (in order to avoid deadlocks when remote-requests are executed during tasks across AdminUnits).

To balance load between Zope instances we use HAProxy. The configuration is pretty standard:

frontend admin-unit-1
    bind *:10001
    default_backend admin-unit-1

backend admin-unit-1
  appsession __ac len 32 timeout 1d
  cookie serverid insert nocache indirect
  balance roundrobin
  option httpchk

  server admin-unit-1-01 10.0.0.1:10101 cookie admin-unit-1-01 check inter 10s maxconn 5 rise 1
  server admin-unit-1-02 10.0.0.1:10102 cookie admin-unit-1-02 check inter 10s maxconn 5 rise 1

Apache

In order to set up a reverse proxy that proxies requests to several HAProxy frontends we use Apache.

Postfix

Mail-In as well as Mail-Out functionality requires an MTA - we recommend Postfix. See ftw.mail's README for details on how to configure Mail-In.

Perl and Email::Outlook::Message module

In order to convert Outlook *.msg messages to RFC822 *.eml when using Drag&Drop upload, we use the msgconvert.pl script. This script requires Perl and the Email::Outlook::Message module.

For production deployments, this module will be installed by Ops via Puppet (it's now packaged as an RPM).

If you need this module for local development on macOS, you can also install it using Perl local::lib and CPAN. You then need to install Perl, perl-YAML and the following Perl modules:

Email::Outlook::Message
Email::LocalDelivery
Getopt::Long
Pod::Usage

In the end, GEVER will look for the msgconvert executable in $PATH.

Sablon

If opengever.meeting is activated (which it is for the default development installation), the Ruby gem Sablon is required to generate documents from *.docx templates. Sablon is executed as subprocess so the sablon script provided by the sablon gem must be accessible as the user that is running gever instances.

In order for buildout to be able to install the Sablon gem, you need to have bundler installed. For local development on Mac OS X it is recommended to set up your Ruby using rbenv and the ruby-build plugin:

git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
source ~/.bash_profile
rbenv install 2.4.5
gem install bundler

The installation of the Sablon gem can then be performed by buildout (by extending from ruby-gems.cfg).

LDAP credentials

LDAP and AD plugins get configured as usual, using an ldap_plugin.xml file in the profile of the respective policy package - with one exception:

Credentials for the LDAP service (bind DN and bind password) will NEVER be checked in in the ldap_plugin.xml, but instead will be stored machine-wide in a file ~/.opengever/ldap/{hostname}.json where {hostname} refers to the hostname of the LDAP server.

When an OpenGever client then is created using opengever.setup, the credentials are read from that file and configured for the LDAPUserFolder as well as the active LDAP connection.

So, for a local development installation, create the following file:

~/.opengever/ldap/ldap.4teamwork.ch.json

with these contents:

{
  "ldap":{
    "user":"<bind_dn>",
    "password":"<bind_pw>"
  }
}

<bind_dn> and <bind_pw> refer to the username and password for the respective user in our development LDAP tree.

Installing and activating Solr

Extend the Solr configuration in your buildout.cfg and add the Solr directive:

[buildout]
extends =
    development.cfg
    https://raw.githubusercontent.com/4teamwork/gever-buildouts/master/solr.cfg
solr-port = 8983

Buildout

$ bin/buildout

Then start Solr

$ bin/solr start

Run the activate_solr maintenance script:

$ bin/instance run src/opengever.maintenance/opengever/maintenance/scripts/activate_solr.py

Setting up a multi-admin environment

If you need a multi-admin environment, make sure the basic development dependencies above are satisfied and run the following steps:

Pleace note that the default database-name for multi-admin environment is opengever-multi-admin

$ git clone git@github.com:4teamwork/opengever.core.git
$ cd opengever.core
$ ln -s development-multi-admin.cfg buildout.cfg
$ python bootstrap.py
$ bin/buildout
$ bin/instance fg

Go to http://localhost:8080/manage_main and click on Install OneGov GEVER,

For the first admin-unit choose the following settings:

Property Value
Deployment profile Choose the Finanzdirektion (FD) (DEV)
LDAP configuration profile OneGovGEVER-Demo LDAP
Import users from LDAP into OGDS True
Development mode False
Purge SQL True

For the second admin-unit choose the following settings:

Property Value
Deployment profile Choose the Ratskanzlei (RK) (DEV)
LDAP configuration profile OneGovGEVER-Demo LDAP
Import users from LDAP into OGDS False
Development mode False
Purge SQL False

After installing both admin-units, you have to set a shared session-secret to share login-sessions between admin-units. To do this, do the following steps for both admin-units:

  • Goto: {admin-unit}/acl_users/session/manage_secret
  • Set a Shared secret

Lastly you have to change the admin-unit urls in the database to localhost.

  • Table: admin_units
  • Properties: site_url and public_url

PostgreSQL-Example:

UPDATE admin_units SET site_url = replace("site_url", 'https://dev.onegovgever.ch', 'http://localhost:8080'), public_url = replace("public_url", 'https://dev.onegovgever.ch', 'http://localhost:8080');

OGDS synchronization

For quick lookups for user information and metadata (that isn't relevant for security), we keep a mirrored list of users, groups, and group memberships in SQL tables in the OGDS.

Among other things, this list of users is used to determine what users are valid assignees for various objects: If a user was removed from the LDAP, he is still supposed to be a valid assignee for existing objects, but should not be suggested for selection for newly created objects.

Therefore users that are already contained in the SQL tables but have disappeared from LDAP are not removed from SQL, but instead flagged as inactive upon synchroniszation.

There's several different ways to perform the OGDS synchronization:

  • It can be triggered manually from the @@ogds-controlpanel (or by directly visiting the @@sync_users or @@sync_groups views)
  • It will automatically be done when setting up a new AdminUnit
  • It can be done from the shell by running the bin/instance sync_ogds zopectl command (the respective instance must not be running)
  • For deployments, a cron job that calls bin/instance0 sync_ogds should be created that syncs OGDS as needed

Since the OGDS is shared between AdminUnits in the same cluster, the synchronization will only have to be performed on one Zope instance per cluster.

Updating translations

Updating translations can be done with the bin/i18n-build script. It will scan the entire opengever.core package for translation files that need updating, rebuild the respective .pot files and sync the .po files.

Usually you work on a specific package and you want to only rebuild this package:

bin/i18n-build opengever.dossier

For building all packages, use the --all option:

bin/i18n-build --all

Theme Development

You will need the sass command for compiling SCSS to CSS. Start the bin/sass-watcher script and it will pick up changes base on filesystem events and compile the style files automatically for you.

There is a Gemfile to help make SASS versions consistent across development environments. Please refer to http://bundler.io/ for more details.

Updating the history file

The docs/HISTORY.txt file is a hotspot for git merge conflicts. In order to reduce merge conflicts we use the git union merge strategy for auto-resolving merge conflicts. For this to work smoothly developmers must follow theese rules when adding changelog entries:

  1. Always add a new entry at the top of the unreleased section.
  2. Add your [name] onto the same line, it should never be on a standalone line, otherwise it might be deleted by the union merge.
  3. Do not insert any empty lines.
  4. Avoid nested lists in your entry, because it makes auto-merging brittle. It is better to add each change as a separate changelog entry and prefix them, as shown below (see Feature x). If you must use nested lists, make sure to add an empty line before and after the list.
  5. You must rebase when you do not "make the release", so that your entry is not added to an already released section. Git cannot resolve that.

Example:

17.12.72 (unreleased)
---------------------

- Fix critical bug. [Susanne]
- Lots of changes after a lot of time. [Fritz]
- Update translations. [Fritz]
- Feature x: implement new things. [Susanne]
- Feature x: fix bug. [Susanne]

Updating API docs

In order to build the Sphinx API docs locally, use the provided bin/docs-build-public script:

bin/api-docs-build

This will build the docs (using the html target by default). If you'd like to build a different output format, supply it as the fist argument to the script (e.g. bin/docs-build-public latexpdf).

If you made changes to any schema interfaces that need to make their way into the docs, you need to run the bin/instance dump_schemas script before running the docs-build-public script:

bin/instance dump_schemas

This will update the respective schema dumps in docs/schema-dumps/ that are then used by the docs-build-public script to render restructured text schema docs.

Versions

Versions are pinned in the file versions.cfg in the opengever.core package.

Versions in development

In order to add a new dependency or to update one or many dependencies, follow these steps:

  1. Append new and changed version pinnings at the end of the [versions] section in the versions.cfg in your local opengever.core checkout.
  2. Run bin/cleanup-versions-cfg, review and confirm the changes. This script removes duplicates and sorts the dependencies.
  3. Commit the changes to your branch and submit it along with other changes as pull request.

Versions in production

For production deployments, the versions.cfg of a tag can be included with a raw github url in buildout like this:

[buildout]
extends =
    https://raw.githubusercontent.com/4teamwork/opengever.core/2017.4.0/versions.cfg

Scripts

Scripts are located in /scripts.

Repository configuration:

convert_csv_repository_to_xlsx.py <https://github.com/4teamwork/opengever.core/blob/master/scripts/convert_csv_repository_to_xlsx.py>: Converts repository configuration from old format (repository.csv) to new format (xlsx).

You have to install openpyxl to run this script!

bin/zopepy scripts/convert_csv_repository_to_xlsx.py <path to repository csv file> <path for new xlsx file>

Creating policies

A script to semi-automatically create policies is provided as bin/create-policy. The script runs in interactive mode and generates policies based on the questions asked. Policies are stored in the source directory src.

Policy templates are avilable from the opengever.policytemplates package. At the time of writing there is only one policy template for simple SAAS policies.

Once a new policy has been generated the following things need to be added manually:

  • an initial repository (as excel file)
  • initial template files, if required
  • initial sablon templates, if required
  • Some more complex confiuration options like retention periods and multiple inboxes/template folders

Tests

Fixture Objects

The fixture objects can be accessed on test-classes subclassing IntegrationTestCase with attribute access (self.dossier).

Users

  • self.administrator: nicole.kohler
  • self.archivist: jurgen.fischer
  • self.committee_responsible: franzi.muller
  • self.dossier_manager: faivel.fruhling
  • self.dossier_responsible: robert.ziegler
  • self.foreign_contributor: james.bond
  • self.manager: admin
  • self.meeting_user: herbert.jager
  • self.member_admin: david.meier
  • self.reader_user: lucklicher.laser
  • self.records_manager: ramon.flucht
  • self.regular_user: kathi.barfuss
  • self.secretariat_user: jurgen.konig
  • self.service_user: service.user
  • self.webaction_manager: webaction.manager
  • self.workspace_admin: fridolin.hugentobler
  • self.workspace_guest: hans.peter
  • self.workspace_member: beatrice.schrodinger
  • self.workspace_owner: gunther.frohlich

Objects

- self.committee_container
  - self.committee
    - self.cancelled_meeting
    - self.decided_meeting
    - self.decided_proposal
    - self.meeting
    - self.submitted_proposal
  - self.committee_participant_1
  - self.committee_participant_2
  - self.committee_president
  - self.empty_committee
  - self.inactive_committee_participant
- self.contactfolder
  - self.franz_meier
  - self.hanspeter_duerr
- self.inbox
  - self.inbox_document
  - self.inbox_forwarding
    - self.inbox_forwarding_document
- self.private_root
  - self.private_folder
    - self.private_dossier
      - self.private_document
      - self.private_mail
- self.repository_root
  - self.branch_repofolder
    - self.leaf_repofolder
      - self.cancelled_meeting_dossier
      - self.closed_meeting_dossier
      - self.decided_meeting_dossier
      - self.disposition
      - self.disposition_with_sip
      - self.dossier
        - self.document
        - self.draft_proposal
        - self.inbox_task
        - self.info_task
        - self.mail_eml
        - self.mail_msg
        - self.private_task
        - self.proposal
          - self.proposaldocument
        - self.removed_document
        - self.sequential_task
          - self.seq_subtask_1
          - self.seq_subtask_2
          - self.seq_subtask_3
        - self.shadow_document
        - self.subdossier
          - self.empty_document
          - self.subdocument
          - self.subsubdossier
            - self.subsubdocument
        - self.subdossier2
        - self.task
          - self.subtask
          - self.taskdocument
      - self.empty_dossier
      - self.expired_dossier
        - self.expired_document
        - self.expired_task
      - self.inactive_dossier
        - self.inactive_document
        - self.inactive_task
      - self.meeting_dossier
        - self.meeting_document
        - self.meeting_task
          - self.meeting_subtask
      - self.offered_dossier_for_sip
      - self.offered_dossier_to_archive
      - self.offered_dossier_to_destroy
      - self.protected_dossier
        - self.protected_document
      - self.protected_dossier_with_task
        - self.protected_document_with_task
        - self.task_in_protected_dossier
      - self.resolvable_dossier
        - self.resolvable_subdossier
          - self.resolvable_document
  - self.empty_repofolder
  - self.inactive_repofolder
- self.templates
  - self.ad_hoc_agenda_item_template
  - self.asset_template
  - self.docprops_template
  - self.dossiertemplate
    - self.dossiertemplatedocument
    - self.subdossiertemplate
      - self.subdossiertemplatedocument
  - self.empty_template
  - self.meeting_template
    - self.paragraph_template
  - self.normal_template
  - self.proposal_template
  - self.recurring_agenda_item_template
  - self.sablon_template
  - self.subtemplates
    - self.subtemplate
  - self.tasktemplatefolder
    - self.tasktemplate
- self.workspace_root
  - self.workspace
    - self.todo
    - self.todolist_general
    - self.todolist_introduction
      - self.assigned_todo
      - self.completed_todo
    - self.workspace_folder

Other values

  • self.committee_id: 1
  • self.empty_committee_id: 2

Parallelisation

Use bin/mtest for running all test in multiple processes. Alternatively bin/test runs the tests in sequence. The multi process script distributes the packages (e.g. opengever.task, opengever.base, etc) into multiple processes, trying to balance the amount of test suites, so that it speeds up the test run.

The bin/mtest script can be configured with environment variables:

  • MTEST_PROCESSORS - The amount of processors used in parallel. It should be no greater than the amount of available CPU cores. Defaults to 4.

Functional or integration testing?

We are shifting the tests from the older functional testing layer to the newer integration testing layer.

Integration testing:

  • Should be used for new tests!
  • Comes with a preinstalled testing fixtures.
  • Transactions are disabled for isolation purposes: transaction.commit is not allowed in tests.
  • Uses ftw.testbrowser's TraversalDriver.

Functional testing:

  • Should not be used for new tests, when possible.
  • Is factory-based, using ftw.builder.
  • Uses transactions.
  • Limited / slow database isolation: a fresh setup is necessary for each test.

Example integration test with browser:

from ftw.testbrowser import browsing
from ftw.testbrowser.pages import statusmessages
from opengever.testing import IntegrationTestCase

class TestExampleView(IntegrationTestCase):

    @browsing
    def test_example_view(self, browser):
        self.login(self.dossier_responsible, browser)
        browser.open(self.dossier, view='example_view')
        statusmessages.assert_no_error_messages()

Best practices

These best practices apply to the new integration testing layer.

Do not commit the transaction

Committing the transaction will break isolation. The testing layer will prevent you from interacting with the transaction.

Use the fixture objects

The testing fixtures create content objects, users, groups and client configurations (admin units, org units) which are available for all tests. They can and should be modified to the needs of the test.

Avoid creating objects (with ftw.builder)

Creating objects with ftw.builder or with ftw.testbrowser is expensive because it takes a moment to index the object. Therefore we want to avoid creating unnecessary objects within the tests so that the tests are faster overall.

Tests which have the job to test object creation (e.g. through the browser) obviously need to actually create an object, all other tests should try to reuse objects from the fixture and modify them as needed.

Use users from fixture

The fixture provides a set of standard users which should be used in tests. Do not use plone.app.testing's test user with global roles as it does not reflect properly how the security model of GEVER works. In order to test features which can only be executed by the system or by a Manager-user, the plone.app.testing's site owner may be used.

Login

Integration tests start with no user logged in. The first thing each test should do, is to log in the user with the fewest privileges required for doing the task under test.

The login command should not be moved to the setUp method; it should be clearly visible at the beginning of each test, so that a reader has the necessary context without scrolling to the top of the file.

When authenticated preparations are required in the setUp method, use self.login as a context manager in order to cleanup the authentication on exit, so that the tests still start anonymously.

from opengever.testing import IntegrationTestCase
from ftw.testbrowser import browsing

class TestExampleView(IntegrationTestCase):

    def setUp(self):
        super(TestExampleView, self).setUp()
        with self.login(self.administrator):
            self.dossier.prepare_for_test()

    def test_server_side(self):
        self.login(self.dossier_responsible)
        self.assertTrue(self.dossier.can_do_important_things())

    @browsing
    def test_client_side_with_browser(self, browser):
        self.login(self.regular_user, browser)
        browser.open(self.dossier)
        browser.click_on('Do important things')

Do not assert browser.contents

The statement self.assertIn('The label', browser.contents) will print the complete HTML document as failure message. This is distracting and not useful at all.

Instead you should select specific nodes and do assertions on those nodes, e.g.

from opengever.testing import IntegrationTestCase
from ftw.testbrowser import browsing

class TestExampleView(IntegrationTestCase):

    @browsing
    def test_label(self, browser):
        self.assertEquals('The label',
                          browser.css('label.foo').first.text)

This allows the browser to help when print a nice error message when the node was not found: NoElementFound: Empty result set: browser.css("label.foo") did not match any nodes.

When the view does not return a complete HTML document but, for example, a status only (OK), or it is some kind of API endpoint, browser.contents may be asserted.

Use tearDown carefully

Do not tear down changes which are taken care of by some kind of isolation:

  • Do not tear down ZODB changes: the ZODB is isolated by plone.app.testing.
  • Do not tear down SQL changes: we take care of that in the SQL testing layer with savepoints / rollbacks.
  • Do not tear down component registry changes (e.g. new adapters, utilities, event handlers) as this is taken care of by the COMPONENT_REGISTRY_ISOLATION layer.
  • Do tear down modifications in environment variables (os.environ).
  • Do tear down modifications stored in module globals (e.g. transmogrifier sections).

Use guard assertions

When your test expects a specific state in order to work properly, this state should be ensured by using guard assertions.

def test_closing_dossier(self):
    self.assertTrue(self.dossier.is_open(),
                    'Precondition: assumed dossier to be open')
    self.dossier.close()
    self.assertFalse(self.dossier.is_open())

If the self.dossier is changed to be not open by default anymore, the failure should tell us that a precondition was no longer met rather than implying that the close() method is broken. The statement also acts as "given"-statement and a reader can easily figure out what the precondition is, because it is visually separated.

Alternatively a precondition can be ensured by setting the state of the object:

def test_title_is_journalized_on_action(self):
    self.dossier.title = u'The dossier'
    action(self.dossier)
    self.assertEquals(u'The dossier',
                      last_journal_entry(self.dossier).title)

Activating feature flags

Feature flags can by activated test-case-wide by setting a tuple of all required flags:

class TestDossierTemplate(IntegrationTestCase):
    features = ('dossiertemplate',)

When a feature should not be activated test-case-wide it can be activated within a single test:

class TestTemplates(IntegrationTestCase):

    def test_adding_dossier_template(self):
        self.activate_feature('meeting')

See the list of feature flags.

Cache integration testing setup

When developing opengever.core, a developer often runs a single test module, with bin/test -m opengever.dossier.tests.test_activate for instance. This will set up a complete fixture each time. In order to speed up the feedback loop when developing, we try to cache the database after setting up the fixture. This will speed up the test runs, but it also makes the result inaccurate: if the cachekeys do not detect a relevant change, we may not realize that something breaks.

Because the results are not accurate and this is an experiment, the feature is considered experimental and therefore disabled by default.

You can enable the feature by setting an environment variable:

GEVER_CACHE_TEST_DB=true bin/test -m opengever.dossier.tests.test_activate

There is also a binary which does that for you for just one run for convenience:

bin/test-cached -m opengever.dossier.tests.test_activate

You can manually remove / rebuild the caches:

./bin/remove-test-cache

This feature is disabled on the CI server.

When the environment variable GEVER_CACHE_VERBOSE is set to true, a list of modified files will be printed whenever a cachekey is invalidated. This can be useful to debug problems with the fixture cache:

GEVER_CACHE_VERBOSE=true bin/test-cached -m opengever.dossier.tests.test_activate

Builder API

This project uses the ftw.builder package based on the Builder pattern to create test data. The opengever specific builders are located in opengever.testing

To use the Builder API you need to import the Builder function:

from ftw.builder import Builder
from ftw.builder import create

Then you can use the Builder function in your test cases:

dossier = create(Builder("dossier"))
task = create(Builder("task").within(dossier))
document = create(Builder("document")
                  .within(dossier)
                  .attach_file_containing("test_data"))

Note that when using the OPENGEVER_FUNCTIONAL_TESTING Layer the Builder will automatically do a transaction.commit() when create() is called.

Unit testing and mock tests

opengever.core has some unit tests (without a testing layer) and some mock test cases (usually with the COMPONENT_UNIT_TESTING testing layer).

When writing unit tests (with no layer), the developer must take into account that there is no isolation at all. The developer must make sure that neither the test nor any component used in the test leaks, or isolation must be ensured manually. The developer should also take into account that components under tests (or their dependencies) may be changed in the future.

By leaking we mean any kind of thing changed outside of the test scope. This includes registering components (adapters, utilites), changing globals (setSite, registering transmogrifier blueprints, environment variables) or any other action that can influence other components later.

If a developer cannot guarantee that the test is not leaking he/she shall not write a unit test, but use at least the COMPONENT_UNIT_TESTING layer or write an integration test.

The COMPONENT_UNIT_TESTING provides a minimal isolation of z3 componentes (adapters, utilites) and registers basic adapters such as annotations.

When using mock tests cases, which discourage from in general, always import the MockTestCase from ftw.testing in order to be compatible with COMPONENT_UNIT_TESTING.

Testserver

GEVER provides a testserver which sets up a GEVER in testing mode with a real HTTP server so that other applications and components can be tested. The testserver installs the standard GEVER testing fixture. By telling the server when to setup and teardown for each test it makes sure that the database is isolated and rolled back properly for each test.

Usage

In order to run the testserver, a local Development installation needs to be installed. Once installed properly, the server can be started with bin/testserver:

./bin/testserver
Plone:  http://localhost:55001/plone
XMLRPC: http://localhost:55002
...
18:13:39 [ ready ] Started Zope 2 server

Next you need to tell the testserver that you will now run a test:

./bin/testserverctl zodb_setup

Then you can make requests to http://localhost:55001/plone and use all the content and users generated by the fixture. It will be the exact same each run. The administrator login is admin and secret.

Once your test is finished you should tear down and re-setup for the next test in order to isolate the database properly:

./bin/testserverctl zodb_teardown
./bin/testserverctl zodb_setup

Making REST-API requests

The testserver sets up a service.user which has a REST-API service key and is allowed to impersonate other users. This is important for testing applications which use the REST-API. The service key can be downloaded here <https://github.com/4teamwork/opengever.core/blob/master/opengever/testing/assets/service_user_generic.private.json>.

Changing ports

The ports used by the testserver can easily be changed through environment variables:

  • ZSERVER_PORT - the port of the GEVER http server (default: 55001)
  • TESTSERVER_CTL_PORT - the port of the XMLRPC control server (default: 55002).
  • SOLR_PORT - the port of the Solr server which is controlled by the testserver (default: 55003).

Custom fixtures

A custom fixture can be loaded in the testserver. This is helpful when other projects are testing GEVER integration and need specific content. The custom fixture can be defined with an environment variable:

FIXTURE=~/projects/myproject/gever/fixture.py ./bin/testserver

The fixture will be loaded into the testserver process with the dottedname customfixture.fixture; the package name is always customfixture. It is possible to import local files of this folder with import .otherfile.

Example fixture:

from opengever.testing.fixtures import OpengeverContentFixture

class Fixture(OpengeverContentFixture):

    def __init__(self):
        super(Fixture, self).__init__()
        with self.freeze_at_hour(20):
            self.create_my_custom_content()

The fixture class name defaults to Fixture and can be changed with the environment variable FIXTURE_CLASS.

Testing on the CI

When developing third party applications, it is best practice to use a tape recording system. In local development, a real testserver should be started and tapes of its responses should be recorded. Those tapes should be committed to GIT so that no GEVER needs to be installed when running the tests on the CI - it will simply pull the tapes.

Whenever the application needs to support a new version of GEVER, a developer records all tapes when running a new version of the testserver, so that compatibility with the new version can be proven.

Testing Inbound Mail

For easy testing of inbound mail (without actually going through an MTA) there's a script bin/test-inbound-mail that can be used to test creation of inbound mail:

cat testmail.eml | bin/test-inbound-mail

The script assumes you got an instance running on port ${instance:http-address}, a GEVER client called fd and an omelette with ftw.mail in it installed. It will then feed the mail from stdin to the ftw.mail inbound view, like Postfix would.

Deployment

The following section describes some aspects of deploying OneGov GEVER. If you need an example of a simple deployment profile have a look at the examplecontent profiles, see: https://github.com/4teamwork/opengever.core/tree/master/opengever/examplecontent.

Setup Wizard

The manage_main view of the Zope app contains an additional button "Install OneGov GEVER" to add a new deployment. It leads to the setup wizard where a deployment profile and an LDAP configuration profile can be selected.

The setup wizard can be configured with the following environment variable:

  • IS_DEVELOPMENT_MODE - If set pre-selects the following options in the setup wizard: Import of LDAP users, Development Mode and Purge SQL. Currently these are all available options.

Deployment Profiles

Deployment profiles can be selected in the setup wizard. They are used to link a Plone site with its corresponding AdminUnit and they usually include a policy profile, additional init profiles and further Plone-Site configuration options. Deployment profiles are configured in ZCML:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:opengever="http://namespaces.zope.org/opengever"
    i18n_domain="my.package">

    <opengever:registerDeployment
        title="Development with examplecontent"
        policy_profile="opengever.examplecontent:default"
        additional_profiles="opengever.setup:repository_root,
                             opengever.setup:default_content,
                             opengever.examplecontent:init"
        admin_unit_id="admin1"
        />

</configure>

See https://github.com/4teamwork/opengever.core/blob/master/opengever/setup/meta.py for a list of all possible options.

LDAP Profiles

LDAP profiles can be selected in the setup wizard. They are used to install an LDAP configuration profile. LDAP profiles are configured in ZCML:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:opengever="http://namespaces.zope.org/opengever"
    i18n_domain="my.package">

    <opengever:registerLDAP
        title="4teamwork LDAP"
        ldap_profile="opengever.examplecontent:4teamwork-ldap"
        />

</configure>

See https://github.com/4teamwork/opengever.core/blob/master/opengever/setup/meta.py for a list of all possible options.

Content creation

Opengever defines four additional generic setup setuphandlers to create initial AdminUnit and OrgUnit OGDS entries, create initial documents/document templates, configure local roles and create an initial repository. Of course ftw.inflator content creation is available as well, for details see https://github.com/4teamwork/ftw.inflator.

Creating initial AdminUnit/OrgUnit

Add a unit_creation folder to your generic setup profile. To that folder add the files admin_units.json and/or org_units.json. The content is created when the generic setup profile is applied. Note also that this content is created before ftw.inflator content and before all the other custom gever content creation handlers.

AdminUnit example:

[
  {
    "unit_id": "admin1",
    "title": "Admin Unit 1",
    "ip_address": "127.0.0.1",
    "site_url": "http://localhost:8080/admin1",
    "public_url": "http://localhost:8080/admin1",
    "abbreviation": "A1"
  }
]

OrgUnit example:

[
  {
    "unit_id": "org1",
    "title": "Org Unit 1",
    "admin_unit_id": "admin1",
    "users_group_id": "og_demo-ftw_users",
    "inbox_group_id": "og_demo-ftw_users"
  }
]

Creating initial repositories

Gever repositories are initialized from an excel file. To add initial repository setup add a folder opengever_repositories to your generic setup profile. Each *.xlsx file in that folder will then be processed, the filename will serve as the ID for the repository root. See ordnungssystem.xlsx for an example. Note that this setuphandler is called after ftw.inflator but before custom GEVER content.

Creating GEVER specific content

Documents and Document templates are created with a customized ftw.inflator pipeline since they need special handling to have correct initial file versions. Thus documents should never be created with ftw.inflator but always with our customized pipeline. Since the custom pipeline is based on ftw.inflator we suggest to create all gever-content with this new pipeline.

To create content add an opengever_content folder to your generic setup profile. All JSON files in this folder are then processed similar to ftw.inflator. Note that this setuphandler is called after ftw.inflator.

Configuring local roles

To decouple local role assignment from content creation opengever introduces a separate setuphandler to configure local roles. To configure local roles add a local_role_configuration folder to your generic setup profile. All JSON files in that folder are then processed. Note that this setuphandler is called after ftw.inflator.

Example configuration:

[
    {
        "_path": "ordnungssystem",
        "_ac_local_roles": {
            "og_demo-ftw_users": [
                "Contributor",
                "Editor",
                "Reader"
            ]
        }
    }
]

About

OneGov GEVER core package


Languages

Language:Python 86.4%Language:CSS 9.4%Language:JavaScript 3.6%Language:HTML 0.4%Language:Shell 0.1%Language:TeX 0.1%Language:Ruby 0.0%