The OS-autoinst project aims at providing a means to run fully automated tests. Especially to run tests of basic and low-level operating system components such as bootloader, kernel, installer and upgrade, which can not easily and safely be tested with other automated testing frameworks. However, it can just as well be used to test firefox and openoffice operation on top of a newly installed OS.
os-autoinst can be executed alone, but is currently designed to be executed together with openQA, the web user interface that allows to run more than one os-autoinst instance at the same time.
More information on os-autoinst and openQA can be found on http://os-autoinst.github.io/openQA/
Getting started
Under openSUSE the `os-autoinst` package can be installed from the official repository or from our link:https://build.opensuse.org/project/show/devel:openQA[devel repository]. For further details, have a look at the link:http://open.qa/docs/#_development_version_repository[openQA documentation]. For building os-autoinst manually checkout the build instructions below. To execute an instance of os-autoinst one needs to create a file named `vars.json`. It stores the values of the different variables that will configure the behavior of the test execution. There are some variables used by os-autoinst itself and other that are used by the tests. A minimal `vars.json` file can be: [source, javascript] ------------------------------------------------------------------- { "DISTRI" : "opensuse", "CASEDIR" : "/full/path/for/tests", "NAME" : "test-name", "ISO" : "/full/path/for/iso", "VNC" : "91", "BACKEND" : "qemu", "DESKTOP" : "kde" } ------------------------------------------------------------------- Be advised that the file `vars.json` is also modified by `os-autoinst` so make sure to backup handcrafted versions of this file. For more concrete instructions head down to the "How to run test cases" section below. How to contribute ----------------- If you want to contribute to this project, please clone and send pull requests via https://github.com/os-autoinst/os-autoinst. More information on the contribution can be found on http://os-autoinst.github.io/openQA/contact/, too. Issues are tracked on https://progress.opensuse.org/projects/openqav3/. For an overview of the architecture, see link:doc/architecture.md[doc/architecture.md]. Rules for commits ^^^^^^^^^^^^^^^^^ * Every commit is checked by https://travis-ci.org/travis[Travis CI] as soon as you create a pull request but you *should* run the os-autoinst tests locally. Checkout the build instructions for further details. * For git commit messages use the rules stated on http://chris.beams.io/posts/git-commit/[How to Write a Git Commit Message] as a reference * Every pull request is reviewed in a peer review to give feedback on possible implications and how we can help each other to improve If this is too much hassle for you feel free to provide incomplete pull requests for consideration or create an issue with a code change proposal. Build instructions ------------------ Installing dependencies ^^^^^^^^^^^^^^^^^^^^^^^ On openSUSE one can install the package link:https://build.opensuse.org/project/show/devel:openQA[`os-autoinst-devel`] which pulls all dependencies required to build with CMake. The required dependencies are also declared in `dependencies.yaml`. (The names listed within that file are specific to openSUSE but can be easily transferred to other distributions.) CMake ^^^^^ Create a build directory outside of the source directory. The following commands need to be invoked within that directory. Configure build: ---- cmake $path_to_os_autoinst_checkout ---- You can specify any of the standard CMake variables, e.g. `-DCMAKE_BUILD_TYPE=Debug` and `-DCMAKE_INSTALL_PREFIX=/custom/install/prefix`. The following examples assume that GNU Make is used. It is possible to generate for a different build tool by adding e.g. `-G Ninja` to the CMake arguments. Build executables and libraries: ---- make symlinks ---- This target also creates symlinks of the built executables and libraries within the source directory so `isotovideo` can find them. Run all tests: ---- make check ---- By default CTest is invoked in verbose mode because prove already provides condensed output. Add `-DVERBOSE_CTEST=OFF` to the CMake arguments to avoid that. Run all Perl tests (`*.t` files found within the `t` directory): ---- make test-perl-testsuite ---- Run individual tests by specifying them explicitly: ---- make test-perl-testsuite TESTS="15-logging.t 28-signalblocker.t" ---- Add additional arguments to the `prove` invocation, e.g. enable verbose output: ---- make test-perl-testsuite PROVE_ARGS=-v ---- Gather coverage data while running tests: ---- make test-perl-testsuite WITH_COVER_OPTIONS=1 ---- Generate a coverage report from the gathered coverage data: ---- make coverage ---- If no coverage data has been gathered so far the `coverage` target will invoke the testsuite automatically. Reset gathered coverage data: ---- make coverage-reset ---- Install files for packaging: ---- make install DESTDIR=… ---- Further notes: * When using the `test-perl-testsuite` target, `ctest` is not used (and therefore `ctest` specific tweaks have no effect). * One can always run Perl tests manually via `prove` after the build has been conducted with `make symlinks`. Note that some tests need to be invoked within the `t` directory. An invocation like `prove -vI.. -I../external/os-autoinst-common/lib 28-signalblocker.t` is supposed to work. * It is also possible to run `ctest` within the build directory directly instead of using the mentioned targets. * All mentioned variables to influence the test execution (`TESTS`, `WITH_COVER_OPTIONS`, …) can be combined and can also be used with the `coverage` target. GNU Autotools ^^^^^^^^^^^^^ The following commands need to be invoked within the top-level of the repository checkout. Configure build: ---- ./autogen.sh ---- Build executables and libraries: ---- make ---- Run all tests: ---- make check ---- Run individual tests by specifying them explicitly: ---- make check TESTS=23-baseclass.t ---- How to run test cases --------------------- This following instructions shows how to run test cases. First one needs to clone the test distribution. Checkout link:https://github.com/os-autoinst/os-autoinst-distri-example[os-autoinst-distri-example] for an example of a minimal test distribution. Example for openSUSE's tests: ----------------------------------------------------------------------------- mkdir distri && cd distri git clone git@github.com:os-autoinst/os-autoinst-distri-opensuse.git opensuse cd opensuse/products/opensuse git clone git@github.com:os-autoinst/os-autoinst-needles-opensuse.git needles ----------------------------------------------------------------------------- Example for openQA's self-tests ("openQA-in-openQA" test): ----------------------------------------------------------------------------- mkdir distri && cd distri git git@github.com:os-autoinst/os-autoinst-distri-openQA.git openqa cd openqa git git@github.com:os-autoinst/os-autoinst-needles-openQA.git needles ----------------------------------------------------------------------------- Then create a working directory for the test execution, e.g.: ----------------------------------------------------- mkdir /tmp/os-autoinst-run && cd /tmp/os-autoinst-run ----------------------------------------------------- Create a minimal `vars.json` config file within that directory, e.g.: .vars.json [source,json] --------------------------------------------------------- { "ARCH" : "x86_64", "BACKEND" : "qemu", "CASEDIR" : "/path/to/os-autoinst-distri-opensuse", "DESKTOP" : "gnome", "DISTRI" : "opensuse", "ISO" : "/path/to/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20160715-Media.iso", "PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/opensuse", "VNC" : 90, } --------------------------------------------------------- You will need to correct the file paths to point to real locations. Some of the variables you can use are listed link:doc/backend_vars.asciidoc[here]. Test case specific variables are listed in the distri directories e.g. link:https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/variables.md[os-autoinst-distri-opensuse/variables]. Then you can run the `isotovideo` script within the created working directory. When doing a manual build, that script can be found at the top-level of the os-autoinst Git checkout. When using the QEMU backend it is possible to access the system under test via VNC: ---------------------------------------- vncviewer localhost:91 -ViewOnly -Shared ---------------------------------------- Run `isotovideo` with the environment variable `RUN_VNCVIEWER` set to autostart a VNC viewer on the right port. Run `isotovideo` with the environment variable `RUN_DEBUGVIEWER` to start the internal debug screenshot viewer updated with an always recent screenshot of the test run. Further notes ------------- When using the QEMU backend, also ensure your user running os-autoinst has access to `/dev/kvm`. ----------------------------------------------------------- modprobe kvm-intel || modprobe kvm-amd chgrp kvm /dev/kvm ; chmod g+rw /dev/kvm # maybe redundant # optionally use a new user; just to keep things separate useradd -m USERNAME -G kvm passwd USERNAME # and/or add ~USERNAME/.ssh/authorized_keys -----------------------------------------------------------