INCF / xnnppx

XNAT-NiPype-pyxnat

See file COPYING distributed with the xnnppx package for the copyright
and license.

Xnnppx is a package for launching Python scripts from XNAT
(www.xnat.org).

CONTENTS

    README -- This file.

    COPYING -- Copyright and license.

    XnatPipelineLauncher -- The drop-in replacement script for
    XnatPipelineLauncher that shunts execution to a Python script,
    if available.

    XnatPythonLauncher -- The Python script launcher.

    xnnppx.py -- The XNAT utility module for the Python script.

    create_workflow_xml -- A utility program for converting a Python
    or YAML dictionary into an XML pipeline descriptor for XNAT.

    sample.info -- Example Python dictionary for generating an XML
    pipeline descriptor.

    sample.py -- Example Python pipeline script.

INSTALLATION

Xnnppx requires:

    Suds: https://fedorahosted.org/suds/

    pyxnat: http://packages.python.org/pyxnat/

In $PIPELINE_HOME (typically $XNAT_HOME/pipeline), move
bin/XnatPipelineLauncher to bin/XnatPipelineLauncher.orig and replace
it with XnatPipelineLauncher from this distribution.  Change
pipeline_home in this (new) XnatPipelineLauncher and make sure this
file has executable permissions.

    This step steals execution from XNAT and shunts it to a Python
    script if it is available.  Otherwise it passes control back
    to XNAT's native pipeline launcher.

Install XnatPythonLauncher in $PIPELINE_HOME/bin with executable
permissions.  Change the value for the pipeline config file name 
in the default arguments in the definitions and defaults section.

    If XnatPipelineLauncher finds a Python script that needs to be
    run, it calls this script, which takes the parameters passed
    by XNAT and packages them in a form easily used by the Python
    script.

Install xnnppx.py in your Python path.

    This is the module that the Python script will use to access
    the XNAT parameters and update the workflow information.

CREATING A PYTHON PIPELINE

Create a Python script with extension .py in the same directory as
the XML pipeline descriptor (typically in $PIPELINE_HOME/catalog/.../).
XnatPipelineLauncher detects this script and launches it with
XnatPythonLauncher instead of the native XNAT pipeline launcher.

See sample.* for an example.  Install sample.py in a subdirectory
of $PIPELINE_HOME/catalog, for instance
$PIPELINE_HOME/catalog/local/sample.py.  XNAT will need an XML
descriptor of the pipeline in order to launch it; this can be created
by describing the pipeline as a Python or YAML dictionary (see
sample.info) and converting it with create_workflow_xml.  Put the
result in $PIPELINE_HOME/catalog/local/sample.xml.

The Python script can access the pipeline arguments and parameters
in the dictionaries xnnppx.parameters and xnnppx.arguments.  Because
any parameters may have more than one value, the values in
xnnppx.parameters are all lists of parameter values.

XnatPythonLauncher will redirect output to a log file (typically
in $PIPELINE_HOME/logs), update XNAT (more specifically, the workflow
XML) with the workflow execution environment and pass control to
the Python script.  Once the script has started, it is then responsible
for updating the workflow XML, which can be done by calling:

    xnnppx.workflow_info.update(step_id, 
                                step_description,
                                percent_complete)

    xnnppx.workflow_info.complete()

    xnnppx.workflow_info.fail([step_description])

xnnppx.ContextManager may be used to automatically mark the workflow
XML complete using xnnppx.workflow_info.complete() or
xnnppx.workflow_info.fail() as appropriate and send a success or
failure e-mail as specified by the notify_flag (-supressNotification)
and notify_emails (-notify) arguments.  See sample.py for an example.

xnnppx.send_mail(to_addrs, subject, body) can be used to send mail.

LAUNCHING A PIPELINE

For XNAT to manage a pipeline, it must first be added to XNAT, and
then to a project.  As a site administrator, go to Administer ->
Pipelines, then select Add Pipeline to Repository.  Give the full
path to the XML pipeline descriptor
($PIPELINE_HOME/catalog/.../sample.xml); you may leave the second
line blank.  Then the pipeline needs to be added to each project
it will apply to.  As a site or project administrator, go to the
project, then select Pipelines, Add More Pipelines, and Add for the
pipeline you want to add.  No changes are needed to this form, but
you may choose to select the pipeline for automatic running when
data is archived to the project.

There are (at least) four ways of launching an XNAT pipeline:
automatically on archiving, through the web interface, using a REST
call, and from the XNAT server command line.

If you choose for the pipeline to be automatically launched on data
archive (see above), it will be launched automatically after the
Transfer and AutoRun pipelines have completed.

To launch a pipeline from the XNAT web interface, you will first
need to create the launching mechanism.  As a site administrator,
go to Administer -> Data Types -> xnat:mrSessionData (or whatever
data type you want to work on) -> Edit, then under Available Report
Actions, set:

    Name:                  PipelineScreen_launch_pipeline
    Display Name:          Launch Pipeline
    Grouping:
    Image:                 wrench.gif
    Popup:                 sometimes
    Secure Access:         edit
    Additional Parameters:
    Sequence:              16

Now the MR Session pages will have a Launch Pipeline option in the
Actions menu; click on this and follow the prompts to start the
pipeline.

See also http://xnatdev.wikispaces.com/Pipeline+Engine.

To start a pipeline using a REST call, POST the XML for the pipeline
to be launched to
/data/archive/projects/{PROJECT}/pipelines/{NAME}/experiments/{EXPERIMENT_ID}.
The pipeline names for a project can be found using a GET to
/data/archive/projects/{PROJECT}/pipelines, and a GET to the POST
location above will give template XML that can then be edited and
used for the POST.  For example:

    GET /data/archive/projects/mystudy

will give us the pipelines available for the project "mystudy."  If
we find one called "sample," we can then:

    GET /data/archive/projects/mystudy/pipelines/sample/experiments/myxnat_E00001

to get template XML for this pipeline, and (after modifying the
parameters in the XML as needed), send it back with a POST to launch
the pipeline:

    POST /data/archive/projects/mystudy/pipelines/sample/experiments/myxnat_E00001

If you have access to the XNAT server, you can launch pipelines
from the command line.  XnatPipelineLauncher can be called directly;
this is useful if a pipeline fails and the full command line is
available in the logs.  Similarly, XnatPythonLauncher can be called
directly if the full command line is known.  XnatPythonLauncher
sends all its output to a log file by default, but if XNAT_PYTHON_NOLOG
is defined, output is not redirected and will be displayed at the
command line.