Library used to generate batch scripts that start up web servers, VNC servers, and etc., through batch jobs running on HPC resources. It is also used to generate connection information from these batch jobs so that a user can connect to their batch job server.
Add this line to your application's Gemfile:
gem 'batch_connect'
And then execute:
$ bundle
Or install it yourself as:
$ gem install batch_connect
A simple batch script can be generated as such:
require 'batch_connect'
# Generate batch script object
my_script = BatchConnect::Script.new
# Render the script and output it to a file
File.open("/path/to/my/batch_script.sh", 'w') {|f| f.write(my_script.render)}
This generates a batch script with no PBS directives written in Bash at:
/path/to/my/batch_script.sh
You can take a quick look at it:
$ cd /path/to/...
$ cat batch_script.sh
Notice it calls within the script the following scripts: before.sh
,
script.sh
, after.sh
, and clean.sh
. These should all be located in the
current working directory of the batch script during the batch job submission.
file | description |
---|---|
before.sh |
this file is sourced before the main script is called so you can do any pre-processing such as setting connection information variables |
after.sh |
this file is sourced after the main script is called so you can do any post-processing such as setting connection information variables |
clean.sh |
this file is sourced during an exit of the batch script used to clean up anything the main script may leave behind |
script.sh |
this is the main script that starts and controls the web server of your choice |
You will need to set the connection information required for users to connect to this server:
variable | description |
---|---|
$host |
the host the server is running on (this is already set for you) |
$port |
the port the server is listening on (required) |
There are some Bash helper functions you can use in your sourced scripts defined in templates/scripts/_bash_helpers.mustache.
For example, to define an available port that will be used by the server:
# before.sh
# Use the `find_port` helper function to find available port
port=$(find_port)
Then we can use this port in our main script when setting up the server:
# script.sh
python -m SimpleHTTPServer $port
make this script executable:
$ chmod 755 script.sh
For this example we don't necessarily need after.sh
or clean.sh
. Now let's
start up our server:
$ qsub batch_script.sh -N my_batch -j oe -l nodes=1:ppn=12 -l walltime=01:00:00
123456
Wait until the job is started. When it is running take a look at the yaml file that is generated:
$ cat 123456.yml
host: 'node0001.hpc.edu'
port: '6584'
This is the yaml file that we will parse to get connection information for the user to connect to our server. Lets do a simple SSH tunnel connection to our new server:
require 'batch_connect'
# Create a connection view for this server
my_conn = BatchConnect::Connection.new(yml: "/path/to/123456.yml")
# Render this view object for a terminal making an SSH tunnel. We do need to
# give it some help though.
my_conn.render(:terminal, {local_port: 1234, ssh_user: "my_username", ssh_host: "login.hpc.edu"})
#=> "ssh -L 1234:node0001.hpc.edu:6584 my_username@login.hpc.edu\n"
Copy this into your local machine terminal:
$ ssh -L 1234:node0001.hpc.edu:6584 my_username@login.hpc.edu
Password: ....
...
Now open your browser and navigate to:
http://localhost:1234
You should see the directory listing where you started your batch job.
To start a VNC session you first need to create the batch script:
require 'batch_connect'
# Generate VNC batch script object
my_script = BatchConnect::Scripts::VNC.new
# Render the script and output it to a file
File.open("/path/to/my/batch_script.sh", 'w') {|f| f.write(my_script.render)}
This generates a batch script with no PBS directives written in Bash at:
/path/to/my/batch_script.sh
You should not need to create before.sh
, after.sh
, or clean.sh
. But you
will need to create the main script script.sh
. This script is called within
the VNC session itself, so this is where you launch your GUI applications.
Let's create a script that starts up a Gnome desktop for RHEL6, so modify
script.sh
as such:
#!/bin/bash -l
# Turn off screensaver
gconftool-2 --set -t boolean /apps/gnome-screensaver/idle_activation_enabled false
# Use browser window mode in nautilus
gconftool-2 --set -t boolean /apps/nautilus/preferences/always_use_browser true
# Remove any preconfigured monitors
if [ -f "${HOME}/.config/monitors.xml" ]; then
mv "${HOME}/.config/monitors.xml" "${HOME}/.config/monitors.xml.bak"
fi
# Export the module function for the Gnome session
export -f module
# Start up Gnome desktop
/etc/X11/xinit/Xsession gnome-session
Note: You need to add the shebang #!/bin/bash -l
at the top so that it
starts in login mode. This will initialize the module
function within the
script.
Now we make this script executable:
$ chmod 755 script.sh
Now let's start up our server:
$ qsub batch_script.sh -N my_batch -j oe -l nodes=1:ppn=12 -l walltime=01:00:00
123456
Wait until the job is started. When it is running take a look at the yaml file that is generated:
$ cat 123456.yml
host: 'node0001.hpc.edu'
port: '5901'
display: '1'
websocket: '6790'
password: '6SP7wldi'
spassword: 'rxsKg1dz'
This is the yaml file that we will parse to get connection information for the user to connect to our server. Lets do a websocket connection to the VNC server through our HPC center's OnDemand portal:
require 'batch_connect'
# Create a connection view for this VNC server
my_conn = BatchConnect::Connections::VNC.new(yml: "/path/to/123456.yml")
# Render the connection information for a websocket connection with out HPC
# center's OnDemand portal
my_conn.render(:novnc, {local_port: 1234, ssh_user: "my_username", ssh_host: "login.hpc.edu"})
#=> "/rnode/node0001.hpc.edu/6790/vnc_auto.html?password=6SP7wldi&path=rnode/node0001.hpc.edu/6790/websockify\n"
Now open your browser and navigate to your HPC center's OnDemand portal with the following sub-URI:
https://ondemand.hpc.edu/rnode/node0001.hpc.edu/6790/vnc_auto.html?password=6SP7wldi&path=rnode/node0001.hpc.edu/6790/websockify
and you should see a RHEL6 Gnome desktop accessed through the noVNC client.
You can specify different VNC options when you initialize the batch script object:
# Generate VNC batch script object that starts a VNC session with the name
# "my_vnc" at a resolution of 1024x768
my_script = BatchConnect::Scripts::VNC.new(name: "my_vnc", geometry: "1024x768")
Possible options that developers may be interested in:
option | description |
---|---|
:vnc_mod |
The VNC module to load that sets up the VNC server environment (default: turbovnc/2.0.91 ) |
:name |
Name of the VNC session (default: "vnc") |
:geometry |
Resolution of VNC session (default: "800x600") |
:dpi |
DPI of VNC session (default: "96") |
:fonts |
Comma delimited list of fonts to use in VNC session (default: "") |
:idle |
The idle timeout of VNC session in seconds (default: "0", session runs until walltime) |
- Fork it ( https://github.com/[my-github-username]/batch_connect/fork )
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request