DESCRIPTION OF BACKEND DATA
Written by Eric Heien on Apr 15, 2013
Updated May 20, 2014
This directory contains scripts and data related to the backend management of
the CIG website. This includes things such as download hit records and map
generation, bibliography list data and generation, automated documentation
generation, and automated build testing for the CIG codes.
Each directory contains a README file describing the contents and function in
detail. Here is a brief overview of the directories:
batlab/
Contains scripts, support files, and libraries to perform build and test
operations on CIG codes using the BaTLab system.
Relevant URL as of May 20, 2014
http://submit-1.batlab.org/cig/
c5/
Contains PHP, Javascript, etc scripts related to the Concrete5 website.
Connected to Concrete5 website folder through symbolic links.
Relevant URL as of May 20, 2014
http://geodynamics.org/cig/admin/
http://geodynamics.org/cig/
doxygen/
Contains scripts to generate doxygen based documentation of the different
CIG codes based either on released source or a repository checkout.
Relevant URL as of May 20, 2014
http://geodynamics.org/cig/doxygen/
git/
Contains scripts and submodules related to the SVN to Git conversion in
2013/2014, automated diff email for Git commits, and Github manipulation
scripts.
map/
Contains databases, scripts and related files for tracking downloads of
software distributions from the website, and creating maps to display
downloads. Relevant URL as of May 20, 2014
http://geodynamics.org/cig/maps/
queue/
Contains script to allow queueing of commands. This is primarily used for
the doxygen documentation where we want to avoid possible overwriting and
misordering.
ref_list/
Information regarding online publication list, database, and editor.
Relevant URL as of May 20, 2014
http://geodynamics.org/cig/news/publications/
Directories which may exist:
queues/
Contains queue related files including commands (formed as SUBMISSIONTIME_PID)
and lockfiles.
repos/
Contains checked out versions of the development source. Since checking out
the entire code may take a long time, the doxygen/generate_doxygen.sh
script checks out the code once and updates it to the current revision.
This drastically improves speeds and avoids wasting bandwidth.
NOTES
To ensure necessary processes are run periodically, several entries were
installed into crontab. The commands are run from a specific directory to
ensure relative file access is correct, particularly to cig_codes.py. These are
described below:
0 1 * * * cd /home/backend/cig_backend/map/hit_database ; /home/backend/cig_backend/map/hit_database/parse_log.py /home/backend/cig_backend/map/hit_database/hit_database "/var/log/from_geo/access.log*"
Run the log analysis script nightly and adds the results to the SQLite3
database in map/hit_database/hit_database. The logs analyzed are in
/var/log/from_geo/access.log*. Note that this directory is remotely mounted and
the script will fail if the directory isn't mounted at execution time.
0 2 * * * cd /home/backend/cig_backend/map/map_plot ; /home/backend/cig_backend/map/map_plot/plot.py /home/backend/cig_backend/map/hit_database/hit_database /home/backend/cig_backend/map/ip_database/ip_lookup_db /home/backend/public_html/maps/ all `date --date="1 year ago" +'\%s'`
Nightly generates maps based on the IP lookup database and the hits garnered
from the logs. This generates maps for all codes based on all hits in the past
year. Basic IP filtering is available in the plot.py script.
0 4 * * 1 cd /home/backend/cig_backend/doxygen/ ; ./queue_doxygen.py all release
0 5 * * * cd /home/backend/cig_backend/doxygen/ ; ./queue_doxygen.py all dev
Periodically regenerate the doxygen documentation for code releases (every
Monday) and development repositories (every night). Note that these scripts do
not actually generate the doxygen, but rather queue up commands for it to be
generated. Errors from these commands will be put in the file
queues/backend_queue/queue.log.
0 6 * * * cd /home/backend/cig_backend/git/gitdub ; PATH=$PATH:/home/backend/cig_backend/git/git-notifier ./gitdub ../config.yml >> /home/backend/cig_backend/git/git_backend.log 2>&1
Nightly attempt to start the gitdub server to receive hook notifications from
Github about new commits. If the server is already running this simply aborts
and leaves the previous process running.
0 3 * * * cd /home/backend/cig_backend/git ; ./setup_github.py OAUTH_TOKEN trigger >> /home/backend/cig_backend/git/trigger.log 2>&1
Nightly trigger all Github hooks to ensure we didn't miss any commits in the
past day to send emails for. The OAUTH_TOKEN is generated and stored on the
CIG Github account "geodynamics".
* * * * * cd /home/backend/cig_backend ; python -c "import cig_codes; codes = ['geodynamics/'+code for code in cig_codes.list_cig_codes()]; codes.append('eheien/cig_backend'); print('\n'.join(codes));" > /home/backend/repo_list.txt
Creates a list of current CIG code repositories (along with the backend repository).
This is used to do local backups of all repositories.