GSA CKAN 2.3 in Virtual Machine via Vagrant

The GSA/CKAN 2.3 instance installed in a Ubuntu/Trusty64 VirtualBox managed by Vagrant. The documentation is at CKAN; however, it is not entirely complete. The CKAN 2.3.5 Setup Process follows. This process does not provision via Ansible, but installs the software directly in the virtual box manually.

Host System Pre-requisites

Before a virtual system using Vagrant can be created, the system must be prepared. If the host does not already have the following items installed, installation is required:

Download the host system appropriate package from Oracle VirtualBox and install it, following the instructions closely. Packages exist for Windows, macOS, Linux, and Solaris.
Then download and install Vagrant.
Then install two Vagrant plugins, listed below
The Vagrant VirtualBox Guest: vagrant plugin install vagrant-vbguest.
The Vagrant Hostmanager: vagrant plugin install vagrant-hostmanager.

Now the host system should be ready to procede.

Build the Vagrant file -- Vagrantfile

Two methods exist for this arrangement, the Quick and the Dirty. The Quick is covered first, the Dirty afterwards as it is more complicated.

The Quick

This method requires changing only the name of the preferred box in the Vagrant file, but the box must be a pre-configured for a CKAN stage: Development Testing (CKAN23D), Development Tools Installed (CKAN23T), Plugin and Extension Testing (CKANP23E), or Production Testing (CKAN23P).

If using a pre-created CKAN-prepared Vagrant Box, use the included Vagrantfile, but edit line 6, replacing config.vm.box = "ubuntu/trusty64" with config.vm.box = "Name of box needed" where Name of box needed matches one of the boxed noted above.

Save the file and type vagrant up. Once the box loads, the virtual machine should be ready to go!

The Dirty

Why the Dirty? Because a box must be built out from a basic Vagrant box to one that holds the glory that is CKAN. The following process details what must be done to ready a box. This process assumes the Vagrant box in use is Ubuntu/Trusty64, which pulls from Vagrant Box Search directly.

Differences between CKAN Development and Production

Differences exist between Development, Testing, and Production, as shown in the table below. Plugins and extensions are not included in the table because they are not part of system requirements. They will be detailed later.

Environment Requirements

Server/Software	Development	Testing	Production
Server	Ubuntu 14.04 LTS (Trusty) - 64 bit	Same	Same
Language	Python v 2.7.6	Same	Same
Java Runtime	Java v 1.7.0_171; OpenJDK Runtime Env (IcedTea 2.6.13) (7u171-2.6.13-0ubuntu0.14.04.2)	Same	Same
Web Server	Paster	Paster	Apache 2.4.7
Java Server	Eclipse Jetty v 6	Eclipse Jetty 6	Apache Tomcat 6
Search Server	Solr v 4.2.1	Same	Same
NoSQL Server	Redis v 2.8.4	Same	Same
SQL Server	Postgres SQL v 9.3	Same	Same
Data Management System	CKAN 2.3	Same	Same
Initiation File	development.ini	UNKNOWN NOW	production.ini

Please review the general CKAN 2.3 information before proceeding.

Preparing Trusty for CKAN

Prior to installing CKAN and Solr, the box must be prepared because the Ubuntu/Trusty64 Vagrant box pulled from the Vagrant repository if essentially empty. Consequently a few appetizers need to be circulated before the main course is served.

At the host machine command line (or at an IDE's command line if using one), run vagrant up using the provided Vagrantfile. And wait.
Vagrant will create a .vagrant directory and fill it will a few some files.
Watch the terminal scroll. If you see a message the indicates the box needs updating, move to Step 4, otherwise jump to step 7.
After the machine has launched fully the first time, from the command line shut it down by typing vagrant halt.
Run the box update command, vagrant box update. The update function will run, updating the box to its most recent version.
Rerun vagrant up, which will restart/recreate the updated box.
Once the box instance's GUI window opens, log into the box. Use the vagrant account. Its login and password is the same as the account name. Do not upgrade the box to Ubuntu 16.04. At this initial stage, this box has a command line interface only. That will change later.
Stop the box again with vagrant halt.

Use the Virtual Box GUI to Modify Box Settings

The settings are specialized for this project. A video was created to show what the settings need to be. If creating a box from scratch, please as eKuber Ventures for access to the video.

Once the settings are adjusted, bring the box up again.

Begin Installation of Packages Required by CKAN

Verify that Python 2.7.6 is installed in the default box by typing: python --version. It should return Python 2.7.6, which is the required version and the version that should be installed by default.
If Python is not installed, run sudo apt-get install python2.7-minimal==2.7.6-8ubuntu0.2.
Now look at the Install CKAN 2.3 from Source information. Read it carefully. It must be installed from source because the source is actually GSA's CKAN fork, NOT from the official CKAN repository itself.
Complete Step 1 in the instructions: Install the required packages. When finished, the following should be installed: python-dev, postgresql, libpq-dev, python-pip, python-virtualenv, git-core, solr-jetty, and openjdk-6-jdk. Please notice that the Java JDK is the wrong version. It will be modify later. Do not modify it now, it will break the installation process.

Installation of Ubuntu Desktop GUI

The next step, before continuing with the CKAN process, is the installation of the Ubuntu Desktop GUI, which is necessary for verifying both CKAN and Solr.

Run sudo apt-get update to retrieve current packages for update.
Run sudo apt-get install ubuntu-desktop to begin installation of the desktop. Ubuntu will download a list of packages needed.
When the list is complete, the system will request confirmation of installation request. Accept it by typing Y.
The required packages will install.
Once complete, take the system down again with vagrant halt.
Restart it with vagrant up. The system will load, the new GUI will come up.
Once loaded, select Vagrant as user. Enter the same password as one would at the command line.

Customize the Ubuntu GUI

Firstly, fix the GUI's default root user issues. The GUI's default user is Ubuntu and it has a blank password. That must be fixed to proceed.
Click the icon on the top left of the desktop's sidebar. The tooltip will read Search your computer and online sources. A search interface opens. Type terminal. Icons representing applications will show below the search line. Select Terminal. Terminal will launch.
At the Terminal prompt, type sudo passwd ubuntu. Hit enter. A request for a password will appear. Use the same password as used for the Vagrant account. Repeat a second time. The Ubuntu account will now have a password.
Add the Terminal to the sidebar (if desired) by right-clicking on the icon and selecting Lock to Launcher.
Now install the Chromium web browser (Chrome for Ubuntu). Click on the Ubuntu Software Center (Shopping Bag icon with the 'A'). In the search bar, type Chromium and double click the icon for installation.

Now we are ready to proceed.

Replace Java JDK 6 with 7

While it may seem odd, the replacement was delayed because swapping the JDKs prior to desktop installation seems to cause unexpected issues. It works better from within the GUI.

Read this article before proceeding.

Show which versions are installed: update-java-alternatives -l.
Run in Terminal apt-get -s remove openjdk-6-\* icedtea-6-\* to show dependencies and what will eventually be removed.
Then install the openJDK 7 package: sudo apt-get install openjdk-7-jdk. Openjdk-7-jdk details.
Next run: sudo apt-get remove openjdk-6-\* icedtea-6-\*, which will remove version 6.
Verify the removal of version 6 by showing which versions remain: update-java-alternatives -l.
Finally verify that the new version is available for use: java --version.

Java is now the version required for the Production server.

Install the Virtual Box Guest Additions within the Ubuntu Desktop

The Virtual Box Guest Additions are important. They allow copying and drag and drop between the virtual box and the host desktop. Follow these instructions to setup the guest extensions for your particular host system:

MacOS Note: Make certain the VirtualBox software is up-to-date before proceding.
Windows/Linux

Once installation is finished, halt the virtual box and bring it back up. Once restored, the additions should be functional.

Return to the CKAN Setup instructions

In Terminal, resume installation at Step 2: Install CKAN into a Python virtual environment. Continue until Step 5, Setup Solr, section 1.
In a slight modification of the instructions, open Chromium and view the Jetty homepage: http://localhost:8983/. If for some reason localhost fails, use the IP 127.0.0.1:8983. The Jetty page should load.
Attempting to view the Solr welcome page, however, will result in the error show below in the blockquote:
Why? Because the Java update broke the JSP connectivity. Read and follow the instructions to resolve the error.
Solr's welcome page should now load.
With the Solr error resolved, continue on to Step 2, section 2.

HTTP ERROR 500 Problem accessing /solr/index.jsp. Reason: JSP support not configured Powered by Jetty://

Install Finished

If CKAN and Solr are both working, Stage 1 is complete.

Running the development CKAN instance again after restarting the Virtual Box

Remember that the CKAN development instance runs within a Python virtual environment, which means that the environment must be activated and the server restarted before anything can be seen in the browser or done in either the browser or command line in the development instance.

The commands from within Terminal at the command prompt are as follows:

. /usr/lib/ckan/default/bin/activate
cd /usr/lib/ckan/default/src/ckan
paster serve /etc/ckan/default/development.ini

As a Backup, Export the VirtualBox

To create a backup of the VM environment, from within the Oracle VM VirtualBox Manager, click on the menubar and select File > Export Appliance. From there, carefully follow the screen instructions.

Recovering OR Associating a Vagrant Project with an Existing VirtualBox VM

Important!!! This process will only work if a copy of the needed VirtualBox VM exists. The box will exist as a file: ApplianceName.ova. If such exported applicance does not exist, this procedure will not work. Additionally, the recovery assumes a pre-existing .vagrant structure exists as shown below:

NOTE: The directory above named default may have a different name if the Vagrantfile has it set differently, e.g., config.vm.define "someothername".

This process works from the Host machine' command line:

Make certain the desired machine is visable in the Oracle VM VirtualBox Manager.
1. If not, import it: File > Import Appliance and follow the instructions.
2. If it is, but it is corrupted, remove it and then follow sub step 1 above.
3. If the machine is A-OK, but disassociated from the Vagrant file, continue.
At the command line, type VBoxManage list vms. A list of available machines will print to the screen.
Look for the name of the broken machine. To the right of the name, its ID number will show, e.g., "Trusty 64 - U14.04" {83e7e24f-2aa3-4fba-ba95-3585201c8f07}". The ID is the string between the braces.
Change directories. In Vagrant version 2.x and above, it is in the location shown above. At the prompt type: cd .vagrant/machines/default/virtualbox.
Run ls -alg to see files and their properties.
Using text editor of choice (VI, VIM, EMACS, NANO, whatever), open the action_provision file and set its content to: 1.5:{id}, replacing {id} with the id found in step 3. Save the file, close the editor, and return to the command line.
Using the editor again, open the id and replace id found therein with the id obtained in step 3. Save the file, close the editor, and return to the command line.
Run vagrant up, which will begin to bring up the machine. In all likelihood, the machine will suffer a SSH authentication failure. Let the process run its course. An error message will generate, declaring a SSH failure. Panic not. The command line will return. The VM itself, however, should load.
Still on the host machine, type: vagrant ssh. When the password is requested, enter the Vagrant password. Once accepted, the command line will change to the guest VM's prompt, vagrant@machinename.
In a browser on the host machine, obtain the insecure Vagrant public key by copying it to the clipboard.
Return to the command prompt. Type cd .ssh.
Now in the SSH directory, type ls and find the authorized_keys file.
Using an text editor, open the file and replace its content with the content copied from the clipboard. Save the file, close the editor, and return to the command line.
Exit the guest machine command lineby typing exit.
At the host command line again, type vagrant reload.
Vagrant should halt the machine, destroy it, and then recreate it. It will replace the insecure key. Once restarted, the machine should be up and running in full.

Vagrant Box Creation Process

First off, boxes are stowed as shown below at download:

OS	Path Location
macOS or Linux	~/.vagrant.d/boxes
Windows	C:/Users/USERNAME/.vagrant.d/boxes

Install Redis

Redis installation instructions

Redis Releases

Redis Server Commands

sudo service redis_6379 start

sudo service redis_6379 stop

Redis port: 6379 Redis configuration file path: /etc/redis/6379.conf Redis log file: /var/log/redis_6379.log Data directory: /var/lib/redis/6379 Executable path: /usr/local/bin/redis-server

CKAN Extensions

Harvest: Installed here -- /home/vagrant/ckan/lib/default/src/ckanext-harvest

SkipKeats / ckan_2.3_vagrant