README for Termite, a topic model visualization tool. --------------- INFORMATION --------------- Termite is a visualization tool for inspecting the output of statistical topic models based on the techniques described in the following publication: Termite: Visualization Techniques for Assessing Textual Topic Models Jason Chuang, Christopher D. Manning, Jeffrey Heer Computer Science Dept, Stanford University http://vis.stanford.edu/papers/termite This tool is developed by: * Jason Chuang <jcchuang@cs.stanford.edu> * Ashley Jin <ashpjin@cs.stanford.edu> and is distributed under the BSD-3 license. ----------- LICENSE ----------- Copyright (c) 2013, Leland Stanford Junior University All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the <organization> nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ---------------------- ADDITIONAL CREDITS ---------------------- Termite requires the use of the following libraries and tools. We thank their respective authors for developing and distributing the tools. Mallet: Machine learning for language toolkit Project website: http://mallet.cs.umass.edu Developed by Andrew McCallum, et al. Distributed under a CPL license: lib/LICENSE-mallet STMT: Stanford topic modeling toolbox Project website: http://nlp.stanford.edu/software/tmt Developed by Daniel Ramage, et al. Distributed under a GNU license: lib/LICENSE-stmt D3 javascript visualization library Project website: http://d3js.org Developed by Mike Bostock, et al. Distributed under a BSD license: lib/LICENSE-d3 Google closure javascript compiler Project website: https://developers.google.com/closure/compiler/ Developed by Google engineers Distributed under an Apache license: lib/LICENSE-closure-compiler Backbone Project website: http://backbonejs.org Developed by Jeremy Ashkenas, DocumentCloud Inc. Distributed under a MIT license: lib/LICENSE-backbone Underscore Project website: http://underscorejs.org Developed by Jeremy Ashkenas, DocumentCloud Inc. Distributed under a MIT license: lib/LICENSE-underscore jQuery Project website: http://jquery.com Developed by the jQuery Foundation Distributed under a MIT license: lib/LICENSE-jquery html5slider Project website: https://github.com/fryn/html5slider Developed by Frank Yan Distributed under an MIT license: http://opensource.org/licenses/MIT ------------------ ONE-TIME SETUP ------------------ Run the setup script to fetch the following tools, libraries, and prepare all javascript files needed by Termite. This script only needs to be run once when Termite is first downloaded onto a new machine. >> ./setup.sh Libraries fetched include: * mallet * stmt * closure-compiler.js * d3.v3.js * jquery.js * underscore.js * backbone.js * html5slider.js The script creates a minified version of all javascript files. ----------------------- BUILD A TOPIC MODEL ----------------------- Building a topic model in Termite requires running a single python script. The script reads in an input text corpus, and produces an output folder whose content can be inspected using a web browser, described in the next section. Customize configuration file with the following information. A sample configuration file can be found in 'example.cfg' [Corpus] path to text corpus [TopicModel] directory for holding topic model outputs [TopicModel] number of topics to train [TopicModel] topic model (either mallet or stmt) [Termite] number of terms to seriate [Termite] path to save Termite-internal working files Process the text corpus, and build a topic model by running the execution script. Execution time will vary depending on the size of the corpus. >> ./execute.py <your_config_file.cfg> The execution script calls in order: 1. pipeline/tokenize.py Tokenize the text corpus 2. pipeline/train_mallet.py Train a topic model using MALLET pipeline/train_stmt.py or STMT. 3. pipeline/compute_saliency.py Compute term saliency 4. pipeline/compute_similarity.py Compute term similarity 5. pipeline/compute_seriation.py: Seriates terms 6. pipeline/prepare_data_for_client.py: Generates datafiles for client 7. pipeline/prepare_vis_for_client.py: Copies necessary scripts for client ---------------------------- VIEW TOPIC MODEL OUTPUTS ---------------------------- You are now ready to visualize the topic model outputs! Termite's output can be viewed in a web browser. To view the files locally (on your own computer), you need to set up a local web server. Alternatively, you may copy the output folder to a web server to publish the results. Termite outputs are stored in the 'public_html' subfolder within the output directory. To set up a local webserver: 1. Change into output directory (specified in the configuration file) >> cd <output_folder>/public_html 2. Start a local server using python >> ./web.sh 3. Open http://localhost:8888 in a modern web browser (Chrome, Safari, Firefox, or Opera) to view a visualization of the model outputs. To publish the results on a webserver: 1. Copy public_html directory to your remote server. ----------------------------- TOPIC MODEL VISUALIZATION ----------------------------- The visualization should consist of a matrix view with a frequency bar view to the right. The top right contains user controls, two slider bars and one input text box. User controls are explained in more detail in the 'VISUALIZATION USE CASES' section. When the mouse is placed over terms, topics, circles, or bars in the matrix or frequency bar view, the associated term, topic, bar, and circles will be highlighted in both views. --------------------------- VISUALIZATION USE CASES --------------------------- The Termite visualization tool has a set of user controls in the top right section of the webpage. The user may specify * the number of terms with highest affinity to show * the number of terms with highest saliency to show * always display specific terms * always display 20 most frequent terms belonging to selected topics * click on a topic to select/color the topic * mouse over a topic/term to highlight the topic/term (Note: the highest affinity term set and highest saliency term set may contain overlapping words)