----- This Version is deprecated, thanks to Khoa Nguyen a new version in Python 3 is available : https://github.com/nnkhoa/daniel_v3 ----- NOTES: - The core algorithm of Daniel still being coded in Python2.7, you will have to change the commands below if Python3.x is your default version - Using the pypy compiler usually makes Daniel 10 times faster (see https://pypy.org/download.html and choose the appropriate Python2.7 version) - On a standard laptop, in standard conditions for temperature and pressure, Daniel processes around 100 documents/second with the pypy2 compiler, around 10 with the Python2.7 compiler [daniel.py] For testing simple files Can be tested with the following command (-v to print results on stdout): python daniel.py -l LANGUAGE -d FILE -v Example : python daniel.py -l id -d some_document_in_indonesian.html -v [process_corpus.py] For processing a corpus : The command : python process_corpus.py -c JSON_FILE NB: Needs a JSON file (see below for the format) Example : python2.7 process_corpus.py -c docs/Indonesian_GL.json [evaluate.py] For evaluating results Compares the content of a groundtruth JSON file and an output from daniel python evaluate.py GROUNDTRUTH DANIEL_OUTPUT [The JSON format] A dictionnary where each key is the ID of a document The value is a dictionnary with informations on the document: - mandatory information : - file path - useful informations (by decreasing importance) : - source - language - url - comment -information for evaluation : - annotations See docs/Indonesian_GL.json for an example