parshakova/apsn

APSN

Latent Question Interpretation Through Parameter Adaptation Using Stochastic Neuron

In this work we propose a training framework and the APSN module for learning question interpretations that help to find various valid answers within the same document.

_{Structure overview of integrated APSN module with DrQA. In this illustration number of interpretations is 2 and a sampled interpretation is 1.}

	_{Sample answers (Model) produced by inducing the value of a latent interpretation given D, Q}
`D`	_{ITV Tyne Tees was based at City Road for over 40 years after its launch in January 1959. In 2005 it moved to a new facility on The Watermark business park next to the MetroCentre in Gateshead.}
`Q`	_{Where did ITV Tyne Tees move in 2005?}
`A`	_{[’a new facility’]}
`Model`	_{['The Watermark business park', 'Gateshead']}

`D`	_{Research shows that student motivation and attitudes towards school are closely linked to student-teacher relationships. Enthusiastic teachers are particularly good at creating beneficial relations with their students.}
`Q`	_{What type of relationships do enthusiastic teachers cause?}
`A`	_{[’beneficial’]}
`Model`	_{['student-teacher', 'beneficial relations']}

`D`	_{For Luther, also Christ’s life, when understood as an example, is nothing more than an illustration of the Ten Commandments, which a Christian should follow in his or her vocations on a daily basis.}
`Q`	_{What should a Christian follow in his life?}
`A`	_{[’Ten Commandments’]}
`Model`	_{['Ten Commandment', 'vocations on a daily basis']}

`D`	_{It is believed that the civilization was later devastated by the spread of diseases from Europe, such as smallpox.}
`Q`	_{What was believed to be the cause of devastation to the civilization?}
`A`	_{[’spread of diseases from Europe’]}
`Model`	_{['smallpox', 'spread of diseases from Europe']}

Tested on GeForce GTX Titan X.

Setup

GPU and CUDA 8 are required

Install

Python >= 3.5
PyTorch
CuPy, pynvrtc
Spacy 1.10.1
Cython, Pandas, NumPy, Scikit-learn
msgpack, tensorboardX, Matplotlib

Download

the SQuAD
GloVe word vectors
Sent2vec with wiki_bigram.bin
Use checkpoint init1 to initialize models for training

using bash download.sh

Preprocess Dataset

# prepare the data
python prepro.py

# prepare semi-supervised labels for question-answer pairs
python semisup_labels.py

# make sure CUDA lib path can be found, e.g.:
export LD_LIBRARY_PATH=/usr/local/cuda/lib64

Running Code

# training in semi-supervised VAE framework
python train.py -e 240 -bs 50 -rd init1 -rs best_model.pt -lr  0.0005 --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --critic_loss --n_actions 5 --vae --semisup --rl_start 80 --rl_tuning pg

# sample answers with induced values for interpretations
python interpret.py -bs 32 -rd m1_80.9 -rs best_model.pt --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --n_actions 5 --vae

# visualize document encodings with interpretation-adapted parameters
python tsne.py -bs 1 -rd m1_80.9 -rs best_model.pt --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --n_actions 5 --vae

_{Indices represent different documents, colors correspond to induced interpretations (interpretation marked with cross was chosen by the policy during testing)}

Credits

Author of the Document Reader model: @danqi

Pytorch implementation of DrQA with SRU cells @taolei87

parshakova / apsn