donglixp / confidence

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Confidence Modeling for Neural Semantic Parsing



Install Python dependency

pip install -r requirements.txt

Install KenLM


Download data and pretrained model

Download the zip file from Google Drive, and copy it to the folder of code.


Precompute Confidence Metrics

Train a language model

./ ifttt
./ django

Precompute all the confidence metrics, and perform uncertainty backpropagation

(This step can be skipped. The precomputed data have been cached in the zip file.)

# arg2: gpu id
./ ifttt 0
./ django 0

The intermediate results are saved to "data_model/[ifttt|django]/*.eval".


Confidence Estimation

  • Train a confidence scoring model.
  • Compute spearman's rho between confidence scores and F1 for the full model and ablation models (w/o model uncertainty, w/o data uncertainty, and w/o input uncertainty).
./ ifttt
./ django

Uncertainty Interpretation

Evaluate confidence backpropagation and attention-based method against inferred ground truth.

./ ifttt
./ django


ezoic increase your site revenue



Language:Python 98.1%Language:Shell 1.9%