egillanton / ice-atis-nlu-evaluation

Method to measure precision, recall and f1-score at a token level for a NLU system

ICE-ATIS NLU Evaluation

Table of Contents

Click to expand

• 1. Introduciton
  • 1.1. Comparing NLU System Output and Golden Standard
    • 1.1.1. Tokens and Slot-Tags match
    • 1.1.2. System hypothesized a slot-tag
    • 1.1.3. System misses a slot-tag
    • 1.1.4. System assigns the wrong slot-tag
    • 1.1.5. System gets the boundaries of the tokens wrong
    • 1.1.6. System gets the boundaries and slot-tag wrong
• 2. Setup
  • 2.1. Create a virtual environment
  • 2.2. Activate virtual environment
  • 2.3. Install dependecies
  • 2.4. Set evironment variable
• 3. Results
• 4. Contributors

1. Introduciton

1.1. Comparing NLU System Output and Golden Standard

When comparing the golden standard annotations with the output of a NLU system different scenarios might occur:

1.1.1. Tokens and Slot-Tags match

1.1.2. System hypothesized a slot-tag

1.1.3. System misses a slot-tag

1.1.4. System assigns the wrong slot-tag

1.1.5. System gets the boundaries of the tokens wrong

1.1.6. System gets the boundaries and slot-tag wrong

2. Setup

Make sure you have python version 3.6 or higher.

Install python

2.1. Create a virtual environment

$ python3 -m venv venv

2.2. Activate virtual environment

$ . ./venv/Scripts/activate

2.3. Install dependecies

(venv)$ pip install -r requirements.txt

2.4. Set evironment variable

(venv)$ export PYTHONPATH=src

3. Results

4. Contributors

Become a contributor

🌟 PLEASE STAR THIS REPO IF YOU FOUND SOMETHING INTERESTING 🌟