Issues in setting up demo for SQuAD 1.1 data
pragnakalpdev opened this issue · comments
Hello there,
I am facing issue in setting up this code. Here is what I did:
I have downloaded Pretrained Model by running this command : "gsutil cp -r gs://denspi/v1-0/model .", and then created the Custom Phrase Index for "dev-v1.1" by running below command:
python run_piqa.py --do_dump --filter_threshold -2 --save_dir SAVE3_DIR/ --load_dir ROOT_DIR/model --metadata_dir ROOT_DIR/bert --data_dir ROOT_DIR/data/dev-v1.1 --predict_file 0:2 --output_dir ROOT_DIR/your_dump/phrase --dump_file 0-1.hdf5
After that I am serving the API and run the Demo by using following command :
python run_piqa.py --do_serve --load_dir ROOT_DIR/model --metadata_dir ROOT_DIR/bert --do_load --parallel --port 8000
python open/run_demo.py ROOT_DIR/dump ROOT_DIR/wikipedia --api_port 8000 --port 3000 --index_name 64_flat_SQ8 --sparse_type p
But the demo is not working properly. I have tested the demo by providing the questions from SQUAD 1.1 Dataset but it's not giving proper answers. Instead of expected answers, it looks like it gives random answers.
I am not able to understand why it is not providing accurate answers. Is there something which I have missed or doing wrong?
Is it compulsory to train the model on our own or the pre-trained model provided at "gs://denspi/v1-0/model ." will work instead of training our own?
Hi,
I am facing a similar issue when I downloaded the Pretrained Model and then tried to eval on dev-v1.1 using: "python run_piqa.py --do_predict --output_dir tmp --do_load --load_dir model --predict_file dev-v1.1.json --do_eval --gt_file dev-v1.1.json --metadata_dir bert"
The predicted answer seems to be random span, resulting in a metric like: {"exact_match": 0.47303689687795647, "f1": 4.43806570152543}. 0.47% EM means something is totally wrong.
Did I do anything incorrectly? Thanks in advance.
Hi, the model at gs://denspi/v1-0/model should work fine. I think the model wasn't loaded properly somehow. Did you try using the full dump provided at gs://denspi/v1-0/dump and run the demo?
Hi,
I have not used the full dump provided at gs://denspi/v1-0/dump, I have created my own custom phrase index. Used the created custom phrase index with the model given at gs://denspi/v1-0/model. I am not getting the correct answers, answers seems to be very random.
Hi, i think it's related to #10. Have you tried giving --parallel option when making the dump?
Hello @jhyuklee Thanks for your comment. We are in the same team. So consider this fixed as well. 👍
Hi @mittalpatel, great! Thank you.