English | 简体中文
- QANET_paddle
This project is based on the paddlepaddle_v2.2.0rc0 framework to reproduce QANET(ICLR 2018)链接 。
The current end-to-end machine reading and question answering models are mainly based on recurrent neural networks that include attention. Apart from the advantages, the main disadvantage of these models is that they are less efficient in training and inference. Therefore, the author proposed a question and answer architecture called QANet. This network does not need to use a recursive network. Its encoder is completely composed of convolution and self-attention. The convolutional network processes local information, and self-attention processes the global scope. information.
paper:
- [1] Yu, A. W. , D Dohan, Luong, M. T. , Zhao, R. , Chen, K. , & Norouzi, M. , et al. (2018). Qanet: combining local convolution with global self-attention for reading comprehension.
Reference project:
Project AI Studio address:
The model is trained + verified on the SQuAD1.1 dataset.
! ! ! Indicators and Results
Please refer to the specific steps of align: https://github.com/PaddlePaddle/models/blob/develop/docs/ThesisReproduction_CV.md
When aligning the model structure, there are generally 3 main steps:
- Network structure code conversion
- Weight conversion
- Model network correctness verification
Align result is as follows:
Align result is as follows:
Align result is as follows:
Original repo operation results(Tried several times, and the original author of the same configuration and parameters, can not run his effect):
Running results of this code:
weight file: https://aistudio.baidu.com/aistudio/datasetdetail/114931
- dataset size:
- train:87.5K
- dev:10.1K
- data format:TEXT,JSON
- Hardware:CPU、GPU(16G and above)
- Framework:
- PaddlePaddle >= 2.2.0rc0
- package:
- spacy
- ujson
# clone this repo
git clone git@github.com:27182812/QANet_paddle.git
cd QANet_paddle
Installation dependency
pip install -r requestments.txt
-
Create the "datasets/original" folder in the top-level directory, put glove.840B.300d.txt into the "Glove" folder under the secondary file, and place the SQuAD dataset into the "SQuAD" folder. As shown below:
-
First run, add
--processed_data
to preprocess the data, and put the processed data under the "datasets/original/processed" folder. It can be loaded directly in the next run.
- Put pre-training weights in the top-level directory(I Initialized pytorch weight and converted to pdparams format),here:https://aistudio.baidu.com/aistudio/datasetdetail/114636 。
- run
QANet_main.py
python QANet_main.py --batch_size 32 --epochs 60 --with_cuda --use_ema
├─data_loader # data load
├─datasets # dataset
├─imgs # illustrative picture
├─log_reprod # align files
├─model # model
├─trainer # train
├─util # utility function
│ README.md # English readme
│ README_CN.md # Chinese readme
│ requirements.txt # rely
│ QANet_main.py # run file
Parameters related to training and evaluation can be set in QANet_main.py
, as follows:
参数 | 默认值 | 说明 | 其他 |
---|---|---|---|
--batch_size | 32, Optional | Training batches | |
--epochs | 30, Optional | Training epochs | |
--with_cuda | False, Optional | Whether use GPu | 无GPU可不加 |
--use_ema | False, Optional | Whether use exponential moving average | |
--lr | 0.001,Optional | Learning rate |
See 5、Quick Start
For other information about the model, please refer to the following table:
information | description |
---|---|
Author | AshlingQian |
Date | 2021.10 |
Framework version | Paddle 2.2.0rc0 |
Application scenarios | NLP、MRC |
Support hardware | GPU、CPU |
Download link | trained model |
Online operation | notebook |