NLPCC 2022 Shared Task 3 - Multimodal Product Summarization Demo
m-kplug is an extension of the pre-trained model k-plug, which simply injects the visual signals to the decoder layer.
Dependency
pip install torch==1.6.0 torchvision==0.7.0
pip install psutil transformers==4.12.5
cd fairseq-mirror
pip install -e ./Command
- train
cd m_kplug
bash run_train.sh
- predict
cd m_kplug
bash run_predict.sh
- eval(rouge-1/2/L)
cd m_kplug
bash run_eval.sh
Data Process
raw text preprocessing
user_tag=liweikang
cd m_kplug/raw_data_process
python get_data_raw_from_json.py /data/${user_tag}/jdsum/home_appliances/home_appliances_dev_new.json /data/${user_tag}/jdsum/home_appliances/raw_demo/ valid
image vector extracting
user_tag=liweikang
cd m_kplug/data_process
python extract_patch_vector.py /data/${user_tag}/jdsum/home_appliances/raw_demo/valid.sku /data/${user_tag}/jdsum/home_appliances/img /data/${user_tag}/jdsum/home_appliances/raw_demo/ valid
text bpe and bin processing
cd m_kplug
bash preprocess_pipline.sh