juilin / InversePrompting

Code, Data and Demo for Paper: Controllable Generation from Pre-trained Language Models via Inverse Prompting

InversePrompting

Paper: Controllable Generation from Pre-trained Language Models via Inverse Prompting

Code: The code is provided in the "chinese_ip" and "english_ip" package.

Chinese Inverse Prompting:

based on https://github.com/THUDM/Chinese-Transformer-XL

Packages Required

torch,apex,boto3,sentencepiece,nltk,jsonlines,filelock,deepspeed,pypinyin,pandas

Train:

bash scripts/ds_pretrain_gpt2_29B.sh

Direct Generation:

bash scripts/generate_text.sh

Generate Poems:

python generate_pms_refined.py  --Inverse Prompting for TCP Generation

Generate QA:

python generate_qa_desc.py  --Inverse Prompting for QA

English Inverse Prompting:

based on megatron-lm https://github.com/NVIDIA/Megatron-LM, follow its guide to download model weights and put them under the correct path, then run

python tools/generate_samples_sgpu.py --use-set 1

for inverse prompting.

Data:

Chinese Language Model:

See https://github.com/THUDM/Chinese-Transformer-XL

English Language Model:

See https://github.com/NVIDIA/Megatron-LM

Generated TCPs:

jiuge:

data/poems_jiuge.jsonl

jiuge generated from http://jiuge.thunlp.org/

IP+RL:

data/poems_ip_rl.zip

IP-only:

data/poems_ip_norl.zip

Base Model:

data/poems_noip.zip

QAs:

CPM:

data/qa_cpm.zip

IP:

data/qa_ip.zip

base model:

data/qa_basemodel.zip

Human:

data/qa_human.jsonl

Human Evaluation Raw Data (results listed in paper):

based on evaluator:

data/user-records.jsonl

based on prompts: QA:

data/qa-records.jsonl

poem:

data/poem-records.jsonl

Paper: full version of paper(generated using XeLaTeX) is included in this repo. The arXiv version uses pdflatex and tables with Chinese characters are transferred to English as pdflatex does not allow UTF-8 characters(non-English languages) presence.

paper.pdf

There's also a demo where you can try your own questions/titles for QA/poem generation.

QA: https://pretrain.aminer.cn/app/qa

Poem Generation: https://pretrain.aminer.cn/apps/poetry.html

Note that the demo version is updating frequently and may be different from the repo version.

Some examples of poems it generates:

比特币

外挖无穷洞,机神犹未休。
卡中窥币影,池里验沙流。
屡载吸金主,孤深渍盗求。
方知区块链,本是古来游。

夜过虹桥机场 

卢浦斜晖里,西楼醉客行。
影侵双塔晚,灯落一城明。
空客还频顾,航灯未可惊。
空留城市夜,月映水帘星。

排队购房作 

向晚万人候,售楼幢馅齐。
验资堪买主,瞧室亦堪栖。
回柱瞻佳处,连楼仰远姿。
殷勤申买者,莫待扣扉期。

论资本主义 

若为自由故,如今逐利逃。
入城操法律,两股战空槽。
漂白藏珠玉,欢呼夺锦袍。
管窥矜势利,夸视堕尘劳。

赠美国友人

清远寄吴士,华州逢旧知。
大洋环万里,学馆阻三时。
道别殷勤意,地连海峤西。
同来艰运日,异域远风姿。

安克雷奇中美会谈

特务狂声振,朗官降虏庭。
普天皆窃笑,攻守几无惊。
入市商人拜,国殇将士迎。
会同诛狡寇,世界定清明。

If you have any questions, please contact [e-mail]

Please cite

@article{zou2021controllable,
  title={Controllable Generation from Pre-trained Language Models via Inverse Prompting},
  author={Zou, Xu and Yin, Da and Zhong, Qingyang and Yang, Hongxia and Yang, Zhilin and Tang, Jie}, 
  journal={arXiv preprint arXiv:2103.10685},  
  year={2021}  
}