- A Chinese Poetry generation system combining the power of Transfer Learning and Reinforcement Learning.
- One of works in 2018 International Collegiate Competition for Brain-inspired Computing Finale.
- The purpose of this project is to design a more interesting and brain-like poetry generation system (PGS). Other than previous PGS which requires users to input the topic and rhythm pattern of a poetry, the proposed PGS takes a full consideration of the poetic imagery beauty of Chinese traditional poetry and emulates the motivation arising process of a poet, users input a picture, then the PGS can 'write' a line of verse based on this picture. For the model, we propose some changes based on previous encoder-decoder structure to capture the complex mapping between vision and semantics.
- Xiaoice by MICROSOFT
- Jiuge(九歌) by THUNLP Group leaded by Prof. Maosong Sun
- SeqGAN by Lantao Yu et al.
- Ming Xu, Junbo Wang, Hao Darry, Qingzhan Wang, Chen Zhang
- Tensorflow r1.8.0
- PIL (Pillow)
- Python 3.6
- CUDA 9.0 (For GPU)
The model is composed of a discriminator and a CNN-MLP-RNN generator. CNN is pre-trained as a visual feature extractor. LSTM unit is applied in RNN, this is a decoder for visual information. In consideration of the complexity of Chinese poetry, we add MLP between CNN and RNN, and train it as a component of agent. Discriminator is a CNN, the gate is applied, and it can calculate the possibility of 'generated poetry is real'. The structure of the network is shown below
Samples generated by the system with ONLY Supervised Learning and with Reinforcement Learning
- Modify the parameters in test.py such as the directory of the input image, e.g.
DICO = 'tang.txt'
DICO_PKL = 'dict.pkl'
IMAGE = 'test/3.jpg'- Run command in command line
python test.py