Artistic Sentence

Incheon National University - Computer Science Capstone Design(Graduation product)

You & AI

We make an application that generates picture through text. In addition, not only one sentence but also several sentences are summarized to create a picture that fits the atmosphere of the text. Finally, the created picture is converted into the style of the artist you want, and recommends according to the user

Member (Team : You & AI)

Enviornment

OS : Ubuntu 20.04-server
GPU : A4000 x 2

How to do (AI)?

Text2Image

1. Enviornment setting

git clone https://https://github.com/winston1214/Text2Drawing-Service
cd Artistic-Sentence/AI
git clone https://github.com/openai/CLIP # CLIP
git clone https://github.com/CompVis/taming-transformers # VQGAN

pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install torchvision==0.10
pip install ftfy regex tqdm omegaconf pytorch-lightning IPython kornia imageio imageio-ffmpeg einops torch_optimizer

2. Download pretrained model

mkdir checkpoints

curl -L -o checkpoints/vqgan_imagenet_f16_16384.yaml -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fconfigs%2Fmodel.yaml&dl=1' #ImageNet 16384
curl -L -o checkpoints/vqgan_imagenet_f16_16384.ckpt -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fckpts%2Flast.ckpt&dl=1' #ImageNet 16384

if you use other model, you check pretrained_model.sh

3. Inference

python generate.py -p "A painting of an apple in a fruit bowl"

Demo

_._._.mp4

Etc Matarials

Poster
- https://github.com/winston1214/Artistic-Sentence/blob/master/picture/Poster.jpg
- http://www.ideaboom.net/project/project/view?seq=1101
Full Presentation : https://www.youtube.com/watch?v=TBCMjT09w0Y
Presentation Slide : https://www.slideshare.net/ssuserd555d6/artistic-sentence

Reference

https://github.com/nerdyrodent/VQGAN-CLIP

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021.

Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

winston1214 / Artistic-Sentence