The pipeline for generating image from question that users are curious about.
The best scenario is Novel-well-known user makes pretty good prompt for generating model. But novel has lots of inofrmations and users can't remember all the information about novel.
So, user makes question thath users are curious about, and pipeline makes image which can representate answer about question.
- Novel: A Game of Thrones
- Quesiton: Where is hometown of Arya Stark?
- Answer: Winterfell
- Prompt: a beautiful painting of Winterfell by Amir Zand, Trending on artstation.
- Generated image
- Question about novel is put in to Qusetion-Answering(QA) model.
- QA model generates best answer.
- Post-processing answer for making pretty prompt.
- Prompt is put in Disco Diffusion and model generates image.
- DPR(Dense Passage Retriever) and ES(Elastic Search) try to find the most relevant top-k passage or page.
- Extraction-based Reader model try to find the answer about question.
CLIP guidance Diffusion model, pretrained by OpenAI.
Original Disco Diffusion: https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb
Diffusion model: https://github.com/openai/guided-diffusion