CyberAgentAILab / OpenCOLE

OpenCOLE: Towards Reproducible Automatic Graphic Design Generation [Inoue+, CVPRW2024 (GDUG)]

OpenCOLE: Towards Reproducible Automatic Graphic Design Generation

Naoto Inoue*  Kento Masui*  Wataru Shimoda*  Kota Yamaguchi  (*: equal contribution)

CyberAgent

Workshop on Graphic Design Understanding and Generation (at CVPR2024)

Overview

🤔 Automatic generation of graphic designs has recently received considerable attention.

😦 However, the state-of-the-art approaches are complex and rely on proprietary datasets, which creates reproducibility barriers.

🔥 In this paper, we propose an open framework for automatic graphic design called OpenCOLE, where we build a modified version of the pioneering COLE [Jia+, arXiv'23] and train our model exclusively on publicly available datasets.

🚀 Based on GPT4V evaluations, our model shows promising performance comparable to the original COLE. We release the pipeline and training results to encourage open development.

Setup

Requirements

• poetry
• direnv

Install

poetry install

Dataset

OpenCOLE dataset (v1) is available at cyberagent/opencole in HuggingFace dataset hub.

Pre-trained models

• text_to_image: cyberagent/opencole-stable-diffusion-xl-base-1.0-finetune
• typography_lmm: cyberagent/opencole-typographylmm-llava-v1.5-7b-lora

Environment variables

Some part requires additional environment variables. We recommend to use direnv. Please copy the template in .envrc.example and modify it on your own.

cp .envrc.example .envrc

Inference

Please refer to inference.md.

Evaluation

We provide a script for GPT4V-based evaluation on generated images.

poetry run python -m opencole.evaluation.eval_gpt4v --input_dir <INPUT_DIR> --output_path <OUTPUT_PATH>

Training

Please refer to training.md.

Citation

If you find this code useful for your research, please cite our paper:

@inproceedings{inoue2024opencole,
  title={{OpenCOLE: Towards Reproducible Automatic Graphic Design Generation}},
  author={Naoto Inoue and Kento Masui and Wataru Shimoda and Kota Yamaguchi},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  year={2024},
}

Acknowledgement

This repository has been migrated from the internal repo. Despite the fact that commit logs are not visible, all the contributors have made significant contributions to the repository.

• @proboscis: OpenCOLE dataset construction
• @shimoda-uec: TypographyLMM
• @kyamagu: renderer
• @naoto0804: other (bunch of) stuffs