Andrechang / clip_draw_wave

Clip draw wave

A repo for making a AI-generated draw based on a sound/music file with Wav2CLIP and CLIP.

Repo code based on Clipdraw

The CLIP embedding for audio was from: Wav2CLIP

Differential rendering from clipdraw: diffvg

CLIP from: CLIP

Colab playground

Sample

A sample of a drawing created with this sound input birds singing

You can make one with your own sound/music too

Citations

@misc{unpublished2021clip,
    title  = {CLIP: Connecting Text and Images},
    author = {Alec Radford, Ilya Sutskever, Jong Wook Kim, Gretchen Krueger, Sandhini Agarwal},
    year   = {2021}
}

@misc{frans2021clipdraw,
      title={CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders}, 
      author={Kevin Frans and L. B. Soros and Olaf Witkowski},
      year={2021},
      eprint={2106.14843},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@article{wu2021wav2clip,
  title={Wav2CLIP: Learning Robust Audio Representations From CLIP},
  author={Wu, Ho-Hsiang and Seetharaman, Prem and Kumar, Kundan and Bello, Juan Pablo},
  journal={arXiv preprint arXiv:2110.11499},
  year={2021}
}