The must-have resource for anyone who wants to experiment with and build on the OpenAI Vision API. This repository serves as a hub for innovative experiments, showcasing a variety of applications ranging from simple image classifications to advanced zero-shot learning models. It's a space for both beginners and experts to explore the capabilities of the Vision API, share their findings, and collaborate on pushing the boundaries of visual AI.
Experimenting with the OpenAI API requires an API 🔑. You can get one here.
- 100 API requests per single API key per day
- Can't be used for object detection or image segmentation out of the box, but can be used for both with Autodistill (i.e. using Autodistill's DINO-GPT4V model).
| experiment | complementary materials | authors |
|---|---|---|
| WebcamGPT - chat with video stream |   |
@SkalskiP |
| HotDogGPT - simple image classification application | |
@SkalskiP |
| zero-shot image classifier with GPT-4V | |
@capjamesg |
| zero-shot object detection with GroundingDINO + GPT-4V | |
@capjamesg |
| GPT-4V vs. CLIP | |
@capjamesg |
| GPT-4V with Set-of-Mark (SoM) | |
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao |
| GPT-4V on Web | |
@Jiayi-Pan |
| automated voiceover of NBA game |  |
@SkalskiP |
webcamgpt.mov
- How CLIP and GPT-4V Compare for Classification
- Experiments with GPT-4V for Object Detection
- Distilling GPT-4 for Classification with an API
- DINO-GPT4-V: Use GPT-4V in a Two-Stage Detection Model
- First Impressions with GPT-4V(ision)
- Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V by Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao
- The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) by Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, Lijuan Wang
- GPT-4 System Card by OpenAI
We would love your help in making this repository even better! Whether you want to add a new experiment or have any suggestions for improvement, feel free to open an issue or pull request.