joshbickett / multimodal-gamer

A framework to enable multimodal models to play games on a computer.

Multimodal Gamer

A framework to enable multimodal models to play games on a computer.

Using the same inputs and outputs as a human operator, the model plays a game!

Key Features

• Compatibility: Designed for various multimodal models and games
• Integration: Currently integrated with GPT-4
• Supported Games:
  • Super Mario 64: https://www.smbgames.be/super-mario-64.php
  • Poker: https://www.247freepoker.com/
  • Coming soon: Chess.com

See MultiModal Gamer play Super Mario 64 on YouTube (click below)

Quick install

Create venv

python3 -m venv env

Activate it (Mac)

source env/bin/activate

Install requirements

pip install -r requirements.txt

Add OpenAI Key

export OPENAI_API_KEY=yourkeyhere

Run it

python main.py