llm_oracle

LLM Oracle is a GPT-4 powered tool for predicting future events. It's like a Magic 8 Ball that is able to perform basic research, calculations, and reasoning.

Demo Site

How Does This Work?

Prompting

At a high level this entire thing is just a wrapper for GPT-4.

You ask a question
We ask GPT Is {question} a valid question for a prediction market?, if not ERROR
We ask GPT

Will the answer to "{question}" be yes?
Today is {date}

You can use these tools: Calculator, Wolfram Alpha, Google, etc.

Respond with arguments for and against, then with a final prediction.

GPT enters a prompt chain (ReAct) to gather information to best answer the question
We parse the final prediction (like SOMEWHAT_LIKELY) into a probability between 0 and 100%.

Limitations

The outputs of the model are not expected to be calibrated (e.g. of the events it says 80%, is possible only 50% end up being correct)
The model is limited to what is obviously google-able to answer the question. It can not perform detailed research, build forecasting models, and run advanced multipart calculations.
The typical limitations of LLMs also apply like vulnerability to prompt injections in both the question and gathered internet text, hallucinations, bias, etc.

App Privacy

Here's a quick overview of what's saved when you use the demo app. In the event of a breach, this is also what could be obtained.

Stored "Forever"

The questions you ask (anyone with the prediction ID or anyone generally if "public" set in settings can view)
A unique anonymous user id associated with the questions (you can bypass with new browser, incognito, etc)
The GPT agent logs which may contain publically obtainable information about the question

Not Used / Stored

Social media accounts you share with

Python API

Install

pip install git+https://github.com/sshh12/llm_oracle
Environment

OPENAI_API_KEY=
SERPER_API_KEY= (required for tool agents)
SCRAPINGBEE_API_KEY= (required for tool agents)
WOLFRAM_ALPHA_APPID= (required for tool agents)
KALSHI_EMAIL= (required for kalshi API)
KALSHI_PASSWORD= (required for kalshi API)

DATABASE_URL= (required for demo app)

Example

from llm_oracle.markets.kalshi import KalshiMarket
from llm_oracle.markets.custom import CustomMarket, CustomEvent
from llm_oracle.markets.manifold import ManifoldMarket
from llm_oracle.agents.agent_basic import BasicAgentv1, BasicAgentv2, BasicAgentv3
from llm_oracle.agents.agent_tools import ToolAgentv1, ToolAgentv2, ToolAgentv3
import datetime
import os

manifold_market = ManifoldMarket()
kalshi_market = KalshiMarket(email=os.environ["KALSHI_EMAIL"], password=os.environ["KALSHI_PASSWORD"])

kalshi_event_ids = [
    "GTEMP-23-P1.02",
    "NPPC-24DEC31",
    "BIDENVNEBRASKA-24DEC31",
    "TIKTOKBAN-23DEC31",
    "SFFA-COMPLETE",
    "COIN-23DEC31",
    "HURCTOTMAJ-23DEC01-T3",
    "SCOTUSN-23",
    "MOON-25",
]

manifold_event_ids = [
    "will-lex-fridman-interview-ai-by-20",
    "will-biden-be-the-2024-democratic-n",
    "will-a-nuclear-weapon-be-detonated-b71e74f6a8e4",
]

custom_market = CustomMarket(
    [
        CustomEvent(
            "Will a humanity be replaced by AI by 2050?",
            datetime.datetime(2050, 1, 1),
        ),
        CustomEvent(
            "Will a random number that I pull from a uniform distribution [0, 100] be greater or equal to 99?",
            datetime.datetime(2025, 1, 1),
        ),
    ]
)


EVENTS = (
    [kalshi_market.get_event(kid) for kid in kalshi_event_ids]
    + [manifold_market.get_event(mid) for mid in manifold_event_ids]
    + custom_market.events
)

AGENTS = {
    "basic_v1": BasicAgentv1(),
    "basic_v2": BasicAgentv2(),
    "basic_v3": BasicAgentv3(),
    "tool_v1": ToolAgentv1(),
    "tool_v2": ToolAgentv2(),
    "tool_v3": ToolAgentv3(),
}


for event in EVENTS:
    if not event.is_active():
        continue
    title = event.get_title()
    event_uid = event.get_universal_id()
    market_p = event.get_market_probability()
    for agent_name, agent in AGENTS.items():
        p = agent.predict_event_probability(event)
        with open("predictions.tsv", "a") as f:
            f.write(f"{event_uid}\t{title}\t{market_p}\t{agent_name}\t{p}\n")

sshh12 / llm_oracle