fcoury / Cursor-Hero

A desktop inspection experience

Cursor Hero

The plan: enhance LLM cooperation by leveraging Windows UI Automation.

Video demo

If you have a DualShock controller, you will need something like DS4Windows.

Integrations

• TeamDman/voice2text: A python program that types what you say if you're holding the hotkey (github.com)
• ollama/ollama: Get up and running with Llama 2, Mistral, and other large language models. (github.com)

Research Notes

Inference

• LM Studio

Fine Tuning

• allenai/OLMo: Modeling, training, eval, and inference code for OLMo (github.com)

Vision

• PTA-Text: A Text Only Click Model - Prompt image, it tells you where it would click (demo)
• Set-of-Mark Visual Prompting for GPT-4V
• LLaVA
• YOLOv9

RPA

• OpenAdaptAI/OpenAdapt: AI-First Process Automation with Large Multimodal Models (LMMs)

• askui/askui: "What can be said can be solved-"

• ddupont808/GPT-4V-Act: AI agent using GPT-4V(ision) capable of using a mouse/keyboard to interact with web UI

• TobiasNorlund/UI-Act: An AI agent for interacting with a computer using the graphical user interface

• KillianLucas/open-interpreter: A natural language interface for computers

• Accessibility tools - AccEvent (Accessible Event Watcher) - Win32 apps | Microsoft Learn

• Accessibility tools - Inspect - Win32 apps | Microsoft Learn

• Accessibility Insights

• Process Monitor - Sysinternals | Microsoft Learn

• Navigation events for WebView2 apps - Microsoft Edge Developer documentation | Microsoft Learn

• (1) Building 25+ years of SysInternals: Exploring ZoomIt | BRK200H - YouTube

• c# - Getting icon of "modern" Windows app from a desktop application? - Stack Overflow

RL

• stillonearth/bevy_rl
• Saving RenderTarget image data to a file #5603
• paulkre/bevy_image_export: Bevy plugin for rendering image sequences

Prompting

• guidance-ai/guidance: A guidance language for controlling large language models.
• Eladlev/AutoPrompt: A framework for prompt tuning using Intent-based Prompt Calibration (github.com)

RAG, Tools and Actions

• ACT-1: Transformer for Actions
• LlamaIndex 🦙 v0.10.6

Sandboxing

• copy/v86: x86 PC emulator and x86-to-wasm JIT, running in the browser (demo)

Speech to Text

• openai/whisper-large-v2: Hugging Face
• m-bain/whisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)
• SYSTRAN/faster-whisper: Faster Whisper transcription with CTranslate2
• collabora/WhisperLive: A nearly-live implementation of OpenAI's Whisper
• gaborvecsei/whisper-live-transcription: Live-Transcription (STT) with Whisper PoC (github.com)
• FL33TW00D/whisper-turbo: Cross-Platform, GPU Accelerated Whisper 🏎️ (github.com)

Multiplayer

• Veilid: open-source, peer-to-peer, mobile-first, networked application framework.

Impl

• beartype
• facebookresearch/torchdim: Named tensors with first-class dimensions for PyTorch
• Are we learning yet? A work-in-progress to catalog the state of machine learning in Rust
• PyO3/pyo3: Rust bindings for the Python interpreter

Assets

• Euphoric Heavy Regular
• Kenney Fonts
• Fira Mono - Google Fonts
• Xelu's FREE Controller Prompts (thoseawesomeguys.com)

Game inspiration

• The Last Clockwinder

Issues 👀

• Transparent example not working - Issue #2502 - rust-windowing/winit (github.com)