A curated list of awesome publications and researchers on prompting framework updated and maintained by The Intelligent System Security (IS2).
Large language models (LLMs) have made significant advancements in both academia and industry, bringing about a fundamental engineering paradigm shift in many areas. While LLMs are powerful, it is also crucial to best use their power where "prompt"๐ง plays a core role. However, the booming LLMs themselves, including excellent APIs like ChatGPT, have several inherent limitations: 1) temporal lag of training data, and 2) the lack of physical capabilities to perform external actions.
Recently, we have observed the trend of utilizing prompt-based tools to better utilize the power of LLMs for downstream tasks, but a lack of systematic literature and standardized terminology, partly due to the rapid evolution of this field. Therefore, we survey related prompting tools and promote the concept of the ๐ "Prompting Framework" (PF) ๐, i.e. the framework for managing, simplifying, and facilitating interaction with large language models. We define the lifecycle of the PF as a hierarchical structure, from bottom to top, namely: Data Level, Base Level, Execute Level, and Service Level.
We also systematically depict the overall landscape of the emerging PF field and discuss potential future research and challenges. To continuously track the developments in this area, we maintain this repo, which can be a useful resource sharing platform for both academic and industry in this field.
Prompting Framework is a framework for managing, simplifying, and facilitating interaction with large language models, which adheres to four essential properties: modularity, abstraction, extensibility, and standardization. We define the lifecycle of the PF as a hierarchical structure, from bottom to top, namely: Data Level, Base Level, Execute Level, and Service Level.
๐ Paper Link: Prompting Frameworks for Large Language Models: A Survey
Looking forward to your attention and lively discussion๏ผ
Sincerely welcome:
๐โโ๏ธ Share good prompting frameworks with us!!!
๐โโ๏ธ Share your experience of using different prompting frameworks and your urgent needs and services!!!
๐ Brainstorm the future shape of prompting frameworks!!!
and more ...
๐ฎ Contact Details: liuxiaoxia@zju.edu.cn
๐ฌ ใ2023.11.21ใFirst version of "Prompting Frameworks for Large Language Models: A Survey" published on arXiv!
- ๐ Related Resource
- ๐ State-of-the-art Prompting Frameworks
- ๐ Comparative Analysis of Prompting Frameworks
- ๐ Key Components of Prompting Frameworks
Fingers crossed these awesome resources sprinkle some extra magic on your tasks too!
| ๐พ Tool | ๐ฑ Description |
|---|---|
| Trending Papers | A useful tool to help us researchers find and explore papers worth reading first. |
| Connected Papers | Explore academic papers in a visual graph. |
Taking into consideration the technical features, design objectives, and application scenarios, the current prompting framework can be broadly covered by three types: The Shell of LLMs (LLM-SH), Language for Interaction with LLMs (LLM-LNG), and Output Restrictors of LLMs (LLM-RSTR). The rationale behind designing the prompting framework is to facilitate the interaction between LLMs and the external world, and different types of prompting frameworks manifest this enhancement effect from different perspectives.
LLM-SH functions are much like a shell or interface layer in computer systems, emphasizing interaction with LLMs by facilitating their engagement with highly capable third parties, thereby enabling stronger interaction between LLMs, users, and external models.
-
๐ค Universal LLM-SH
๐ฆ Prompting Framework ๐งโโ๏ธ Subcategory ๐ Introduction Haystack Universal LLM-SH LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots. Semantic Kernel Universal LLM-SH Integrate cutting-edge LLM technology quickly and easily into your apps. Langchain Universal LLM-SH Building applications with LLMs through composability. Griptape Universal LLM-SH Modular Python framework for AI agents and workflows with chain-of-thought reasoning, tools, and memory. Griptape is an enterprise grade alternative to LangChain. PromptFlow Universal LLM-SH Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring. LLM-chain Universal LLM-SH A powerful rust crate for building chains in large language models allowing you to summarise text and complete complex tasks. LinGoose Universal LLM-SH LinGoose it's a Go framework for developing LLMs-based application using pipelines. LLMStack Universal LLM-SH No-code platform to build generative AI apps, chatbots and agents with your data. OpenDAN Universal LLM-SH OpenDAN is an open source Personal AI OS , which consolidates various AI modules in one place for your personal use. Hyv Universal LLM-SH Chaining AI & API agents to streamline software development and achieve goals collaboratively. -
๐ค Domain-Specific LLM-SH
๐ฆ Prompting Framework ๐งโโ๏ธ Subcategory ๐ Introduction LlamaIndex Domain-Specific LLM-SH LlamaIndex (formerly GPT Index) is a data framework for your LLM applications. Embedchain Domain-Specific LLM-SH Data platform for LLMs - Load, index, retrieve and sync any unstructured data. AgentVerse Domain-Specific LLM-SH AgentVerse is designed to facilitate the deployment of multiple LLM-based agents in various applications, which primarily provides two frameworks: task-solving and simulation. SuperAGI Domain-Specific LLM-SH A dev-first open source autonomous AI agent framework. Enabling developers to build, manage & run useful autonomous agents quickly and reliably. Txtai Domain-Specific LLM-SH All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows. AutoChain Domain-Specific LLM-SH Build lightweight, extensible, and testable LLM Agents. TermGPT Domain-Specific LLM-SH Giving LLMs like GPT-4 the ability to plan and execute terminal commands. Botpress Domain-Specific LLM-SH The open-source hub to build & deploy GPT/LLM Agents.
LLM-LNG, on the other hand, is designed to create a language (programming or pseudo-language) for interaction with LLMs, focusing on providing users with a more concise and compact interaction channel.
-
๐ค Programming LLM-LNG
๐ฆ Prompting Framework ๐งโโ๏ธ Subcategory ๐ Introduction LMQL Programming LLM-LNG A language for constraint-guided and efficient LLM programming. -
๐ค Pseudocode LLM-LNG
๐ฆ Prompting Framework ๐งโโ๏ธ Subcategory ๐ Introduction PromptLang Pseudocode LLM-LNG A Prompt based programming language for prompts and AI interactions. Simple and human-readable syntax for easy integration with APIs and data. SudoLang Pseudocode LLM-LNG SudoLang is a programming language designed to collaborate with AI language models including ChatGPT, Bing Chat, Anthropic Claude, and Google Bard. It is designed to be easy to learn and use. It is also very expressive and powerful. gpt-jargon Pseudocode LLM-LNG Jargon is a natural language programming language specified and executed by LLMs like GPT-4.
LLM-RSTR, meanwhile, achieves controlled generation by emphasizing interactions with LLMs that are of higher quality and better aligned with requirements.
-
๐ค Content LLM-RSTR
๐ฆ Prompting Framework ๐งโโ๏ธ Subcategory ๐ Introduction NeMo-Guardrails Content LLM-RSTR NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. Guardrails Content LLM-RSTR Guardrails is a Python package that lets a user add structure, type and quality guarantees to the outputs of large language models (LLMs). -
๐ค Structure LLM-RSTR
๐ฆ Prompting Framework ๐งโโ๏ธ Subcategory ๐ Introduction Guidance Structure LLM-RSTR guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting and chaining. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditional, loops) and generation seamlessly. Promptify Structure LLM-RSTR Prompt Engineering, Solve NLP Problems with LLM's & Easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify. ReLLM Structure LLM-RSTR Exact structure out of any language model completion. TypeChat Structure LLM-RSTR TypeChat is a library that makes it easy to build natural language interfaces using types.
Comparison of state-of-the-art prompting frameworks follows three macro dimensions: 1) compatibility, 2) capabilities and features, 3) documentation and support. The more detailed dimensions are shown below.
