gyunggyung / AGI-Papers

Papers and Book to look at when starting AGI 📚

🌟 AGI-Papers 🌟

LLM · NLP
Text2All · All2All
Multi-modal · Multi-task

_{Let's find out the latest and various LLM-related papers. 🙇‍♂️🙇‍♀️ by Stargazers}

New Papers

• MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
• MoVA: Adapting Mixture of Vision Experts to Multimodal Context
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
• Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages
• From r to Q∗: Your Language Model is Secretly a Q-Function
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Many-Shot In-Context Learning

before 2024

• Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration

Human intelligence thrives on the concept of cognitive synergy, where collaboration and information integration among different cognitive processes yield superior outcomes compared to individual cognitive processes in isolation. Although Large Language Models (LLMs) have demonstrated promising performance as general task-solving agents, they still struggle with tasks that require intensive domain knowledge and complex reasoning. In this work, we propose Solo Performance Prompting (SPP), which transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas. A cognitive synergist refers to an intelligent agent that collaborates with multiple minds, combining their individual strengths and knowledge, to enhance problem-solving and overall performance in complex tasks. By dynamically identifying and simulating different personas based on task inputs, SPP unleashes the potential of cognitive synergy in LLMs. We have discovered that assigning multiple, fine-grained personas in LLMs elicits better problem-solving abilities compared to using a single or fixed number of personas. We evaluate SPP on three challenging tasks: Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, encompassing both knowledge-intensive and reasoning-intensive types. Unlike previous works, such as Chain-of-Thought, that solely enhance the reasoning abilities in LLMs, SPP effectively elicits internal knowledge acquisition abilities, reduces hallucination, and maintains strong reasoning capabilities. Code, data, and prompts can be found at: this https URL.

• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs). Our framework consists of two modules: PairRanker and GenFuser, addressing the observation that optimal LLMs for different examples can significantly vary. PairRanker employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs. It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one. Our results demonstrate that PairRanker exhibits the highest correlation with ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses. To facilitate large-scale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.

• LeanDojo: Theorem Proving with Retrieval-Augmented Language Models

Large language models (LLMs) have shown promise in proving formal theorems using proof assistants such as Lean. However, existing methods are difficult to reproduce or build on, due to private code, data, and large compute requirements. This has created substantial barriers to research on machine learning methods for theorem proving. This paper removes these barriers by introducing LeanDojo: an open-source Lean playground consisting of toolkits, data, models, and benchmarks. LeanDojo extracts data from Lean and enables interaction with the proof environment programmatically. It contains fine-grained annotations of premises in proofs, providing valuable data for premise selection— a key bottleneck in theorem proving. Using this data, we develop ReProver (Retrieval-Augmented Prover): the first LLM-based prover that is augmented with retrieval for selecting premises from a vast math library. It is inexpensive and needs only one GPU week of training. Our retriever leverages LeanDojo's program analysis capability to identify accessible premises and hard negative examples, which makes retrieval much more effective. Furthermore, we construct a new benchmark consisting of 96,962 theorems and proofs extracted from Lean's math library. It features challenging data split requiring the prover to generalize to theorems relying on novel premises that are never used in training. We use this benchmark for training and evaluation, and experimental results demonstrate the effectiveness of ReProver over non-retrieval baselines and GPT-4. We thus provide the first set of open-source LLM-based theorem provers without any proprietary datasets and release it under a permissive MIT license to facilitate further research.

• ViperGPT: Visual Inference via Python Execution for Reasoning

Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ViperGPT utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.

• LongNet: Scaling Transformers to 1,000,000,000 Tokens

Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. In this work, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between tokens; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.

• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).

• 🦍 Gorilla: Large Language Model Connected with Massive APIs

Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, enabling flexible user updates or version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, we introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep up with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. The model and code of Gorilla are available at https://github.com/ShishirPatil/gorilla.

• ÷🦎 Chameleon: Plug-and-Play Compositional Reasoning with GPT-4

Large language models (LLMs) have achieved remarkable progress in various natural language processing tasks with emergent abilities. However, they face inherent limitations, such as an inability to access up-to-date information, utilize external tools, or perform precise mathematical reasoning. In this paper, we introduce Chameleon, a plug-and-play compositional reasoning framework that augments LLMs to help address these challenges. Chameleon synthesizes programs to compose various tools, including LLM models, off-the-shelf vision models, web search engines, Python functions, and rule-based modules tailored to user interests. Built on top of an LLM as a natural language planner, Chameleon infers the appropriate sequence of tools to compose and execute in order to generate a final response. We showcase the adaptability and effectiveness of Chameleon on two tasks: ScienceQA and TabMWP. Notably, Chameleon with GPT-4 achieves an 86.54% accuracy on ScienceQA, significantly improving upon the best published few-shot model by 11.37%; using GPT-4 as the underlying LLM, Chameleon achieves a 17.8% increase over the state-of-the-art model, leading to a 98.78% overall accuracy on TabMWP. Further studies suggest that using GPT-4 as a planner exhibits more consistent and rational tool selection and is able to infer potential constraints given the instructions, compared to other LLMs like ChatGPT.

• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

How to efficiently transform large language models (LLMs) into instruction followers is recently a popular research direction, while training LLM for multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter demonstrates the potential to handle visual inputs with LLMs, it still cannot generalize well to open-ended visual instructions and lags behind GPT-4. In this paper, we present LLaMA-Adapter V2, a parameter-efficient visual instruction model. Specifically, we first augment LLaMA-Adapter by unlocking more learnable parameters (e.g., norm, bias and scale), which distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, we propose an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset. During inference, we incorporate additional expert models (e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal instructions by merely introducing 14M parameters over LLaMA. The newly designed framework also exhibits stronger language-only instruction-following capabilities and even excels in chat interactions. Our code and models are available at this https URL.

• Generative Agents: Interactive Simulacra of Human Behavior

Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools. In this paper, we introduce generative agents--computational software agents that simulate believable human behavior. Generative agents wake up, cook breakfast, and head to work; artists paint, while authors write; they form opinions, notice each other, and initiate conversations; they remember and reflect on days past as they plan the next day. To enable generative agents, we describe an architecture that extends a large language model to store a complete record of the agent's experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior. We instantiate generative agents to populate an interactive sandbox environment inspired by The Sims, where end users can interact with a small town of twenty five agents using natural language. In an evaluation, these generative agents produce believable individual and emergent social behaviors: for example, starting with only a single user-specified notion that one agent wants to throw a Valentine's Day party, the agents autonomously spread invitations to the party over the next two days, make new acquaintances, ask each other out on dates to the party, and coordinate to show up for the party together at the right time. We demonstrate through ablation that the components of our agent architecture--observation, planning, and reflection--each contribute critically to the believability of agent behavior. By fusing large language models with computational, interactive agents, this work introduces architectural and interaction patterns for enabling believable simulations of human behavior.

• Reflexion: an autonomous agent with dynamic memory and self-reflection

Recent advancements in decision-making large language model (LLM) agents have demonstrated impressive performance across various benchmarks. However, these state-of-the-art approaches typically necessitate internal model fine-tuning, external model fine-tuning, or policy optimization over a defined state space. Implementing these methods can prove challenging due to the scarcity of high-quality training data or the lack of well-defined state space. Moreover, these agents do not possess certain qualities inherent to human decision-making processes, specifically the ability to learn from mistakes. Self-reflection allows humans to efficiently solve novel problems through a process of trial and error. Building on recent research, we propose Reflexion, an approach that endows an agent with dynamic memory and self-reflection capabilities to enhance its existing reasoning trace and task-specific action choice abilities. To achieve full automation, we introduce a straightforward yet effective heuristic that enables the agent to pinpoint hallucination instances, avoid repetition in action sequences, and, in some environments, construct an internal memory map of the given environment. To assess our approach, we evaluate the agent's ability to complete decision-making tasks in AlfWorld environments and knowledge-intensive, search-based question-and-answer tasks in HotPotQA environments. We observe success rates of 97% and 51%, respectively, and provide a discussion on the emergent property of self-reflection.

• Self-Refine: Iterative Refinement with Self-Feedback

Like people, LLMs do not always generate the best text for a given generation problem on their first try (e.g., summaries, answers, explanations). Just as people then refine their text, we introduce SELF-REFINE, a framework for similarly improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an output using an LLM, then allow the same model to provide multi-aspect feedback for its own output; finally, the same model refines its previously generated output given its own feedback. Unlike earlier work, our iterative refinement framework does not require supervised training data or reinforcement learning, and works with a single LLM. We experiment with 7 diverse tasks, ranging from review rewriting to math reasoning, demonstrating that our approach outperforms direct generation. In all tasks, outputs generated with SELF-REFINE are preferred by humans and by automated metrics over those generated directly with GPT-3.5 and GPT-4, improving on average by absolute 20% across tasks.

• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace

Solving complicated AI tasks with different domains and modalities is a key step toward advanced artificial intelligence. While there are abundant AI models available for different domains and modalities, they cannot handle complicated AI tasks. Considering large language models (LLMs) have exhibited exceptional ability in language understanding, generation, interaction, and reasoning, we advocate that LLMs could act as a controller to manage existing AI models to solve complicated AI tasks and language could be a generic interface to empower this. Based on this philosophy, we present HuggingGPT, a framework that leverages LLMs (e.g., ChatGPT) to connect various AI models in machine learning communities (e.g., Hugging Face) to solve AI tasks. Specifically, we use ChatGPT to conduct task planning when receiving a user request, select models according to their function descriptions available in Hugging Face, execute each subtask with the selected AI model, and summarize the response according to the execution results. By leveraging the strong language capability of ChatGPT and abundant AI models in Hugging Face, HuggingGPT is able to cover numerous sophisticated AI tasks in different modalities and domains and achieve impressive results in language, vision, speech, and other challenging tasks, which paves a new way towards advanced artificial intelligence.

• Auto-GPT: An Autonomous GPT-4 Experiment

Auto-GPT is an experimental open-source application showcasing the capabilities of the GPT-4 language model. This program, driven by GPT-4, chains together LLM "thoughts", to autonomously achieve whatever goal you set. As one of the first examples of GPT-4 running fully autonomously, Auto-GPT pushes the boundaries of what is possible with AI.

• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

• LeanDojo: Theorem Proving with Retrieval-Augmented Language Models

Large language models (LLMs) have shown promise in proving formal theorems using proof assistants such as Lean. However, existing methods are difficult to reproduce or build on, due to private code, data, and large compute requirements. This has created substantial barriers to research on machine learning methods for theorem proving. This paper removes these barriers by introducing LeanDojo: an open-source Lean playground consisting of toolkits, data, models, and benchmarks. LeanDojo extracts data from Lean and enables interaction with the proof environment programmatically. It contains fine-grained annotations of premises in proofs, providing valuable data for premise selection— a key bottleneck in theorem proving. Using this data, we develop ReProver (Retrieval-Augmented Prover): the first LLM-based prover that is augmented with retrieval for selecting premises from a vast math library. It is inexpensive and needs only one GPU week of training. Our retriever leverages LeanDojo's program analysis capability to identify accessible premises and hard negative examples, which makes retrieval much more effective. Furthermore, we construct a new benchmark consisting of 96,962 theorems and proofs extracted from Lean's math library. It features challenging data split requiring the prover to generalize to theorems relying on novel premises that are never used in training. We use this benchmark for training and evaluation, and experimental results demonstrate the effectiveness of ReProver over non-retrieval baselines and GPT-4. We thus provide the first set of open-source LLM-based theorem provers without any proprietary datasets and release it under a permissive MIT license to facilitate further research.

• Exploring the Curious Case of Code Prompts

Recent work has shown that prompting language models with code-like representations of natural language leads to performance improvements on structured reasoning tasks. However, such tasks comprise only a small subset of all natural language tasks. In our work, we seek to answer whether or not code-prompting is the preferred way of interacting with language models in general. We compare code and text prompts across three popular GPT models (davinci, code-davinci-002, and text-davinci-002) on a broader selection of tasks (e.g., QA, sentiment, summarization) and find that with few exceptions, code prompts do not consistently outperform text prompts. Furthermore, we show that the style of code prompt has a large effect on performance for some but not all tasks and that fine-tuning on text instructions leads to better relative performance of code prompts.

• Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Theory of Mind

Large Language Models (LLMs) perform complex reasoning by generating explanations for their predictions. However, a complementary goal of explanations is to also communicate useful knowledge that improves weaker agents. Hence, we investigate whether LLMs also make good teachers for weaker agents. In particular, we consider a student-teacher framework between two LLM agents and study if, when, and how the teacher should intervene with natural language explanations to improve the student's performance. Since communication is expensive, we define a budget such that the teacher only communicates explanations for a fraction of the data, after which the student should perform well on its own. We decompose the teaching problem along four axes: (1) if teacher's test time intervention improve student predictions, (2) when it is worth explaining a data point, (3) how the teacher should personalize explanations to better teach the student, and (4) if teacher explanations also improve student performance on future unexplained data. We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance. Next, we propose a Theory of Mind approach, in which the teacher builds two few-shot mental models of the student. The first model defines an Intervention Function that simulates the utility of an intervention, allowing the teacher to intervene when this utility is the highest and improving student performance at lower budgets. The second model enables the teacher to personalize explanations for a particular student and outperform unpersonalized teachers. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learning from explained data improves student performance on future unexplained data. Finally, we also verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.

• [Kosmos-2: Grounding Multimodal Large Language Models to the World]

We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at this https URL.

• PaLM 2 Technical Report

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities.

• MotionGPT: Finetuned LLMs are General-Purpose Motion Generators

Generating realistic human motion from given action descriptions has experienced significant advancements because of the emerging requirement of digital humans. While recent works have achieved impressive results in generating motion directly from textual action descriptions, they often support only a single modality of the control signal, which limits their application in the real digital human industry. This paper presents a Motion General-Purpose generaTor (MotionGPT) that can use multimodal control signals, e.g., text and single-frame poses, for generating consecutive human motions by treating multimodal signals as special input tokens in large language models (LLMs). Specifically, we first quantize multimodal control signals into discrete codes and then formulate them in a unified prompt instruction to ask the LLMs to generate the motion answer. Our MotionGPT demonstrates a unified human motion generation model with multimodal control signals by tuning a mere 0.4% of LLM parameters. To the best of our knowledge, MotionGPT is the first method to generate human motion by multimodal control signals, which we hope can shed light on this new direction. Codes shall be released upon acceptance.

• Multimodal Chain-of-Thought Reasoning in Language Models

Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. With Multimodal-CoT, our model under 1 billion parameters outperforms the previous state-of-the-art LLM (GPT-3.5) by 16 percentage points (75.17%->91.68% accuracy) on the ScienceQA benchmark and even surpasses human performance. Code is publicly available available at this https URL.

• Llama 2: Open Foundation and Fine-Tuned Chat Models

• unilm: Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

• Scaling Vision with Sparse Mixture of Experts

• “Low-Resource” Text Classification: A Parameter-Free Classification Method with Compressors

• AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

• Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

• Textbooks Are All You Need

• Model Card and Evaluations for Claude Models

• Full Parameter Fine-tuning for Large Language Models with Limited Resources

• Augmenting Language Models with Long-Term Memory

• Unifying Large Language Models and Knowledge Graphs: A Roadmap

• Knowledge Distillation of Large Language Models

• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

• QLoRA: Efficient Finetuning of Quantized LLMs

• LIMA: Less Is More for Alignment

• Orca: Progressive Learning from Complex Explanation Traces of GPT-4

• RWKV: Reinventing RNNs for the Transformer Era

• Dr. LLaMA: Improving Small Language Models Through Generative Data Augmentation

• The FLAN Instruction Tuning Repository

• Phoenix: Democratizing ChatGPT across Languages

• Long Sequence Modeling with XGen: A 7B LLM Trained on 8K Input Sequence Length

• RedPajama-INCITE

• China’s Baidu claims its Ernie Bot beats ChatGPT on key tests as A.I. race heats up

• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

• LTM-1: an LLM with a 5,000,000 token context window

• Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM

• StarCoder: may the source be with you!

• Cross-lingual Language Model Pretraining

• Language Is Not All You Need: Aligning Perception with Language Models

• Tackling multiple tasks with a single visual language model

• Large Language Models are Zero-Shot Reasoners

• Larger language models do in-context learning differently

• GPT Understands, Too

• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

• Do Prompt-Based Models Really Understand the Meaning of their Prompts?

• ∞-former: Infinite Memory Transformer

• Improving language models by retrieving from trillions of tokens

• Augmented Language Models: a Survey

• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

• Structure and Content-Guided Video Synthesis with Diffusion Models

• MusicLM: Generating Music From Text

• InstructGPT : Training language models to follow instructions with human feedback

• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

• Multimodal Chain-of-Thought Reasoning in Language Models

• Constitutional AI: Harmlessness from AI Feedback

• Provable Copyright Protection for Generative Models

• What learning algorithm is in-context learning? Investigations with linear models

• A Path Towards Autonomous Machine Intelligence

• PAL: Program-aided Language Models

• Toolformer: Language Models Can Teach Themselves to Use Tools

• LLaMA: Open and Efficient Foundation Language Models

• Improving alignment of dialogue agents via targeted human judgements

• Training Compute-Optimal Large Language Models

• LLaMA-based ChatGPT training, ChatLLaMA

• RLHF: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

• BaGuaLu: Targeting Brain Scale Pretrained Models with over 37 Million Cores

• LLaMA-7B, LLAMA Up-data, LLaMA: INT8 edition, UForm

• Flamingo: a Visual Language Model for Few-Shot Learning, Blog

• Multimodal Chain-of-Thought Reasoning in Language Models

• How to use UForm

• How to create KoChatLLaMA

• Competition-Level Code Generation with AlphaCode

• Scaling Language Models: Methods, Analysis & Insights from Training Gopher

• GPU and learning method required for KoChatLlaMA fine-tuning

• Advantages and Problems of UForm

• GPT-4 is coming next week – and it will be multimodal, says Microsoft Germany

• MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation

• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

• PaLM-E: An Embodied Multimodal Language Model

• Tightly-Integrated Generative Encoder-Decoder Representation

• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

• PaLM: Scaling Language Modeling with Pathways

• SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks

• LoRA: Low-Rank Adaptation of Large Language Models

• Language Models are Few-Shot Learners

• Low-rank Adaptation for Fast Text-to-Image Diffusion Fine-tuning

• Improving language models by retrieving from trillions of tokens

• FLAN: Finetuned Language Models Are Zero-Shot Learners

• T0: Multitask Prompted Training Enables Zero-Shot Task Generalization

• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

• The Wisdom of Hindsight Makes Language Models Better Instruction Followers

• Exploring the Benefits of Training Expert Language Models over Instruction Tuning

• Unsupervised Imputation of Non-ignorably Missing Data Using Importance-Weighted Autoencoders

• The Power of Scale for Parameter-Efficient Prompt Tuning

• Constitutional AI: Harmlessness from AI Feedback

• Deep reinforcement learning from human preferences

• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

• Large Language Models with Controllable Working Memory

• Do Prompt-Based Models Really Understand the Meaning of their Prompts?

• Muse: Text-To-Image Generation via Masked Generative Transformers

• Structure and Content-Guided Video Synthesis with Diffusion Models

• Generative Pretraining from Pixels

• A hunt for the Snark: Annotator Diversity in Data Practices

• Accurate global machine learning force fields for molecules with hundreds of atoms

• Algorithms with More Granular Differential Privacy Guarantees

• Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types

• Are we cobblers without shoes? Making Computer Science data FAIR

• Code Generation for In-Place Stencils

• Creating, Calibrating, and Validating Large-Scale Microscopic Traffic Simulation

• Increasing Impact of Mobile Health Programs: SAHELI for Maternal and Child Care

• Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges

• Developer Productivity for Humans: A Human-Centered Approach to Developer Productivity

• Development of a Machine Learning Model for Sonographic Assessment of Gestational Age

• Drug Design on Quantum Computers

• Estimates of broadband upwelling irradiance from GOES-16 ABI

• Information Processing and Management

• Flake Aware Culprit Finding

• Flexible Budgets in Restless Bandits: A Primal-Dual Algorithm for Efficient Budget Allocation

• Helpful Neighbors: Leveraging Neighbors in Geographic Feature Pronunciation

• High-Performance GPU-to-CPU Transpilation and Optimization via High-Level Parallel Constructs

• Helpful Neighbors: Leveraging Neighbors in Geographic Feature Pronunciation

• Infrastructuring Care: How Trans and Non-Binary People Meet Health and Well-Being Needs through Technology

• KwikBucks: Correlation Clustering with Cheap-Weak and Expensive-Strong Signals

• Learning to Bid in Contextual First Price Auctions

• Machine Learning for Healthcare: A Bibliometric Study of Contributions from Africa

• Scalable Decision-Focused Learning in Restless Multi-Armed Bandits with Application to Maternal and Child Health

• Robust Planning over Restless Groups: Engagement Interventions for a Large-Scale Maternal Telehealth Program

• Recitation-Augmented Language Models

• RL4ReAl: Reinforcement Learning for Register Allocation

• Quantum Simulation of Exact Electron Dynamics can be more Efficient than Classical Mean-Field Methods

• Quantum simulation of exact electron dynamics can be more efficient than classical mean-field methods

• Propeller: A Profile Guided, Relinking Optimizer for Warehouse-Scale Applications

• Deepmind: Improving language models by retrieving from trillions of tokens

• Deepmind: Building safer dialogue agents

• Deepmind: Competitive programming with AlphaCode

• Deepmind: Mastering Stratego, the classic game of imperfect information

• Deepmind: DeepMind’s latest research at NeurIPS 2022

• Deepmind: Building interactive agents in video game worlds

• Deepmind: Discovering novel algorithms with AlphaTensor

• Deepmind: AlphaFold reveals the structure of the protein universe

• Deepmind: Exploring the beauty of pure mathematics in novel ways

• Deepmind: Nowcasting the next hour of rain

• Deepmind: Putting the power of AlphaFold into the world’s hands

• Google Research: Deciphering clinical abbreviations with privacy protecting ML

• Google Research: Google Research, 2022 & beyond: Language, vision and generative models

• Google Research: Google Research, 2022 & beyond: Responsible AI

• Google Research: Learning with queried hints

• Google Research: Open Source Vizier: Towards reliable and flexible hyperparameter and blackbox optimization

• Google Research: Google Research, 2022 & beyond: ML & computer systems

• Google Research: Real-time tracking of wildfire boundaries using satellite imagery

• Google Research: Breaching the 2 LMP Approximation Barrier for Facility Location with Applications to k-Median

• Google Research: Chimane-Mosetén

• Google Research: Differentially Private All-Pairs Shortest Path Distances: Improved Algorithms and Lower Bounds

• Google Research: Differentially Private Fair Division

• Google Research: DiffQG: Generating Questions on Paired Sentences

• Google Research: Assessment of Security Defense of Native Programs Against Software Faults

• Google Research: Adaptive mixing of auxiliary losses in supervised learning

• OpenAI: Multimodal Neurons in Artificial Neural Networks

• OpenAI: DALL·E: Creating Images from Text

• OpenAI: CLIP: Connecting Text and Images

• OpenAI: Image GPT

• OpenAI: Jukebox

• OpenAI: Solving Rubik’s Cube with a Robot Hand

• OpenAI: Multimodal Neurons in Artificial Neural Networks

• OpenAI: CLIP: Connecting Text and Images

• OpenAI: Image GPT

• OpenAI: MuseNet

• OpenAI: Emergent Tool Use from Multi-Agent Interaction

before 2023

• [2013/01] Efficient Estimation of Word Representations in Vector Space

• [2014/12] Dependency-Based Word Embeddings

• [2015/07] Neural Machine Translation of Rare Words with Subword Units

• [2014/07] GloVe: Global Vectors for Word Representation : GloVe

• [2016/06] Siamese CBOW: Optimizing Word Embeddings for Sentence Representations : Siamese CBOW

• [2016/07] Enriching Word Vectors with Subword Information : fastText

• [2014/09] Sequence to Sequence Learningwith Neural Networks : seq2seq

• [2017/07] Attention Is All You Need : Transformer

• [2017/08] Learned in Translation: Contextualized Word Vectors : CoVe

• [2018/01] Universal Language Model Fine-tuning for Text Classification : ULMFIT

• [2018/02] Deep contextualized word representations : ELMo

• [2018/06] Improving Language Understanding by Generative Pre-Training : GPT-1

• [2018/10] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding : BERT

• [2019/02] Language Models are Unsupervised Multitask Learners : GPT-2

• [2019/04] Language Models with Transformers

• [2019/08] Neural Text Generation with Unlikelihood Training

• [2019/01] Cross-lingual Language Model Pretraining XLM

• [2019/01] Multi-Task Deep Neural Networks for Natural Language Understanding : MT-DNN

• [2019/01] Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context : Transformer-XL

• [2019/06] XLNet: Generalized Autoregressive Pretraining for Language Understanding : XLNet

• [2019/04] The Curious Case of Neural Text Degeneration

• [2019/09] Fine-Tuning Language Models from Human Preferences

• [2019/01] BioBERT: a pre-trained biomedical language representation model for biomedical text mining : BioBERT

• [2019/03] SciBERT: A Pretrained Language Model for Scientific Text : SciBERT

• [2019/04] ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission : ClinicalBERT

• [2019/06] HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization : HIBERT

• [2019/07] SpanBERT: Improving Pre-training by Representing and Predicting Spans : SpanBERT

• [2019/04] Publicly Available Clinical BERT Embeddings

• [2019/08] Pre-Training with Whole Word Masking for Chinese BERT

• [2019/07] Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment

• [2019/07] R-Transformer: Recurrent Neural Network Enhanced Transformer : R-Transformer

• [2019/09] FREELB: ENHANCED ADVERSARIAL TRAINING FOR LANGUAGE UNDERSTANDING : FREELB

• [2019/09] Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks

• [2019/10] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer : T5

• [2018/07] Subword-level Word Vector Representations for Korean

• [2019/08] Zero-shot Word Sense Disambiguation using Sense Definition Embeddings

• [2019/06] Bridging the Gap between Training and Inference for Neural Machine Translation

• [2019/06] Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts

• [2019/07] A Simple Theoretical Model of Importance for Summarization

• [2019/05] Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems

• [2019/07] We need to talk about standard splits

• [2019/07] ERNIE 2.0: A Continual Pre-training Framework for Language Understanding : ERNIE 2.0

• [2019/05] SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems : SuperGLUE

• [2020/01] Towards a Human-like Open-Domain Chatbot + Google AI Blog

• [2020/03] ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators : ELECTRA

• [2019/04] Mask-Predict: Parallel Decoding of Conditional Masked Language Models : Mask-Predict

• [2020/01] Reformer: The Efficient Transformer : Reformer

• [2020/04] Longformer: The Long-Document Transformer : Longformer

• [2019/11] DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation : DialoGPT

• [2020/01] Towards a Human-like Open-Domain Chatbot

• [2020/04] You Impress Me: Dialogue Generation via Mutual Persona Perception

• [2020/04] Recipes for building an open-domain chatbot

• [2020/04] ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogues : ToD-BERT

• [2020/04] SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model : SOLOIST

• [2020/05] A Simple Language Model for Task-Oriented Dialogue

• [2019/07] ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation : ReCoSa

• [2020/04] FastBERT: a Self-distilling BERT with Adaptive Inference Time : FastBERT

• [2020/01] PoWER-BERT: Accelerating BERT inference for Classification Tasks : PoWER-BERT

• [2019/10] DistillBERT, a distilled version of BERT: smaller, faster, cheaper and lighter : DistillBERT

• [2019/10] TinyBERT: Distilling BERT for Natural Language Understanding : TinyBERT

• [2019/11] Not Enough Data? Deep Learning to the Rescue!

• [2018/12] Conditional BERT Contextual Augmentation

• [2020/03] Data Augmentation using Pre-trained Transformer Models

• [2020/04] FLAT: Chinese NER Using Flat-Lattice Transformer : FLAT

• [2019/12] Big Transfer (BiT): General Visual Representation Learning : BiT

• [2019/04] ERNIE: Enhanced Representation through Knowledge Integration : ERNIE

• [2019/07] ERNIE 2.0: A Continual Pre-training Framework for Language Understanding : ERNIE 2.0

• [2020/06] ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph : ERNIE-ViL

• [2020/12] ERNIE-Doc: A Retrospective Long-Document Modeling Transformer : ERNIE-Doc

• [2021/07] ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation : ERNIE 3.0

• [2022/10] Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

• [2017/03] Distilling Task-Specific Knowledge from BERT into Simple Neural Networks

• [2020/10] DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling : DiPair

• [2021/08] Distilling Transformers for Neural Cross-Domain Search

• [2020/06] DeBERTa: Decoding-enhanced BERT with Disentangled Attention : DeBERTa

• [2020/11] VEGA: Towards an End-to-End Configurable AutoML Pipeline : VEGA

• [2020/12] FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding : FILTER

• [2019/12] StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding : StructBERT

• [2019/04] Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding : MT-DNN

• [2021/05] Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation

중지

MLLMArxivTalk

최신 MLLM 관련 스터디. 기본 오후에 진행. 논문, 강의, 코드, 뉴스, 블로그 등 다양한 자료로 학습.

MLLM, LLM, NLG, Dialogue, Reinforcement learning, Distillation, Efficient, Sentence similarity, multiple tasks, multimodal, Stable diffusion, TTS, Text-To-Video, All-To-All, 우주, 생명, 지능, 윤리, 규제, 법, 노화, 의학, 투자, 개발, 인프라, 디자인, 경영, ETC...

유망 스타트업 C레벨, 국내외 탑티어 연구자, 국내외 탑티어 대학, 대학원 재학생과 졸업생, 석학, 교수 등 A급 인재들이 최신 논문, 강의 등 스터디 및 프로젝트 진행.

기본 매주 수요일 오후 7시반. 사전 학습 없이 논문 읽기 최대 20분, 토론 최대 40분. 한 번에 1 ~ 10개 논문, 강의 등 진행. 지금까지는 항상 3개. 주제 논문 선정은 자유. 탑티어 학회 논문 및 프로젝트 제작 예정.

주말을 포함하여, 거의 매일 추가 스터디 존재. 흥미로운 주제거나 참여 되는 날만 중간에 들어와서 중간에 나가도 무관. 모든 규칙은 협의 가능. 오프라인 모임도 예정. 자율 참여.

스터디 규칙

1. 영어만 사용은 금지. 한국어 중심 사용. 특수 용어는 영어 사용.
2. 1주일에 논문 2개 이상 스터디. 되는 사람은 10개 이상.
3. 3분에서 20분 현장에서 논문 읽기. 5분에서 30분 토론.
4. 1시간 스터디 시, 바로 나가도 됨. 원할 때 10분 이하 참여도 무관. 자유롭게 진행. 2시간 매일도 가능.
5. 각자 더 뛰어난 게 있다는 것을 인지. 다들 대단한 분들이니 질문 많이 하고, 정보 공유 자주.
6. 본인이 하기로 한 일만은 수행. 한다고 말하고, 안 하는 것은 민폐다.
7. 기본적으로 녹화 후 내부 공유.
8. 정보를 혼자 알게 쓰지 말고, 다 같이 알게 말하기.
9. 개인 사정으로 스터디 탈퇴 시, 자기소개에 인사 작성.
10. 여러 기관 좋은 규칙 붙여넣기.
11. 팀에 도움이 된다고 판단하면, 위 규칙을 모두 무시하고 행동.
12. 추가.

Basic knowledge

mathematics	machine learning	Transformer	Hugging Face
mathematics for machine learning	Pattern Recognition and Machine Learning	Getting Started with Google BERT	Natural Language Processing with Transformers