This repo contains material and information from the workshop run by Aggregate Intellect to update and educate our community about the explosive progress in language models (and generally generative models). We will continue adding material to this repository, but feel free to add any questions or suggestions through pull requests or through reporting issues.
An Open Book Project Given the rapid pace of development in LLM space, we are releasing this information as an open book project which means that we welcome and encourage contributions from anyone who is interested and esp anyone who has used LLMs in production setting and wants to share their learnings with the broader community. See the bottom of this page for step by step instructions for how you can contribute. Here are some examples of contributions you can make:
- If you are interested in this topic but you aren't an expert:
- Fix typos or grammatical errors
- Fix formatting issues
- Rewords any sections to make them more understandable
- If you are interested in this topic, and curious about how certain things can be done:
- Provide feedback on existing content through Issues (see the bottom of this page for details)
- Ask questions or make suggestions through Issues (see the bottom of this page for details)
- If you are closely familiar with the topic and / or have hands on experience:
- Add new resources with explanation
- Add new sections for important topics that are missing
- Improving existing content by adding more detail or context
TIP: Follow the gif below to find the Table of Contents on the top left corner of this pane.
DevOps and ModelOps have transformed software development over the past two decades, but now we are seeing a further shift left with the rise of KnowledgeOps. This new era leverages tools to augment our problem-solving, planning, and critical thinking abilities, enabling us to tackle highly complex knowledge work with greater efficiency and effectiveness. KnowledgeOps promises to enhance our ability to experiment with a wider range of ideas and select the most impactful ones, similar to the benefits seen in DevOps and related methodologies.
TWITTER THREAD SUMMARY OF THE TALK:
- KnowledgeOps and Development Processes
- 1/14: #KnowledgeOps is about managing an organization's or community’s knowledge assets and processes to enable reuse and collaboration. #DevOps and #ModelOps (#MLOps) are specific examples that help develop software faster and with lower chances of failure.
- 2/14: #ShiftingLeft and #CI/CD are important #DevOps concepts that bring testing and quality assurance to the beginning of the software development process and provide tools to automate them for consistency and repeatability.
- 3/14: Although automation has improved many parts of the development process, the step of discovering and planning is still largely manual, requiring continuous communication and collaboration with teammates, which creates a single point of failure.
- 4/14: #ModelOps (e.g., #MLOps) has allowed for the automation of data and model handling, but the manual interpretation and decision-making steps are still present. #Automation
- Generative AI, and Continuous Exploration / Continuous Integration / Continuous Delivery
- 5/14: Generative AI can help us shift further left into the exploration, planning, and coding steps, significantly improving our ability to explore options and conduct experiments. #GenerativeAI #SoftwareDevelopment
- 6/14: There will eventually be more automation in our problem-solving processes, but in the meantime, tools built with generative AI will significantly augment our ability for interpretation and decision-making in ensuring the success of the development of complex software systems. #MachineLearning #AI
- 7/14: Emerging tools for thinking allow for a more experimental approach to knowledge-intensive work, allowing for continuous hypothesis generation and experimentation leading to CE/CI/CD. #ContinuousExploration #ContinuousIntegration #ContinuousDelivery
- 8/14: There is a trend towards interfacing generative copilots, retrieval systems, and other knowledge-intensive systems to create thinking machines with memory and reasoning skills. #GenerativeAI #RetrievalSystems #Reasoning
- 9/14: Language models are essential for these tools because they need to interpret users’ instructions usually provided in natural language, communicate results, and facilitate human-human communication. #LanguageModels #Communication
- 10/14: Language models can give us the ability to articulate and communicate complex ideas effectively to stakeholders and team members enabling more efficient problem solving in communities and organizations. #LanguageModels #Communication
- Adoption of Knowledge-Ops
- 11/14: In bigger companies, beyond technology, the biggest barriers to implementing #KnowledgeOps are cultural problems; eg. political reasons that prevent a unified and integrated knowledge and expertise system connected to knowledge bases of all teams across the enterprise. #CulturalProblems
- 12/14: Since all #GPT can reliably provide in short term is the language skill, primarily NLU/NLG, smaller language models trained on internal knowledge can be built to avoid privacy and data access issues. #NaturalLanguageUnderstanding #NaturalLanguageGeneration
- 13/14: Adoption of #LLM enabled thinking tools will start in smaller companies, and with improvements in corporate culture and maturing technology, we will see bigger companies joining the movement.
- 14/14: With these tools being able to talk to us, remember our context, and reason about the world around us without the barriers of coding and formal language, we can accelerate #KnowledgeOps to the point where no idea is too expensive to try. #Automation
Amir is the co-founder of Aggregate Intellect (https://ai.science/), a Smart Knowledge Navigator platform for R&D teams in emerging technologies like AI. Prior to this, Amir was an NLP Product Lead at Royal Bank of Canada, and held a postdoctoral position at University of Oxford conducting research on experimental quantum computing. Amir holds a PhD in Physics from University of Toronto.
In this talk, Josh, an ML expert with experience commercializing NLP-powered services, will discuss the potential for leveraging foundation models to drive agile innovation in both individual and organizational processes. He will share lessons learned from his work with a bootstrapped startup and provide insights on how LLMs can be commercialized effectively.
TWITTER THREAD SUMMARY OF THE TALK:
- Opportunities for large language models in startups and innovation
- 1/21: Large language models offer exciting possibilities for startups and product development. They can revolutionize the innovation process with qualitatively new ways of problem-solving. Before building a product, take a step back and see if an approach powered by these models could serve a purpose.
- 2/21: GitHub co-pilot is a perfect example of how large language models can be used in engineering. By offering new ways of writing code, it streamlines the development process and allows engineers to focus on solving problems.
- 3/21: Quick feedback from users is crucial for product development, even if you are one of the target users. Hackathons and accelerators can be beneficial to startups by providing access to large language models as a service and infrastructure credits.
- 4/21: Building a lean, agile startup focused on R&D is aided by resources such as infrastructure credits, tax credits, subsidies, marketing grants, and IP support. In Canada, AI startups receive federal and provincial programs supporting R&D. Bootstrapping can sustain a company from day one.
- 5/21: Bootstrapping can be a good way to sustain a company based on revenue from day one. It also forces you to start selling as soon as possible to get revenue and establish quick feedback for market fit. Investment is not always necessary for R&D.
- IP strategy for large language models
- 6/21: When building large language models for commercialization, it's important to consider IP strategy and seek advice from experts.
- 7/21: Applying large language models to improve existing capabilities may not be patentable, but rethinking the entire approach to solve a problem in a new way is more likely to be patentable. #MLdevelopment #IPstrategy
- Large language models for R&D
- 8/21: During R&D, specify tasks, models to optimize, & create dataset to evaluate models against. Use large language models for weak label generation & data augmentation to make data set curation easier. #AI #ML #R&D
- 9/21: R&D conversations are often open & difficult to structure, but they can still be turned into components used by large language models. For example, to create a conversational bot for Slack, modularize into 3 LLM calls for message, trigger, & audience classification. #AI #ML #chatbot
- 10/21: With modularization & templated code repository, create conversational bot very quickly. Use LLM-generated info for message content classification, trigger classification, & audience classification. #AI #ML #chatbot #Slack
- Microservices and modularization of products based on large language models
- 11/21: Large language models (LLMs) can be used in various stages of a project, including engineering and production. They can be leveraged to build microservices or components that work together to produce the desired output.
- 12/21: Treating a LLM as one of the microservices of a product enables breaking down the problem into smaller and more manageable pieces. This allows for easier implementation and development, additional R&D, unit testing, and quality control.
- 13/21: This approach also allows for easier explainability and optimization of the process. The components in the architecture of a system that leverages LLMs should interface the language skills of GPT-like models and domain-specific expertise to get the best results.
- 14/21: In our slack app example, the LLM prompts used depend on the context and specifics of what is being done. In some cases, having a good prompt engineer is crucial, while in others, we can just pass exemplars that would have a bigger influence on the performance.
- 15/21: We can add a dialogue analysis component to control the prompts generated by LLMs in a particular domain. This component can infer information necessary to determine the most relevant examples when generating probing questions.
- 16/21: Another component can create dynamic prompts for in-context learning by choosing the best exemplars from a repository or by retrieving documents containing domain knowledge. This can significantly improve the performance of LLMs.
- 17/21: Finally, a quality control component can rank generated candidates in case the LLM generates a question or answer that is not suitable. Rule-based and human-crafted questions generated using question recipes can also be used to ensure that inappropriate responses are avoided.
- Production stage of large language models
- 18/21: In production, large language models can be optimized by combining them with other components to create an ecosystem of microservices that work together. This approach ensures better performance and improved efficiency of the overall system.
- 19/21: When optimizing costs during a project, evaluating both performance and cost is crucial. If a cheaper model performs almost as well as a more expensive one, it's advisable to choose the former. This way, you can save money without compromising on performance.
- 20/21: After some data is collected and the overall system performance is more well established, it's sometimes possible to replace large language models with cheaper and leaner models in production. This approach helps to reduce costs without sacrificing performance.
- 21/21: When building a startup with large language models, it's essential to consider the service level availability uptime of cloud infrastructure. Microsoft Azure Open AI service offers 99.9% uptime, ensuring that your system will be available to users when they need it.
Resources
- Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks
- Canada Government Business Benefits Finder
- Understanding UK Artificial Intelligence R&D commercialisation and the role of standards
Josh is the CTO at Nexxt Intelligence, where he leads R&D on LLMs and NLP to build innovative solutions for the market research industry. He also works in biodiversity and applications of AI for conservation.
The proliferation of ChatGPT and other large language models has resulted in an explosion of LLM-based projects and startups. While these models can provide impressive initial demos, integrating them into a product requires careful consideration and planning. This talk will cover key considerations for creating, testing, and optimizing prompts for LLMs, as well as how to run analytics on key user metrics to ensure success.
TWITTER THREAD SUMMARY OF THE TALK:
- Considerations for adopting LLMs
- 1/16: Running large language models in house can be costly, which is why many people use APIs. It's important to ensure your use case has a good ROI to avoid wasting resources.
- 2/16: When deciding how much to invest in using LLMs, companies need to consider their bet size (level of investment) and company size (effort required to adopt LLMs). This should be done with a careful assessment of consequences on their existing customers and their product development roadmap.
- 3/16: Therefore companies need to understand their business and customers before adopting LLMs. Voiceflow, a conversational AI platform used by various verticals, experimented with LLMs and how they should be combined with existing capabilities.
- 4/16: Even large companies, eg. Google's Bard and Bing's demos have had instances where LLMs generated incorrect information damaging their brands. Having humans in the loop, domain experts when appropriate, is important to ensure accuracy.
- 5/16: UX and UI are critical for LLM adoption. Incorporating fun and natural features, such as shortcuts and accepting / declining recommendations, are important for making LLMs easy to use and understand.
- 6/16: Human-in-the-loop performance improvements at creation time are less risky than having that kind of feature in run time. One should carefully consider the trade offs when implementing user feedback loops.
- 7/16: Caching might be a good way to reduce cost and improve performance. Though it has to be done in a way that context for different users is handled properly to ensure the best experience.
- Deployment methodology for LLMs
- 8/16: When deploying large language models, there are different options to consider. Using a pre-trained model service is easier, but building and hosting your own solution gives you more control.
- 9/16: Building a product that generates revenue is essential, but investing time in a minimally viable platform is also important. Neglecting the platform can lead to technical debt and slow down the iteration process for new models. There should be a balance in effort put in model, product, and platform development.
- Testing and Fine-Tuning for LLMs
- 10/16: When deploying LLMs in production, testing is crucial. Denys Linkov built a test suite to check if prompts work, processing the code, running python, and documenting errors. This is on top of initial manual testing and collecting errors from the data warehouse.
- 11/16: For example, since the output of Open AI API is probabilistic it might return poorly formatted JSON responses. This at inference time might be easy to deal with but at high volume could cause unforeseen failures in production systems.
- 12/16: Fine-tuning can solve formatting issues, especially for few-shot learning including desired output, examples, and chain of thought reasoning.
- Considerations for Third-Party Providers
- 13/16: Running a product with LLMs comes with its own set of challenges, like unpredictable response formatting, unreliable uptime, or other upstream dependencies.
- 14/16: When choosing NLP provider APIs or open-source libraries, it's crucial to consider their reliability and ease of integration. Companies can opt for a single or multiple NLP provider APIs based on their required uptime.
- 15/16: While OpenAI is perceived to have the best models, other providers may offer more reliability. The trade-off between model accuracy and uptime is a critical consideration for companies depending on their service requirements.
- 16/16: With LLM space moving quickly, it might also become easier to deploy own models instead of relying on third-party providers at least for some components of the system.
Resources
Denys is the ML lead at Voiceflow focused on building the ML platform and data science offerings. His focus is on realtime NLP systems that help Voiceflow’s 60+ enterprise customers build better conversational assistants. Previously he worked at large bank as a senior cloud architect.
Incorporating Large Language Models into Enterprise Analytics: Navigating the Landscape of In-Context Learning and Agents
Large Language Models (LLMs) have dramatically changed our expectations for AI. While a few innovators are building proof-of-concept projects using APIs, most enterprise analytic teams still need to figure out how to incorporate LLMs into their analytical toolbox. Rajiv shows the necessity of understanding the growth of "in-context learning" and agents. With these insights, he explains how LLMs will shape enterprise analytics. Along the way, he covers many practical factors, such as the different providers of LLMs, resource costs, and ethical issues.
TWITTER THREAD SUMMARY OF THE TALK:
- Advantages and Use Cases of Large Language Models
- 1/14: NLP use cases in the enterprise primarily focus on text classification, summarization, question answering, and embeddings. Large language models like GPT-3 and ChatGPT can transform how enterprises approach these tasks.
- 2/14: LLMs offer advantages over traditional NLP models, such as better performance in classification and language generation for use cases like financial sentiment analysis and customer service bots.
- 3/14: To leverage LLMs in enterprise workflows, companies can start by exploring pre-trained models, fine-tuning them on their data. However, LLMs are general-purpose models and adding domain-specific information might be challenging. Using dedicated models for certain tasks, combined with LLMs can be more efficient and cost-effective.
- 4/14: LLMs can connect previously siloed domains in an enterprise, allowing for more efficient and natural language searches across multiple domains. This makes it easier to access relevant information and insights from different parts of the organization.
- 5/14: The UI is just as important as the model in solving business problems. Therefore, it's essential to focus on UI/UX design in addition to developing models. While models can add new capabilities, the user interface plays a critical role in presenting them in a valuable way.
- Interfacing LLMs and other Enterprise Systems
- 6/14: LLMs can be decision-makers, but it's crucial to set them up for success by combining them with other tools. For instance, for a math question LLM could be set up to call a calculator and then show the result to prevent it from giving the wrong answer with confidence.
- 7/14: Large Language Models can enhance information retrieval by combining classic data / knowledge bases with LLMs to provide more accurate and human-readable responses to queries. For example, LLMs were used to retrieve information on how to install Transformers from Huggingface documentation with citation to resource.
- Choosing and Keeping Up with LLMs
- 8/14: Choosing the right LLM depends on the specific use case, and there are various options available with different licensing requirements and costs.
- 9/14: When choosing an LLM, consider factors like availability, cost, and accuracy for the specific task at hand. Open source models like Hugging Face can provide a wide variety of models and tools leveraging the collective knowledge of the community.
- 10/14: The field of LLMs is evolving rapidly. Staying up to date on the latest developments and trends is crucial to make the most of this technology. Keep learning to keep improving!
- 11/14: For those looking to learn AI, OpenAI Sandbox and Hugging Face Spaces are excellent starting points. OpenAI Sandbox provides templates and examples for learning AI, while Hugging Face is a useful resource for natural language processing. LangChain and LlamaIndex are also good examples to get started.
- Democratization and Ethical Considerations in AI
- 12/14: The democratization of AI is crucial for its widespread adoption. Co-pilots and natural language interfaces are enabling more people to experiment with AI, even those without a technical background. Lowering the barriers to entry for AI can have a significant impact on the democratization of AI.
- 13/14: Ethical considerations are essential in the development and deployment of AI. It's crucial to include diverse voices in the design and decision-making processes, educate users on the potential dangers of generative AI, and consider the ethical implications of AI.
- 14/14: Bias is a significant challenge in AI. Historic data used for training models can perpetuate biases and historical injustices. Therefore, it's crucial to be aware of these biases and work towards minimizing their impact.
Resources
- Talk Slides
- Sentiment Spin: Attacking Financial Sentiment with GPT-3
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
- Language Models are Few-Shot Learners
- Transformers learn in-context by gradient descent
- News Summarization and Evaluation in the Era of GPT-3
- Benchmarking Large Language Models for News Summarization
- BIG-Bench - Beyond the Imitation Game Benchmark
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- ChatGPT for Robotics: Design Principles and Model Abilities
- Replicate MRKL chain uding LangChain
- Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence
- The CEO’s Guide to the Generative AI Revolution
- A New Era of Creativity: Expert-in-the-loop Generative AI
Rajiv is a machine learning engineer at Hugging Face, whose primary focus is on enabling enterprise teams to succeed with AI. He is a widely recognized speaker on enterprise AI and was part of data science teams at Snorkel AI, Caterpillar, and State Farm.
Generative AI has made impressive advances in creating music, art, and even virtual worlds that were once thought to be exclusively the domain of human creators. However, with such power comes great responsibility, and we must be mindful of the ethical implications of our creations. In this session, we will explore the intersection of generative AI, ethics, and accessibility. We will examine ethical considerations related to bias, transparency, and ownership, as well as the challenges of making generative AI accessible to individuals with disabilities and those from underrepresented communities.
TWITTER THREAD SUMMARY OF THE TALK:
- Use of generative AI to improve accessibility and the lives of people with disabilities
- 1/20: When developing generative AI, ethical considerations should be at the forefront of our minds. Those of us who are building AI systems, and helping others use AI should educate everyone we interact with to build more responsibly. #AIethics #generativeAI #inclusiveAI
- 2/20: For example, voice technology combined with generative capabilities can improve the lives of people with disabilities. Noelle has seen firsthand how these technologies helped her family members, and how much more needs to be done still.
- 3/20: Generative AI has the potential to make conversational agents more efficient by eliminating the need for explicit mapping of intent and therefore better flexibility to various levels of need and speech / communication patterns.
- 4/20: It is important for AI models to be flexible to accommodate new use cases and user personas over time. By incorporating flexible and inclusive design principles, AI developers can ensure that their technologies are accessible and beneficial to everyone.
- 5/20: Inclusive engineering is crucial for building good AI solutions. It involves team building and data collection with diverse perspectives in mind. One of the best ways to achieve this is hiring diverse teams with different backgrounds, opinions, needs, and demographics.
- Managing and moderating LLMs to reduce bias, and increase fairness
- 6/20: Deliberate choices that companies make can reduce the impact of bias, and mitigate risks. Mitigating potential issues with LLMs requires a comprehensive approach, including diverse datasets, fine-tuning foundation models, utilizing hardware resources, managing content, and continuous monitoring.
- 7/20: LLMs are trained on data sets generated by humans, and the awareness of potential sources of bias and inaccuracies in that data is crucial. Diverse datasets are necessary when training LLMs to reduce the impact of those imperfections.
- 8/20: LLMs make assumptions based on training data. This can result in incorrect conclusions. Noelle mentioned that a falsely attributed scholarly article was taken as true by an LLM because it was making assumptions based on the training data.
- 9/20: There is a need for inclusive data collection to ensure that AI solutions represent end users. Inclusive data collection along with keeping track of lineage and context can help mitigate biases that may be present in training data.
- 10/20: LLMs can amplify bias over time, but there are ways to mitigate its impact. Having a human-in-the-loop process to monitor, manage and control the LLMs is crucial. #AmplificationOfBias #HumanInLoop
- 11/20: LLMs can speed up the process of generating content, but if not managed at scale, it can slow down the process by generating inappropriate or poor-quality content. Moderation and management of the content generated by LLMs are crucial. #ContentGeneration #Moderation
- 12/20: Specific performance metrics for LLMs are essential to understand what is a good model and what is not. Continuous monitoring by a human team is necessary to ensure that the model is working correctly. #PerformanceMetrics #ContinuousMonitoring
- Versatility and scalability of potential LLM use cases
- 13/20: LLMs are built on foundation models that can be fine-tuned to fit specific business needs. This allows businesses to create LLMs tailored to their specific needs and goals, making them increasingly popular with companies willing to invest in them.
- 14/20: The demand for LLMs is growing rapidly, and ML developers need to be able to build and deploy them quickly to meet the demand. This puts a premium on rapid development and deployment processes.
- 15/20: LLMs can do more than just conversation. They can generate natural language requests and responses from various sources, including customer signals, website data, and ticketing systems. They can even be used to automatically formulate human questions and generate Power BI dashboards.
- 16/20: Next generations of LLMs might be multi-modal which means that they can combine different types of input, such as images and text, to generate output. This makes them valuable in a variety of contexts and use cases.
- 17/20: LLMs require significant hardware resources to operate, and only a few labs in the world can support the required hardware. A balance has to be made between building internally, using offerings from mainstream vendors, and ones from emergent providers.
- Challenges and Risks of LLMs
- 18/20: Organizations have a responsibility to guard against potential legal issues when using LLMs. Noelle emphasized the responsibility of organizations to think through potential legal issues and approach projects with an awareness of the risks involved.
- 19/20: Evaluating risks and mitigating them in the solution is important. There are use cases for LLMs that are relatively easy to approach and mitigate risks, such as customer call centers and customer service ticketing. However, more rigor and discipline are required for projects using codex, which were trained on Github repos.
- 20/20: Indemnification is important when using LLMs to protect against ownership challenges that may arise in the future. Enterprise level solutions provide more indemnification than research models like Dalle, especially if the model is not custom trained on your own art.
Noelle Silver Russell is a multi-award-winning technologist and entrepreneur who specializes in advising companies on emerging technology, generative AI and LLMs. She is the Global AI Solutions Lead as well as the Global Industry Lead for Generative AI at Accenture. She has led teams at NPR, Microsoft, IBM, and Amazon Alexa, and is a consistent champion for AI literacy and the ethical use of AI based on her work building some of the largest AI models in the world. She is the founder of AI Leadership Institute and she was recently awarded the Microsoft Most Valuable Professional award for Artificial Intelligence as well as VentureBeat’s Women in AI Responsibility and Ethics award.
This talk explores how language models can be integrated with external tools, such as Python interpreters, API's, and data stores, to greatly expand their utility. We will examine the emerging field of 'LLMOps' and review some promising tools. Additionally, we will push the boundaries of what's possible by exploring how a language model could accurately answer complex questions like, "Who was the CFO at Apple when its stock price was at its lowest point in the last 10 years?"
TWITTER THREAD SUMMARY OF THE TALK:
Resources
- Augmented Language Models: a Survey
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
- LangChain Docs
- LlamaIndex (formerly GPT-Index
Suhas is the CTO & Co-founder of Bedrock AI, an NLP startup operating in the financial domain, where he conducts research on LLMs, domain adaptation, text ranking, and more. He was the co-chair of the Privacy WG at BigScience, the chair at TMLS 2022 and TMLS NLP 2022 conferences, and is currently writing a book on Large Language Models.
An emerging aspect of large language models is their ability to generate datasets that allow them to self-improve. A fascinating recent example is Toolformer (Schick et al.) in which LLMs generate fine-tuning data that helps them learn how to use tools at run-time. In this talk, we’ll examine this trend by taking a close look at the Toolformer paper and other related research.
TWITTER THREAD SUMMARY OF THE TALK:
- Large Language Models and Synthetic Data
- Research on using unlabeled data to improve large language models is exciting, and the potential impact on natural language processing is vast. These models are changing the way we think about language and the possibilities of AI.
- Large language models are trained on vast amounts of unlabeled data in a self-supervised manner. They continue to show impressive results as they scale, producing higher quality and more human-like text even for tasks they are not explicitly trained to perform.
- As AI adoption increases, there will be a growing demand for human annotators soon surpassing human capacity to keep up with data needs of increasingly larger models and more complex use cases.
- One of the interesting new areas is to use large language models themselves to create new data for training. For example, synthetic data can be generated to augment existing datasets for improving LLMs themselves or other types of models.
- These kinds of data augmentation techniques can be used to improve large language models reducing the need for human annotation. This can reserve the more expensive human labor for creating high-quality or mission critical datasets.
- Another trend we're seeing in the industry is that human annotations will be used more for creating evaluation or quality control datasets, while LLMs will be used for generating training data. #machinelearning #datageneration #humansintheLoop
- This approach combines the strengths of both human annotation and machine learning, and has the potential to increase research capacity by generating more training data. #machinelearning #datageneration #humansintheLoop #researchcapacity
- Using Large Language Models for Data Generation
- Recent research papers have shown that we can use large language models to generate weak labels for tasks such as named entity recognition, sentiment analysis, and question answering. We can then have humans revise or validate these labels to create high-quality training data. #machinelearning #datageneration #humansintheLoop
- Toolformer is one example of a system that uses LLMs to generate data for training other models. It splits up the data set and samples API calls to generate possible inputs and outputs for different tools. Toolformer then computes the model's loss to predict the next words in the sequence.
- Techniques for Filtering Data for LLM Fine-Tuning
- Choosing the right data for fine-tuning is essential for improving LLMs' performance. Various approaches can be used to filter down the generated data for direct use, or human annotator intervention.
- There are multiple ways to filter a data set for LLM fine-tuning. Techniques like perplexity-based approach (Toolformer), diversity of fine-tuning samples (Self-instruct), AI annotator (RLAIF), and self-consistency (Constitutional AI) can help with that.
- Fine-Tuning Language Models with Self-Consistency
- Self-instruct and self-consistency approaches are suitable for fine-tuning with available (frozen model) endpoints. These approaches involve generating new tasks and instructions for the model to fine-tune on. ???
- Self-instruct papers use human-created examples to train models to generate instructions and outputs for tasks. Language models can also use self-consistency to fine-tune themselves by generating different outputs and comparing them to select the most frequent one.
- This technique does not require the model to know the ground truth, but as the models become larger, the most frequent output is often the correct one. It is observed in literature that larger language models generate more accurate responses.
- The model filters down data using self-consistency, and if the majority of generations produce a specific output, e.g., "nine," the model takes all the cases where "nine" was generated as the output, assuming that these are correct, and feeds them back into the model which creates a feedback loop that improves the model's performance over time.
- Reinforcement learning from AI feedback (RLAIF) for Harmless and Helpful Language Models
- RLAIF is a promising application for large language models where models can learn from their own mistakes and improve over time. It is a method for training language models to be more helpful and harmless. It uses a Constitution to critique the model and train it to rank outputs based on preferences.
- To train the model, harmful responses are generated through red teaming requests, and the Constitution is used to guide the model's behavior and critique its responses. The model is then fine-tuned on a dataset of revisions based on its critiques. #RedTeaming #ModelTraining
- The Constitution is created by humans as a guideline for the model's behavior, but the model is able to critique itself and generate revisions based on the Constitution. This allows for more training data to be generated using the model itself, increasing research capacity. #AIResearch
Resources
Gordon is the Senior Engineering Manager of the Applied Machine Learning team at Ada where he's helped lead the creation of Ada's ML engine and features. Gordon's background is in Engineering Physics and Operations Research, and he's passionate about building useful ML products.
Large language models (LLMs) have the potential to perform a wide range of tasks by understanding human queries, but they are often sensitive to the wording of the prompts, which can greatly affect the output. This talk will introduce RLPrompt, an efficient algorithm that uses reinforcement learning to systematically search for the best prompts to improve LLM performance across diverse tasks.
TWITTER THREAD SUMMARY OF THE TALK: coming soon...
Resources
- RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning blog | paper | code
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Making Pre-trained Language Models Better Few-shot Learners
- Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
- A Recipe For Arbitrary Text Style Transfer with Large Language Models
Mingkai Deng is a PhD student at Carnegie Mellon University working at the intersection of machine learning, computer vision, and natural language processing. Prior to that, he was a data scientist who led award-winning projects and built analytics products that serve multiple Fortune 500 clients.
To make changes to this project, you will need to fork the repository to your own GitHub account, make the changes to your forked repository, and then submit a pull request to our main repository. Here are the steps:
- Fork the repository to your own GitHub account by clicking the "Fork" button on the top right of the repository page.
- Clone the forked repository to your local machine by running the following command in your terminal:
git clone https://github.com/your-username/repo-name.git - Create a new branch for your changes by running the following command:
git checkout -b your-branch-name - Make the desired changes to the book.
- Commit your changes with a descriptive commit message by running the following command:
git commit -m "your commit message" - Push your changes to your forked repository by running the following command:
git push origin your-branch-name - Go to the main repository and submit a pull request with a description of your changes.
Once your pull request is submitted, it will be reviewed by our team and either merged or commented on for further changes.
NOTE: If the change you are considering is fairly major, please suggest it as an issue first so that we can coordinate before you put in the effort.
If you would like to suggest changes to the book without making direct changes, you can create an issue in our repository. Here are the steps:
- Go to the repository and click on the "Issues" tab.
- Click on the "New Issue" button.
- Provide a descriptive title and description of your suggested change.
- Optionally, you can label your issue to make it easier to categorize.
- Submit the issue.
Our team will review the issue and either implement the suggested change or respond with feedback.