sobamchan

Perplexica

Perplexica is an AI-powered search engine. It is an Open source alternative to Perplexity AI

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

haiku-dpo

Using open source LLMs to build synthetic datasets for direct preference optimization

InPars

Inquisitive Parrots for Search

RAGElo

RAGElo is a set of tools that helps you selecting the best RAG-based LLM agents by using an Elo ranker

bibsearch

Download, manage, and search a BibTeX database.

semantic-grep

grep for words with similar meaning to the query

DSI-transformers

A huggingface transformers implementation of "Transformer Memory as a Differentiable Search Index"

GenIR-Survey

This is the repository for the GenIR survey.

overlapy

Python package developed to evaluate textual overlap (N-Grams) between two volumes of text.

ccsum

summary-of-a-haystack

Codebase accompanying the Summary of a Haystack paper.

unarXive

A data set based on all arXiv publications, pre-processed for NLP, including structured full-text and citation network

RaLLe

RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models

mbr-anomaly

Code for "On the True Distribution Approximation of Minimum Bayes-Risk Decoding," NAACL 2024

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

with_a_little_push

MSciNLI

The code, and data for the NAACL 2024 paper "MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference" will be released here.

py-setproctitle

A Python module to customize the process title

PICL

Code for ACL2023 paper: Pre-Training to Learn in Context

improved-t5

Experiments for efforts to train a new and improved t5

zola

A fast static site generator in a single binary with everything built-in. https://www.getzola.org

tasksource

Datasets collection and preprocessings framework for NLP extreme multitask learning

rank_bm25

A Collection of BM25 Algorithms in Python

pyserde

Yet another serialization library on top of dataclasses, inspired by serde-rs.

WebVOWL

Visualizing ontologies on the Web

rdflib

RDFLib is a Python library for working with RDF, a simple yet powerful language for representing information.

nanoT5

Fast & Simple repository for pre-training and fine-tuning T5-style models

ocr-fileformat

Validate and transform various OCR file formats (hOCR, ALTO, PAGE, FineReader)

orjson

Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy