AI-powered search engine
txtai executes machine-learning workflows to transform data and build AI-powered text indices to perform similarity search.
Summary of txtai features:
🔎 Large-scale similarity search with multiple index backends (Faiss, Annoy, Hnswlib)📄 Create embeddings for text snippets, documents, audio and images. Supports transformers and word vectors.💡 Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction↪️ ️ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.🔗 API bindings for JavaScript, Java, Rust and Go☁️ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)
NeuML uses txtai and/or the concepts behind it to power all of our Natural Language Processing (NLP) applications. Applications include:
| Application | Description |
|---|---|
| paperai | AI-powered literature discovery and review engine for medical/scientific papers |
| tldrstory | AI-powered understanding of headlines and story text |
| neuspo | Fact-driven, real-time sports event and news site |
| codequestion | Ask coding questions directly from the terminal |
txtai is built with Python 3.6+, Hugging Face Transformers, Sentence Transformers and FastAPI
Installation
The easiest way to install is via pip and PyPI
pip install txtai
You can also install txtai directly from GitHub. Using a Python Virtual Environment is recommended.
pip install git+https://github.com/neuml/txtai
Python 3.6+ is supported. txtai has the following environment specific prerequisites.
Linux
Optional audio transcription requires a system library to be installed
macOS
Run brew install libomp see this link
Windows
Install C++ Build Tools
Examples
The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.
Notebooks
| Notebook | Description | |
|---|---|---|
| Introducing txtai | Overview of the functionality provided by txtai |  |
| Build an Embeddings index with Hugging Face Datasets | Index and search Hugging Face Datasets | |
| Build an Embeddings index from a data source | Index and search a data source with word embeddings | |
| Add semantic search to Elasticsearch | Add semantic search to existing search systems | |
| Extractive QA with txtai | Introduction to extractive question-answering with txtai | |
| Extractive QA with Elasticsearch | Run extractive question-answering queries with Elasticsearch | |
| Apply labels with zero shot classification | Use zero shot learning for labeling, classification and topic modeling | |
| API Gallery | Using txtai in JavaScript, Java, Rust and Go | |
| Building abstractive text summaries | Run abstractive text summarization | |
| Extract text from documents | Extract text from PDF, Office, HTML and more | |
| Transcribe audio to text | Convert audio files to text | |
| Translate text between languages | Streamline machine translation and language detection | |
| Similarity search with images | Embed images and text into the same space for search | |
| Run pipeline workflows | Simple yet powerful constructs to efficiently process data | |
Applications
| Application | Description |
|---|---|
| Demo query shell | Example application for search and indexing |
| Image search | Streamlit image search application |
Documentation
Full documentation on txtai including configuration settings for pipelines, workflows, indexing and the API.
Contributing
For those who would like to contribute to txtai, please see this guide.