Instill VDP

Build, test and share your pipelines - No code required!

💧 Instill VDP (Versatile Data Pipeline) is a source-available tool, designed to streamline your data processing pipelines from inception to completion. If your goal is to develop versatile AI features using Large Language Models (LLM), Generative AI, Vision, or Audio models, Instill VDP empowers you to:

Effortlessly connect to your unstructured data
Build pipelines to enable diverse AI functionalities in your applications
Visually test pipelines with a single click, viewing output at each stage
Easily share your pipelines to showcase your work

☁️ Instill Cloud offers a fully managed public cloud service, providing you with access to all the fantastic features of Instill VDP without the burden of infrastructure management.

Highlights

🚀 Accelerate AI applications by building end-to-end AI-powered pipelines for unstructured data up to 10 times faster
🔌 Utilize pre-built connectors to access data from various sources, powerful AI models, and third-party tools
🌟 Benefit from a no-code drag-and-drop pipeline builder, enabling quick and customizable application development
🪢 Choose between real-time inference (SYNC) and on-demand workload (ASYNC) processing modes
🧁 Enjoy a scalable API-first microservice design, offering an excellent developer experience
⚡️ Leverage high-performing backends implemented in Go
📊 Gain visibility into pipeline performance through a detailed dashboard
🤠 Access no-code/low-code interfaces, making VDP suitable for every AI and data practitioner

Prerequisites

macOS or Linux - VDP works on macOS or Linux, but does not support Windows yet.
Docker and Docker Compose - VDP uses Docker Compose (specifically, Compose V2 and Compose specification) to run all services locally. Please install the latest stable Docker and Docker Compose before using VDP.

Quick Start

Note Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.

Execute the following commands to start pre-built images with all the dependencies:

The stable release version

$ git clone -b v0.18.0-alpha https://github.com/instill-ai/vdp.git && cd vdp

# Launch all services
$ make all

The latest version for development

$ git clone https://github.com/instill-ai/vdp.git && cd vdp

# Launch all services
$ make latest PROFILE=all

🚀 That's it! Once all the services are up with health status, the UI is ready to go at http://localhost:3000!

To shut down all running services:

$ make down

Explore our documentation to discover all available deployment options.

Dive into the Projects

Explore our open-source unstructured data infrastructure stack, comprising a collection of source-available projects designed to streamline every aspect of building versatile AI features with unstructured data. Dive into the potential in our documentation.

Open Source Unstructured Data Infrastructure Stack

🔮 Instill Core: The starting point for self-hosting Instill VDP and Instill Model

Instill Core, or Core, serves as the bedrock upon which open-source unstructured data stack thrive. Essential services such as user management servers, databases, and third-party observability tools find their home here. Instill Core also provides deployment codes to facilitate the seamless launch of both Instill VDP and Instill Model.

💧 Instill VDP: AI pipeline builder for unstructured data

Instill VDP, or VDP (Versatile Data Pipeline), represents a comprehensive unstructured data infrastructure. Its purpose is to simplify the journey of processing unstructured data from start to finish:

Extract: Gather unstructured data from diverse sources, including AI applications, cloud/on-prem storage, and IoT devices.
Transform: Utilize AI models to convert raw data into meaningful insights and actionable formats.
Load: Efficiently move processed data to warehouses, applications, or other destinations.

Embracing VDP is straightforward, whether you opt for Instill Cloud deployment or self-hosting via Instill Core. Consult our comprehensive documentation to delve into VDP deployment.

⚗️ Instill Model: Scalable AI model serving and training

Instill Model, or simply Model, emerges as an advanced ModelOps platform. Here, the focus is on empowering you to seamlessly import, train, and serve Machine Learning (ML) models for inference purposes. Like other projects, Instill Model's source code is available for your exploration.

No-Code/Low-code Access & Support

To dive into Instill Core and Instill Cloud, we provide:

✨ Console for non-developers, empowering them to dive into AI applications and process unstructured data without any coding.
🧰 CLI and SDKs for developers to seamlessly integrate with their existing data stack in minutes.
🙌 Join our Discord server to exchange ideas about unstructured data processing, AI, MLOps, and get support from the Instill AI team. We're here to support you every step of the way!

Documentation

📔 Documentation - Check out the documentation website.
📘 API Reference - Access http://localhost:3001 after make all, or simply run make doc to access the OpenAPI documentation. The gRPC protocols in protobufs provide the single source of truth for the VDP APIs. The genuine protobuf documentation can be found in our Buf Scheme Registry (BSR).

Be Part of Our Community

We strongly believe in the power of community collaboration and deeply value your contributions. Head over to our Community repository, the central hub for discussing our open-source projects, raising issues, and sharing your brilliant ideas.

Contributing

Please refer to the Contributing Guidelines for more details.

Big thanks to these amazing contributors!

License

See the LICENSE file for licensing information.

tidydee / vdp