meetshah15 / mage-ai

🧙 Mage is an open-source tool for building and running data pipelines that transform your data.

Intro

Mage is an open-source tool for building and running data pipelines that transform your data.

Here is a sample data pipeline defined across 3 files:

# data_loaders/load_data_from_file.py
@data_loader
def load_csv_from_file():
    return pd.read_csv('default_repo/titanic.csv')

# transformers/select_columns.py
@transformer
def select_columns_from_df(df, *args):
    return df[['Age', 'Fare', 'Survived']]

# data_exporters/export_to_file.py
@data_exporter
def export_titanic_data_to_disk(df) -> None:
    df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI:

New? We recommend reading about blocks and learning from a hands-on tutorial.

Table of contents

1. Quick start
2. Demo
3. Tutorials
4. Core design principles
5. Features
6. Documentation

🏃‍♀️ Quick start

Install Mage using Docker or pip:

Using Docker

Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai mage start demo_project

_{Follow the guide if you want to use PySpark kernel in your notebook.}

Using pip

1. Install Mage

pip install mage-ai

_{For additional packages (e.g. spark, postgres, etc), please see Installing extra packages.}

_{If you run into errors, please see Install errors.}

2. Create new project and launch tool (change demo_project to any other name if you want):

mage start demo_project

Open tool in browser

Open http://localhost:6789 in your browser and build a pipeline.

🎮 Demo

Live demo

Try a hosted version of the tool here: http://demo.mage.ai.

WARNING

The live demo is public, please don’t save anything sensitive.

Demo video (2 min)

_{Click the image to play video}

👩‍🏫 Tutorials

• Train model on Titanic dataset
• Load data from API, transform it, and export it to PostgreSQL
• Integrate Mage into an existing Airflow project

🏔️ Core design principles

Every user experience and technical design decision adheres to these principles.

1. Easy developer experience
2. Engineering best practices built-in
3. Data is a first-class citizen
4. Scaling made simple

_{Read more here.}

🛸 Core abstractions

These are the fundamental concepts that Mage uses to operate.

• Project
• Pipeline
• Block
• Run
• Trigger
• Data product

_{Read more here.}

🔮 Features

• Data pipeline management
• Notebook for building data pipelines

_{Read more here.}

📚 Documentation

Read more here.

🙋‍♀️ Contributing

Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.

🧙 Community

We love the community of Magers (/ˈmājər/); a group of mages who help each other realize their full potential!

To live chat with the Mage team and community, please join the free Mage Slack channel.

For real-time news and fun memes, check out the Mage Twitter.

To report bugs or add your awesome code for others to enjoy, visit GitHub.

🪪 License

See the LICENSE file for licensing information.