AIFlow is an open source framework that bridges big data and artificial intelligence. It manages the entire machine learning project lifecycle as a unified workflow, including feature engineering, model training, model evaluation, model service, model inference, monitoring, etc. Throughout the entire workflow, users can choose to use the computing engine like Python, Flink etc.
In addition to the capability of orchestrating a group of batch jobs, by leveraging an event-based scheduler(enhanced version of Airflow), AIFlow also supports workflows that contain streaming jobs. Such capability is quite useful for complicate real-time machine learning systems as well as other real-time workflows in general.
You can use AIFlow to do the following:
-
Define the machine learning workflow including batch/stream jobs.
-
Manage metadata(generated by the machine learning workflow) of date sets, models, artifacts, metrics, jobs etc.
-
Schedule and run the machine learning workflow.
-
Publish and subscribe events
To support online machine learning scenarios, notification service and event-based schedulers are introduced.
AIFlow's current components are:
-
SDK: It defines how to build a machine learning workflow and includes the api of the AIFlow.
-
Notification Service: It provides event listening and notification functions.
-
Meta Service: It saves the meta data of the machine learning workflow.
-
Event-Based Scheduler: It is a scheduler that triggered jobs by some events happened.
You can follow our Quick Start to get your hands on AIFlow quickly. Besides, you can also take a look at our Tutorial to learn how to write your own workflow. You can use AIFlow according to the guidelines of.
Please refer to the Python API to find the details of the API supported by AIFlow.
If you are interested in design principles of AIFlow, please see the Design for more details.
We have provided some examples of AIFlow to help you get a better understanding of how to write a workflow. Please see the Examples directory.
If you encounter any issues please open an issue in GitHub and we encourage you to provide a patch through GitHub pull request as well.
We happily welcome contributions to AIFlow. Please see our Contribution for details.
For more information, you can join the AIFlow Users Group on DingTalk to contact us. The number of the DingTalk group is 35876083.
You can also join the group by scanning the QR code below:
