HDF-AI Tools and Information Center

This is a central hub for exchanging AI solutions in HDF5. It also hosts sample AI data / model files in HDF5.

Ideas

Make Bigdata AI-ready using HDF5

Save metadata with data
- algorithms and their versions used, model parameters, authors, etc.
Save training / model / testing data in hiearchy with groups.
Save knowledge graph (semantic network) in HDF5.

Reduce Bigdata Storage using HDF5 AI Filter

Filtering Bigdata with AI is a solution to reduce the burden of managing a large amount of training and testing data.

HDF5 AI filter can automatically sanitize your data in a scalable manner when you archive data in HDF5. It can save a lot of space by storing only models, not real data.

HDF5 AI Filter can

store everything in hierarchy including algorithms to use and learned models.
link to the raw data for provenance.
set a time to remove raw data and a desired accuracy threshold to prune models.
run several ML algorithms in parallel according to the HDF5's group hierarchy.

FAQ

What is H5AI API? This is a high level API that can perform I/O efficient AI tasks for HDF5 data.

User Guide

H5AI Reference Manual

Data

Cat vs. Non-Cat
Core ML Specification
Joke Generator (Jump to 14:40 in the video.)
- textgenrnn_weights.hdf5
Face Emotion Recognition
A Deep Learning-Based Hybrid Model of Global Terrestrial Evaporation
JFT-3B

Code

Deep Learning for Climiate Modeling Data: specifically, data_helpers.py
Kubeflow HOW-TO
Keras Spark Rossmann Run Example
Deep Learning IO Benchmark
FlexFlow
parallelformers
Keras/TensorFlow - Save Model in HDF5
flowEQ = MATLAB + Python (Keras)

Reference

About

HDF-AI Tools and Information Center