mlcommons / cm4mlops

A collection of portable, reusable and cross-platform automation recipes (CM scripts) with a human-friendly interface and minimal dependencies to make it easier to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data sets, software and hardware (cloud/edge)

Unified and cross-platform CM interface for DevOps, MLOps and MLPerf

.

This repository contains reusable and cross-platform automation recipes to run DevOps, MLOps, and MLPerf via a simple and human-readable Collective Mind interface (CM) while adapting to different operating systems, software and hardware.

All СM scripts have a simple Python API, extensible JSON/YAML meta description and unified input/output to make them reusable in different projects either individually or by chaining them together into portable automation workflows, applications and web services adaptable to continuously changing models, data sets, software and hardware.

We develop and test CM scripts as a community effort to support the following projects:

• CM for MLPerf: modularize and automate MLPerf benchmarks (maintained by MLCommons and originally developed by cKnowledge.org, OctoML and cTuning.org)
• CM for research and education: provide a common interface to automate and reproduce results from research papers and MLPerf benchmarks (maintained by cTuning foundation and cKnowledge.org)
• CM for ABTF: provide a unified CM interface to run automotive benchmarks (maintained by MLCommons and originally developed by cKnowledge.org)
• CM for optimization: co-design efficient and cost-effective software and hardware for AI, ML and other emerging workloads via open challenges (maintained by cKnowledge.org)

You can read this ArXiv paper to learn more about the CM motivation and long-term vision.

Please provide your feedback or submit your issues here.

Catalog

Online catalog: cKnowledge, MLCommons.

Citation

Please use this BibTeX file to cite this project.

A few demos

Install CM and virtual env

Install the MLCommons CM automation language.

Pull this repository

cm pull repo mlcommons@cm4mlops --branch=dev

Run image classification using CM

cm run script "python app image-classification onnx _cpu" --help

cm run script "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e
cm run script "python app image-classification onnx _cpu" --input=computer_mouse.jpg

cmr "python app image-classification onnx _cpu" --input=computer_mouse.jpg
cmr --tags=python,app,image-classification,onnx,_cpu --input=computer_mouse.jpg
cmr 3d5e908e472b417e --input=computer_mouse.jpg

cm docker script "python app image-classification onnx _cpu" --input=computer_mouse.jpg

cm gui script "python app image-classification onnx _cpu"

Re-run experiments from the ACM/IEEE MICRO'23 paper

Check this script/reproduce-ieee-acm-micro2023-paper-96.

Run MLPerf ResNet CPU inference benchmark via CM

cm run script --tags=run-mlperf,inference,_performance-only,_short  \
   --division=open \
   --category=edge \
   --device=cpu \
   --model=resnet50 \
   --precision=float32 \
   --implementation=mlcommons-python \
   --backend=onnxruntime \
   --scenario=Offline \
   --execution_mode=test \
   --power=no \
   --adr.python.version_min=3.8 \
   --clean \
   --compliance=no \
   --quiet \
   --time

Run MLPerf BERT CUDA inference benchmark v4.1 via CM

cmr "run-mlperf inference _find-performance _full _r4.1" \
   --model=bert-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker \
   --docker_cm_repo=mlcommons@cm4mlops \
   --docker_cm_repo_flags="--branch=mlperf-inference" \
   --test_query_count=100 \
   --quiet

Run MLPerf SDXL reference inference benchmark v4.1 via CM

cm run script \
	--tags=run-mlperf,inference,_r4.1 \
	--model=sdxl \
	--implementation=reference \
	--framework=pytorch \
	--category=datacenter \
	--scenario=Offline \
	--execution_mode=valid \
	--device=cuda \
	--quiet

License

Apache 2.0

Acknowledgments

We thank cKnowledge.org, cTuning foundation and MLCommons for sponsoring this project! We also thank all volunteers, collaborators and contributors for their support, fruitful discussions, and useful feedback!