The Collective Knowledge project is motivated by our tedious experience reproducing experiments from 150 research papers and validating them in the real world. We decided to collaborate with the community and MLCommons to develop a free, open-source and technology-agnostic platform that can help everyone reproduce, optimize and compare any novel technology across any rapidly evolving AI models, software, hardware and data(sets) from different vendors in an automated way via collaborative challenges and reproducible experiments.
This platform is powered by the Collective Mind workflow automation framework (CM aka CK2) - the 2nd version of the CK framework originally designed by the cTuning foundation and donated to MLCommons in 2022. It is being developed by the open MLCommons taskforce to help users of the CK platform solve the "dependency hell" and interconnect diverse and rapidly evolving software and hardware from any company including Nvidia, Intel, Qualcomm, AMD, Microsoft, Amazon, Google, Neural Magic, Meta, OctoML, Krai, cKnowledge and Hugging Face in a transparent and non-intrusive way using portable CM scripts developed by the community.
For example, CK has already helped to automate more than 80% of all recent MLPerf inference benchmark submissions (and 98% of all power results), make them more reproducible and reusable, and obtain record inference performance on the latest Qualcomm and Nvidia devices.
The long-term goal for our Collective Knowledge platform is to help everyone automatically generate the most efficient, reproducible and deployable solutions for their applications using the most suitable software and hardware stack at any given time (model, framework, inference engine and any other related dependency) based on their requirements and constraints including costs, throughput, latency, power consumption, accuracy, target devices (cloud/edge/mobile/tiny), environment and data.
Our ultimate dream is to accelerate deep-tech innovation and help AI, ML and systems developers by automating all their tedious and repetitive tasks and slashing development, benchmarking, optimization, deployment and operational costs for any novel technology by 10..100 times in the rapidly evolving world.
Join our Discord server to learn more about our technology, participate in public developments and discussions, and request platform features and support for your technology.
2021-2023 MLCommons
Apache 2.0
This open-source technology is being developed by the open MLCommons taskforce led by Grigori Fursin and Arjun Suresh.
This project is currently supported by MLCommons, cTuning foundation, cKnowledge and individual contributors. We thank HiPEAC and OctoML for sponsoring initial development.