[2021 Theme Proposal] Improve and expand IPFS to global hyper scale distributed content storage and real time delivery
RhettSampson opened this issue · comments
Note, this is part of the 2021 IPFS project planning process - feel free to add other potential 2021 themes for the IPFS project by opening a new issue or discuss this proposed theme in the comments, especially other example workstreams that could fit under this theme for 2021. Please also review others’ proposed themes and leave feedback here!
Theme description
Improve and expand IPFS from a distributed content storage and retrieval system to a hyper scale distributed content storage and real time delivery network with significantly improved efficiency and QoS.
It is generally agreed that the IPFS protocol stack needs improvement in order to support the hyper scale of the future Internet. Theme proposals #76, #69 and #61 speak to this, as do Protocol Labs R&D RFPs https://github.com/protocol/research-RFPs/blob/master/RFPs/rfp-7-MLDHT.md and https://github.com/protocol/research-RFPs/blob/master/RFPs/rfp-8-pubsub.md to which we responded. While IPFS and Filecoin are great distributed storage systems, they need work to reach hyper scale and to improve delivery.
Hypothesis
Our hypothesis is that IPFS can be expanded on, in order to improve scaling of IPFS content storage; and to add real time content delivery; using name-based routing design principles, particularly Name Domain Networking (NDN); combined with hierarchical AI/ML; and some innovations. The hypothesis is supported by the links below to a SIGCOMM ACM-ICN paper and to our proposed system architecture repo.
We believe there are a number of very achievable key factors in doing this:
o Expansion of IPFS DHT and TCP/IP based name resolution routing (NRR) with name-based routing (NBR), especially Named Domain Networking (NDN) design principles for timely delivery;
o expansion of the DHT to multi-level with context
o combined with a new, unified naming system, unifying NRR and NBR;
o and a new, unified, universal discovery system; with:
o intelligent machine learning hybrid adaptive routing for storage and delivery (AI-HARD);
o in-network storage (the network IS the cloud);
o scalable publish/subscribe (pub/sub);
o integrated and embedded security;
o a hierarchical, pluggable AI framework with an associated, scalable simulation, training, and development pipeline that embeds AI agents with varying degrees of awareness and optimization capabilities at peer, edge, and core network levels;
o self-optimising local ML/AI agent swarms;
o feeding operational and routing information up to a data lake;
o with global optimising ML/AIs;
o in a fully containerised, distributed, virtual network architecture.
Vision statement
If successful, this would transform IPFS and Filecoin from a great first implementation of a distributed storage system and market to a hyper scalable distributed storage and delivery network with vastly improved efficiency and QoS compared to the current Internet. It would be a viable candidate for the next generation of the Internet.
Why focus this year
The timing is perfect. IPFS and Filecoin have reached their initial milestones but require improvement. The required component technologies and platforms have reached levels of maturity that make implementation achievable. In particular, the development of open switch/route/AI platforms has reached an inflection point. Combining the decade of R&D by Protocol Labs in distributed storage protocols (IPFS stack) with GT Systems’ decade of R&D into real time content delivery network protocols (SPAN-AI stack) will produce a Unified Content Delivery Network (UCDN) that is capable of content storage and delivery on a global scale.
Example workstreams
We propose a mix of funded and community R&D. At the highest level, the workstreams would be as described below and in more detail in our system architecture at the GitHub page below. The algorithms should be modelled initially and then implemented on open, production switch development platforms to demonstrate and prove real world operability. We have interest from both Nokia and Nvidia to do this on their platforms. We would love to work with the Protocol Labs community to build this. We are more than happy to discuss how we achieve that, including releasing critical, core parts of the Intellectual Property into the public domain.
High level project milestones and deliverables:
o Modelling, optimisation and simulation in Matlab and appropriate simulation tools. Initial proof of efficiency and QoS. Scaling of containerised agent models to demonstrate hyper scale in cloud architectures.
o Build the sub-systems described in the systems architecture below;
o Development of a Proof of Routing and associated token economy:
o Implementation of the integrated TCP/IP, IPFS and SPAN-AI protocol stacks on
a Nokia SR Linux datacentre switch (scalable to big iron core switches)
Nvidia ARM/Mellanox/Cumulus-Linux AI core and edge switches (DGX/EGX)
Possible x86 SONIC/SAI implementation to extend reach
o Build demo: a fully distributed video origin store and delivery service
Other content
LINKS
SPAN-AI white paper and detailed system architecture on GitHub
https://github.com/RhettSampson/IPFS-SPAN_AI-UCDN
Ascigil et al: Towards Peer-to-Peer Content Retrieval Markets: Enhancing IPFS with ICN.
https://conferences.sigcomm.org/acm-icn/2019/proceedings/icn19-34.pdf
Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:
- "Priority" labels will show how urgent this is for the team.
- "Status" labels will show if this is ready to be worked on, blocked, or in progress.
- "Need" labels will indicate if additional input or analysis is required.
Finally, remember to use https://discuss.ipfs.io if you just need general support.
Theres more info here as part of a good discussion #81 (comment)
We are starting to get traction for SPAN-AI and UCDN with some very large, innovative and interesting global network operators and projects. There may be an opportunity to link these back with IPFS development. I will post updates here.
We are starting to get traction for SPAN-AI and UCDN with some very large, innovative and interesting global network operators and projects. There may be an opportunity to link these back with IPFS development. I will post updates here.
@RhettSampson : no expectation here, but I'm curious if this progressed at all?
We are starting to get traction for SPAN-AI and UCDN with some very large, innovative and interesting global network operators and projects. There may be an opportunity to link these back with IPFS development. I will post updates here.
@RhettSampson : no expectation here, but I'm curious if this progressed at all?
@BigLep Steve, to say yes would be the understatement of the century! ;) We have been selected as next gen CDN for https://www.laserlightcomms.com/ and are about to do PoCs for them and a large North American telco. We are in investment discussions with them and doing a PoC raise that is getting very positive response. Our federated UCDN concept is generating significant interest with major global telcos and datacentre X-connect NaaS providers https://gtsystems.io/the-federated-universal-content-distribution-network/ and we are in discussions with a leading next gen CDN who share our same base innovation to participate in the PoCs and take our SPAN-AI tech. We are confident we can unite Filecoin and other blockchains with distribution of content such as movies and XR in the next gen Internet. Lots more info on our new website https://gtsystems.io/ Love to have a chat.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.