IPFS to store videos
alxlg opened this issue · comments
I think that what limits PeerTube adoption is that instances are perfect for personal/organization use but not to build a free service like YouTube where everyone can upload videos without limits. The issue is that storage has a cost and videos make the necessary storage grow quickly.
IPFS (InterPlanetary File System) can be used to solve the storage issue because every user can store the files by himself but it doesn't have a way to browse and interact with them. PeerTube instead has an awesome UI that can be used by everyone.
Would it be possible to combine PeerTube and IPFS? Ideally the instance administrator would limit the classic upload for each user but eventually let users upload videos by specify an IPFS address. I guess when a second user browse a PeerTube instance and want to watch a video hosted on IPFS, PeerTube provides it by reading from IPFS and not from its local storage. PeerTube instances would cache IPFS contents like IPFS users and admins would monitor IPFS cache impact on their storage. If a PeerTube user wants to be sure its video is available he just have to keep it on IPFS with his machine. This could have another advantage: if the used PeerTube instance won't be available anymore its users won't need to upload videos on other PeerTube instances if they are on IPFS: they would just "upload" the IPFS addresses.
I will be grateful to those who answer by denying or confirming my assumptions.
Ideally the instance administrator would limit the classic upload for each user but eventually let users upload videos by specify an IPFS address. I guess when a user […] wants to watch a video hosted on IPFS, PeerTube provides it by reading from IPFS and not from its local storage.
@alxlg indeed that's more or less how I envisioned the potential use of IPFS. But then there's the fact an IPFS endpoint is not a webseed[¹] nor holds versions of different quality. In other words, IPFS would only be a second class citizen feature-wise.
¹: let me extend a bit on that issue. The fact is that we use WebTorrent (BitTorrent/WebRTC) on the client side to watch videos. It provides a handy pool of content seeders and direct browser connection. Watching a video via IPFS would mean to replace entirely that component with an IPFS client in the browser. So it's not just thinking of a different storage/upload mechanism.
If you have any ideas as to how to solve these problems, I'm all ears :)
P.S.: we have also not heard a lot about IPFS performance-wise when it comes to streaming videos.
@rigelk thanks for your reply!
I had not thought of different quality versions of videos. Since IPFS is really low in the stack the only solution I can think of is storing a different IPFS file for each version. The user should be able to specify the IPFS address for each quality version he wants to maintain... This doesn't seem user-friendly but with a desktop client that automatically manage versions on IPFS it could gain adoption... Ideally the desktop client could use some API to upload the video to PeerTube by specifying many IPFS addresses. Desktop client's users should just pick a video from their HDD and the desktop client would generate different versions, upload them to IPFS and load the addresses to a PeerTube instance.
It seems like a big amount of work but promising to me and the idea could get many contributors.
@alxlg we're not even close to writing a Desktop client. This is a non-option considered our resources.
I was considering leaving the video uploaded without transcoding, thus leaving a single quality available. It's always better than no video at all.
But now that I come to think of it, about ¹: do we even have to replace the webtorrent client for ipfs videos? If we could manage to mark ipfs endpoints as WebSeed, we could just use them under the hood of webtorrent by making the webtorrent client aware of them.
@rigelk in fact I did not intend to replace the video player. I thought that the server could run both PeerTube and a IPFS node, and the PeerTube instance see the file cached with IPFS like local files... I hope it has sense now...
This feature doesn't depend on a desktop client, but it would just help normal PeerTube users to automatically store their videos locally.
I would be happy to store some videos with IPFS on my HDD without a PeerTube instance that need much more maintenance.
I think the change would mostly be in PeerTube UI providing a way to upload a video through a IPFS address and not uploading the entire file to the PeerTube instance. Of course the server admin should configure it to run IPFS...
Would not a webRTC torrent app running in the users PC do the same thing when she/he leave it seeding the video? This also allows user to create "seedboxs" using RSS auto downloading torrent apps. Easy simple/KISS way of "distributing" the video hosting.
@Openmedianetwork good point! I think your proposal is easier to implement but using IPFS too could have some advantages. For example, I'm pretty sure that if an instance of PeerTube is no longer available an user can reupload his/her IPFS videos on another instance just sharing IPFS addresses, much better than reuploading video files! Do you think this could be achieved with torrents/WebTorrents too? Would changes to PeerTube be needed? Maybe uploading a *.torrent file instead of a video file?
issues with testing webrtc http://hamishcampbell.com/index.php/2018/05/18/testing-peertube-webrtc-seedboxes/
@Openmedianetwork this has nothing to do with IPFS. Please find a related issue and detail your problem there, not in a blog post.
Sorry this was a update for alexig that my suggestion for webRTC torrent as an alternative for seeding dues not appear to work after actually testing. Will start a new thread after further of tests.
Closing this issue since we have no really use cases for IPFS for now.
@rigelk
@Chocobozzz
IPFS can serve as a backup for local storage and dedicated seeding pools, as it effectively transfers each newly added file to an entire pre-existing network of 300+ peers from the get-go. In such an instance, PeerTube might not necessarily even have to double as a WebRTC-based IPFS node but simply run along side a regular one (which in-itself can be optional), first to ensure the files initially propagate through the network and secondly to provide an optional method for retrieving them through a local gateway. However simply linking to an IPFS address should suffice if it would be possible to configure a PeerTube instance to use external public gateways for retrieval.
In a case where a PeerTube instance should go down with no pre-existing seeding pools in place, as long as the videos are still present on IPFS, it should be possible to retrieve them by simply following each video's address (that presumably was shared beforehand with other federated instances). This way each video will remain accessible and therefor could be later conventionally reseeded via a different instance.
As a by-product, if it'll be possible to authenticate each user's identity, perhaps it might also be possible to use this method for transferring channels between different PeerTube instances.
@Chocobozzz shouldn't this be reopened? It seems what @NightA mentioned was a pretty good idea.
I'm actually interested in implementing this, but I think a roadmap should be discussed.
What I propose:
Phase 1: Server uploads to IPFS and stores hashes of videos
This phase has the potential of requiring double the disk space since the files will be stored on disk normally, then uploaded to IPFS and pinned. In order to prevent that, ipfs-fuse could be used. That will allow mounting ipfs to a directory and designating all videos to be stored there / storing them in one place : IPFS.
I assume there's a json or a table in the db with the videos, where a field or column for the hashes of the video files can be added.
Phase 2: Import from IPFS
In this phase, the user will have the option to provide a hash that the server can download and process. Maybe it's possible to check the filetype before downloading it in order to save bandwidth, I dunno.
If phase 1 is done intelligently, hashes already present in the db will be rejected e.g fil
If phase 1 is done intelligently, hashes of videos already in the db can be rejected since that would create an unnecessary duplicate. But if another server uses the same hash with different properties (title, description) that might not be good. Up for discussion
Phase 3: Syncing with other instances and downloading hashes
This is separated from Phase 2 only if the code is in a separate area. Importing from IPFS might be a different procedure from receiving a video file another instance.
Since I don't know what data is sent over activity pub, I assume it's either the torrent or a link for a server-to-server API call, which include information about the video e.g title, description, resolutions and (of course) hashes.
Phase 4: (wishful thinking) ipfs:// links for videos
Users running their own IPFS nodes with IPFS companion could then stream using IPFS.
I haven't actually done a lot of research into this, so I don't know if it's possible. Maybe plugin for videojs would be necessary - I dunno.
End goal
Instances running IPFS nodes and using that to download and pin hashes, which would allow :
- greater resilience to takedowns or simple storage failure
- additional data sources since users could host the data too
- possibly less bandwidth consumption if Phase 4 actually is possible and is done
Server uploads to IPFS and stores hashes of videos
This phase has the potential of requiring double the disk space since the files will be stored on disk normally, then uploaded to IPFS and pinned. In order to prevent that, ipfs-fuse could be used. That will allow mounting ipfs to a directory and designating all videos to be stored there / storing them in one place : IPFS.
Where is it uploaded exactly? A third party (a pinning service?)?
ipfs-fuse requires a the Go IPFS runtime alongside, so this will complexify the deployment.
Syncing with other instances and downloading hashes
This is trivially done by adding another Link object in the ActivityPub Video.url field.
Users running their own IPFS nodes with IPFS companion could then stream using IPFS.
What bothers me is that users are expected to have this extension and a running IPFS Go runtime.
I haven't actually done a lot of research into this, so I don't know if it's possible. Maybe plugin for videojs would be necessary - I dunno.
I haven't found any videojs plugin for ifps, or its companion extension.
Instances running IPFS nodes and using that to download and pin hashes, which would allow :
* greater resilience to takedowns or simple storage failure * additional data sources since users could host the data too * possibly less bandwidth consumption if Phase 4 actually is possible and is done
- resilience can be achieved in IMHO simpler ways. Right now what makes a video's origin instance disappearance a problem is that the BitTorrent tracker is the instance. Sharing the video over DHT alongside would already solve the problem, without changing our infrastructure.
- users can already host the data, with a webtorrent-compatible torrent client.
- less bandwidth consumption is already achieved with WebTorrent: people watching share their bandwidth (and not just those with an extension)
- less bandwidth consumption is already achieved with WebSeeds/replication, with less impact to the buffering speed.
Where is it uploaded exactly? A third party (a pinning service?)?
It's pinned locally and uploaded when somebody else requests it over IPFS. If someone does so over a third party, then they'll have it, but not pinned.
ipfs-fuserequires a the Go IPFS runtime alongside, so this will complexify the deployment.
That depends. It's very possible to do that in stages too:
- stage 1: tell admin that IPFS has to be installed
- stage 2: provide config interface for admin to target a IPFS node of choice (IP:port of IPFS HTTP API used by
ipfs-fuse) - stage 3: provide option to install IPFS for the admin
What bothers me is that users are expected to have this extension and a running IPFS Go runtime.
This is not a proposal to force users to use IPFS, merely to give users the option. Right now they have the opportunity to use HTTP or webtorrent. This would merely be another one
I haven't found any videojs plugin for ifps, or its companion extension.
Yes, that would have to be developed if necessary. The companion provides the useful feature of redirecting URLs to IPFS instances :
Requests for IPFS-like paths (
/ipfs/{cid}or/ipns/{peerid_or_host-with-dnslink}) >>are detected on any website.
If tested path is a valid IPFS address it gets redirected and loaded from a local gateway, e.g:
https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
→http://127.0.0.1:8080/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
I assume we use HLS for streaming, so serving a different .m3u8 playlist with /ipfs links would be the only work required.
Your final points are valid, but I don't see the harm in providing an additional option. In case you missed it, I am willing to implement this, so your workload would be reduced to code-reviews and handling future bugs (since no code is perfect).
Of course, if you are firmly against having it in the original code base, I'll investigate if a plugin can be written and if not, simply fork it.
This is not a proposal to force users to use IPFS, merely to give users the option. Right now they have the opportunity to use HTTP or webtorrent. This would merely be another one more.
Your final points are valid, but I don't see the harm in providing an additional option.
👍
In case you missed it, I am willing to implement this, so your workload would be reduced to code-reviews and handling future bugs (since no code is perfect).
I am not sure we could handle the future bugs part, especially when dealing with technologies we don't use regularly. And since merging code means taking responsibility in its maintainance…
Of course, if you are firmly against having it in the original code base, I'll investigate if a plugin can be written and if not, simply fork it.
I would suggest waiting for @Chocobozzz to answer about that - regarding plugins, not everything can be changed via their API. Depending on how many and where changes to the codebase are required, the plugin API could be expanded to facilitate your changes.
Now that being said, implementing it directly in the codebase at first is not a bad idea. It is not time lost, as this will serve as a POC and help us understand the reach of the needed changes - and thus the potential changes to the plugin API.
Where is it uploaded exactly? A third party (a pinning service?)?
To add a bit more details to what @LoveIsGrief mentioned, when a file is added to IPFS it is given a UnixFX structure, cryptographically signed, hashed, registered as a content-identifier (CID) in an IPLD, broken into sets of Merkle-DAG blocks and then distributed peer-to-peer using a DHT.
* resilience can be achieved in IMHO simpler ways. Right now what makes a video's origin instance disappearance a problem is that the BitTorrent tracker is the instance. Sharing the video over DHT alongside would already solve the problem, without changing our infrastructure.
From what i understood, unlike the DHTs utilized with torrents, the IPFS network doesn't focus on seeding each file individually rather focuses on distributing the individual file-blocks themselves among many peers. In which case, a typical IPFS nodes doesn't really "seed" individual files, rather only temporarily caches blocks of various others and only stores complete sets of blocks for specific files when those are explicitly pinned. Otherwise, the cache gets garbage collected and erased after a specific time period that's configured on each individual node.
So unlike with a PeerTube instances that goes under with a specific torrent which happened to be seeded from one specific location (the instance itself), once the same file gets cached throughout enough IPFS nodes, it has this sort-of a grace-period for being retrieved. During that period it has the chance of being saved/pinned on another IPFS node or be imported unto another hypothetical PeerTube instance that supports retrieval from IPFS.
The IPFS node in this regard also doesn't have to have anything to do with PeerTube instances to begin with, as it simply provides the files as long as there's someone requesting them.
TL;DR - Torrent DHT's only replace a tracker and in that regard only point to files that may or may not be seeded anymore. IPFS provides a P2P CDN of sorts that can cache those files independently of their initial seeding PeerTube instance, thus preserves them over a pool that operates independently and is not restricted to any particular instance and/or file.
* users can already host the data, with a webtorrent-compatible torrent client.
In practice users who just consume content tend not to do so, unlike IPFS nodes who do so from the get-go upon request. E.g, while the initial "seed" has to come from an IPFS node running along with a PeerTube instance, once the file had propagated through multiple requests, it can be retrieved from other non-associated nodes within a given time-span.
That being said, yes, to ensure the file doesn't disappear from the network there has to be an IPFS node somewhere that pinns some of the content from the aforementioned PeerTube instance, which is a concept that sounds similar to a basic seedbox.
However considering the pre-existing given network of peers that can automatically participate, this gives more chances in terms of availability for those files.
* less bandwidth consumption is already achieved with WebTorrent: people watching share their bandwidth (and not just those with an extension)
With IPFS there are also dedicated nodes that run on Independent servers/VPS's who distribute the content, in addition to users who just seed some of it for the duration of its run and then move along.
So essentially if a video can be streamed from each public IPFS gateway within a pre-defined list, all the PeerTube instance has to do to offload traffic is just pick and point to one of those IPFS "edge" gateways and serve the page to the user as usual.
This phase has the potential of requiring double the disk space since the files will be stored on disk normally, then uploaded to IPFS and pinned. In order to prevent that, ipfs-fuse could be used.
You can just use --nocopy option on add. And --raw-leaves if you want to store it only in ipfs and have same hash as with --nocopy.
ipfs add --nocopy file_name
ipfs add --nocopy -r directory_name
Additionally if you use --chunker=rabin different files will share same parts in it.
ipfs add --nocopy --chunker=rabin file_name
ipfs add --nocopy -r --chunker=rabin directory_name
In torrent file can be added local webseed http://127.0.0.1:8080/ipfs/{cid(hash) of file or directory}. Torrent clients will use it if it will be available.
Webtorrent can fetch public gateways and replase ":hash" with {cid(hash) of file or directory} from local webseed link. And use that gateways as alternative webseeds.
You can add a comment to torrent file with options that used when file added in ipfs. And user than can re add it to ipfs with this options and get same root hash.
Any updates on this issue?
IPFS is very important because it's an efficient tool against censorship.
I don't think IPFS support can be added to Peertube that easy. My suggestion would be to create a different client-side intended to use IPFS network-based backend. Adding IPFS to Peertube as it would add several compatibility issues and increase code complexity. Also may be a PoC about IPFS video streaming with benchmarking should be done first?
Can someone explain to me why it makes sense to store videos on IPFS beyond being able to upload/publish them by IPFS address? You're backing a distributed video storage system with another distributed storage system. IPFS works great with small files that people are likely to rehost within some other system, but in PeerTube it just duplicates the necessity of clients sharing video over webtorrent to other peers. It could help simplify the process of sharing videos with other instances but webtorrent already does that. There's some sense to using a Sia host as a backend for storing data within an instance, as an alternative to just the filesystem or S3, but there isn't much advantage (at least not enough to justify the increased complexity) for something as involed as IPFS.
At now time all public IPFS gateways can be auto webseeds for webtorrent. Public Gateways will be cache proxy between IPFS peers and Webtorrent peers.
IPFS is better than webtorrent. You can rename file, chage it container and cut part of video without reencoding and IPFS will share unchanged chunks with original file.
Example:
I changed container from mp4 to mkv.
ffmpeg -i video.mp4 -c copy video.mkv
And added both files to IPFS with chunker rabin
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mp4
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mkv
And they have 95% same blocks. And will be 95% source for each other. Each file size is 1.3 GB.
@delbonis you can't seed WebTorrent without a Web browser and it's impossible to do so with PeerTube i.e. you can't take a list of PeerTube URLs, download the videos and tell to your PC to seed them for other PeerTube users.
Recently after years of work libtorrent added WebTorrent support so maybe we will see torrent clients supporting it too. Then PeerTube will need some kind of "seed this video from you torrent client of choice" button.
There's some sense to using a Sia host as a backend for storing data within an instance, as an alternative to just the filesystem or S3, but there isn't much advantage (at least not enough to justify the increased complexity) for something as involed as IPFS.
In that perspective, Tardigrade, a decentralized cloud storage platform, could also be a good option to serve as an S3-like storage bucket. Not the same as IPFS though, but still interesting, maybe to implement via a plugin
I'm just mentioning it so that you know this decentralized cloud platform exists and might be interesting 🙂
Hi, you can upload from browser directly with js-ipfs, we have a PoC in ipfs-upload
Regards 🆙
I've been experimenting with this, here's a player that kinda works on PeerTube's existing HLS files after importing them into ipfs with --nocopy. This requires a lot of extra CPU to do the hashing on import but it doesn't use much extra disk space.
For what it's worth there is a ton of hype around IPFS and a ton of use cases that make no sense, so I should explain what brought me here:
I was interested in having a more unified data model between the "peer" part of peertube and the actual video storage, and for both of those parts to just generally work a lot better.
Drilling down into the technical details, a big decision is how the data gets hashed. In the current system, we use either WebTorrent (hash everything once the content is fully uploaded) or the novage p2p loader, which is very YOLO and trusts everyone. By contrast, IPFS hashes data in blocks that default to 256kB, max 1MB, which among other things lets processing and replication start before a video upload or stream is finished.
I was investigating architectures that let us keep a consistent data model between different parts of the network (clients, small servers, clustered servers, desktop apps, various caches) and IPFS came up. It's interesting because it solves the P2P problems we were solving on the client in broadly similar but specifically more competent ways.
I've been experimenting with this, here's a player that kinda works on PeerTube's existing HLS files after importing them into ipfs with --nocopy. This requires a lot of extra CPU to do the hashing on import but it doesn't use much extra disk space.
Hi i was testing this https://github.com/moshisushi/hlsjs-ipfs-loader and made one hybrid version to do with p2p in the client too, that could be interesting to share two p2p platforms one in IPFS and another in the client, but this doesnt work yet i need some IPFS pending implementations in JS client version, best regards 👌
hlsjs-ipns-p2p-loader.js.txt
I'm not sure what you mean by "two p2p platforms" here? If you mean retaining compatibility with webtorrent or with the novage loader, that may be doable but I'm not sure the existing systems are working well enough that it's useful to retain compatibility with them.
I'm using that same moshisushi/hlsjs-ipfs-loader project but I forked it to fix some bugs, that's probably what you're running into.
Another note on storage overheads: I have about 4.5 TB loaded in so far, and the additional disk usage is only about 8GB. But the memory usage has been consistently fairly high, about 8GB on this system. That's mostly for the DHT i suspect.
One idea that's been kicking around in my head is to build a media server (possibly based on this, https://gitlab.com/valeth/javelin) that can take live streams in/out as well as handling long term storage. One way to do the storage may be to have a very lightweight built-in IPFS server which would manage the hash database and communicating content with other servers as well as with clients that have native ipfs support. Clients would still get web video from the media server side of that daemon over whatever formats they need.
I'm still interested in finding a storage strategy for this which can unify the two formats we currently use (original webtorrent and HLS fmp4) as well as providing a path to introduce future formats without compounding the storage overhead further. The DAG data structure of IPFS may make it possible to do some kinds of container conversions in a way that references chunks of the original data rather than copying it? That's something I still need to experiment with.
And added both files to IPFS with chunker rabin
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mp4 ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mkvAnd they have 95% same blocks. And will be 95% source for each other. Each file size is 1.3 GB.
That sounds very impressive! Have you calculated the overall space savings though? That's such a small block size that I'd expect the parent nodes that point to all the chunks to themselves require a huge amount of space. Is it enough to make up for avoiding duplicating the video content?
I'd expect that, best case, rabin identifies all the best block boundaries and separates the audio/video streams from the container perfectly at each packet, now you have to store as metadata a reference tree with several hashes per packet.
I'd like to explore this still maybe, but I'm also interested in a way to teach some kind of ipfs-compatible-ish server, possibly either the real go-ipfs, or a light ipfs implementation that's part of a media server, to do remuxing or real-time transcoding to recreate blocks that don't exist. I'm wondering if this could work internally with a mechanism similar to how --nocopy is implemented, where the database stores pointers to filesystem data. In this case the database would store enough information to quickly remux or transcode a section of the input file. This wouldn't prevent having to do the transcode or remux once at first though, in order to determine what the hashes are. It would just be a way to garbage-collect resolutions that aren't used much and then regenerate them quickly as they're needed again.
FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)
I'm not sure what you mean by "two p2p platforms" here? If you mean retaining compatibility with webtorrent or with the novage loader, that may be doable but I'm not sure the existing systems are working well enough that it's useful to retain compatibility with them.
I'm using that same moshisushi/hlsjs-ipfs-loader project but I forked it to fix some bugs, that's probably what you're running into.
Because you have two p2p networks one behind IPFS it is like work internally, and another in the client level using p2p software to share the content, the plugin i have uploaded could connect two worlds, you are watching a video over IPFS and sharing the content, the advantage of IPFS vs WT is that torrents must be closed and IPFS can be opened to share live content for example. Regards
And they have 95% same blocks. And will be 95% source for each other. Each file size is 1.3 GB.
That sounds very impressive! Have you calculated the overall space savings though? That's such a small block size that I'd expect the parent nodes that point to all the chunks to themselves require a huge amount of space. Is it enough to make up for avoiding duplicating the video content?
@scanlime
My mistake. Files sizes is 1.6 GB and 76% of same blocks. 1.8% is metadata.
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mp4
Result:
Blocks size: 1 655 040 126 bytes -
File size: 1 624 999 455 bytes =
Metadata size: 30 040 671 bytes (1.8%)
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mkv
Result:
Blocks size: 1 654 562 731 bytes -
File size: 1 624 528 837 bytes =
Metadata size: 30 033 894 bytes (1.8%)
I use this script (js-ipfs-same) to compare blocks tree.
Same data: 1 246 038 243 bytes (76%)
But for converting container from MP4 to HLS, the rabin will not work.
FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)
That's interesting, seems like a niche that many folks would appreciate being filled! One question though, you are striving for privacy, but IPFS is really antithetical to that goal by default as it's constantly broadcasting information about what content your node has locally. This seems fine for a public site like most PeerTube instances are, but it might be less appropriate for your use case here.
I'm not sure what you mean by "two p2p platforms" here? If you mean retaining compatibility with webtorrent or with the novage loader, that may be doable but I'm not sure the existing systems are working well enough that it's useful to retain compatibility with them.
I'm using that same moshisushi/hlsjs-ipfs-loader project but I forked it to fix some bugs, that's probably what you're running into.Because you have two p2p networks one behind IPFS it is like work internally, and another in the client level using p2p software to share the content, the plugin i have uploaded could connect two worlds, you are watching a video over IPFS and sharing the content, the advantage of IPFS vs WT is that torrents must be closed and IPFS can be opened to share live content for example. Regards
Ok. I think I get what you're saying. To me the interesting part of using IPFS for this is that you technically would not have two different networks, you would have one network (with different tuning and capabilities for sure) but with the same data model on each side.
I think there's a lot of architectural fuzziness to get lost in here but I find it helpful to think concretely about how the data is being identified and authenticated. With IPFS there's a consistent way to identify and check data across the network. With PeerTube as it stands now, we use bittorrent hashes in part of the system and no hashing at all elsewhere.
Also keep in mind that browsers have a very limited ability to help share video data to others. It might be useful for networks where many clients are sharing data with each other locally through a limited internet connection. But for most PeerTube-like setups I'd expect that the benefit of having better p2p file sharing is in making it easier for loose federations of server operators to mirror each other's data in a way that increases reliability rather than decreasing it :)
In the current system, we use either WebTorrent (hash everything once the content is fully uploaded) or the novage p2p loader, which is very YOLO and trusts everyone.
My mistake, I did not notice that we actually have another hashing system that's used by the HLS P2P loader. That one uses SHA256 hashes computed on entire segments. So, not really compatible with either other system directly, but it wouldn't be a big deal to retain compatibility with the system.
My main issue with the novage p2p loader is that I haven't come across very many situations where it's actually useful. It might help with individual extremely popular videos, but does anyone have those right now? Does it serve blocks fast enough for clients to see any benefit?
PeerTube's redundancy system is almost great... it only works with trusted peers right now though. A single slow or malicious redundancy peer will ruin the client's experience. That's the problem I'm interested in potentially solving with IPFS: having a good way to open the floodgates to crowdsourced data redundancy in a way that actually works. Browsers are really bad at this, they might be able to help a bit but the core of the network really needs to be servers that link together if we will have p2p that fits users expectations for loading video.
@scanlime
But for most PeerTube-like setups I'd expect that the benefit of having better p2p file sharing is in making it easier for loose federations of server operators to mirror each other's data in a way that increases reliability rather than decreasing it :)
This is exactly what I mean when opening the issue, thank you and all the others for the very interesting replies so far!
FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)
That's interesting, seems like a niche that many folks would appreciate being filled! One question though, you are striving for privacy, but IPFS is really antithetical to that goal by default as it's constantly broadcasting information about what content your node has locally. This seems fine for a public site like most PeerTube instances are, but it might be less appropriate for your use case here.
@scanlime What I mean by privacy is analytic and user tracking. IPFS is just a storage, so if you want to keep your storage private, you can create your own IPFS Swarm with a set of peers. From what I've understood, you don't even have to isolate the pinset behind a private network, you just have to configure a trusted peers list. You can then create a gateway to control access to the cluster content. Is that answering your question ? I'm new to IPFS by the way but from what I've read in the docs, IPFS works fine for both cases.
@scanlime What I mean by privacy is analytic and user tracking. IPFS is just a storage, so if you want to keep your storage private, you can create your own IPFS Swarm with a set of peers. From what I've understood, you don't even have to isolate the pinset behind a private network, you just have to configure a trusted peers list. You can then create a gateway to control access to the cluster content. Is that answering your question ? I'm new to IPFS by the way but from what I've read in the docs, IPFS works fine for both cases.
I think you're misunderstanding what ipfs-cluster is for.. it doesn't change who the individual IPFS nodes contact, it's an additional layer on top of ipfs so that groups of servers can agree to reliably serve a set of content together. That's all about reliability, not about privacy.
You have to understand that IPFS (and all p2p protocols I know about) are quite promiscuous, they will just ask anyone they're connected to for the data they need, or broadcast the data they have to random people. To retain privacy, you would really need to have a completely closed network, in which case there isn't much point to the protocol.
This seems fine for PeerTube and is roughly equivalent to our existng level of privacy here. But I wouldn't use IPFS as a basis for creating a project that has privacy as its focus.
@scanlime Alright thanks for taking the time to explain that to me. I'll look closer at the design of IPFS and its use-cases. On a side note, what I can do, is simply put IPFS as a storage option in my project. You could then choose between local storage, IPFS or whatever (S3, SFTP...).
I should stop discuss about that there since it's off-topic, but feel free to create an issue in dreamvo/gilfoyle to discuss further about it :)
I'm still experimenting with IPFS. There's a lot to like, but the RAM usage is a real drag, and might be the main reason not to use it in most PeerTube-style video sites. Each IPFS node needs enough memory to quickly invert hashes into disk blocks, and... on my ~9TB dataset the ipfs daemon wants to use around 11GB of RAM even with few peers. The actual hashes are large but i'm not sure how to explain the bulk of that usage. A 32 byte hash for each 256kB block is 128MB per terabyte. (Edited: math was off by 1024x, oops)
FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)
Will it somehow relate to peertube? Maybe federate with peertube?
FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)
Will it somehow relate to peertube? Maybe federate with peertube?
I don't think so as it's not designed to work with Peertube or other P2P platforms. But feel free to discuss about the design in the repository.
It's not a real project yet but I've still been experimenting with an IPFS-backed video server that would be compatible with PeerTube federation. There are a lot of performance hurdles to get over but the large amount of existing tooling for sharing hash-addressed blocks still feels compelling to me.
I think you're misunderstanding what ipfs-cluster is for.. it doesn't change who the individual IPFS nodes contact, it's an additional layer on top of ipfs so that groups of servers can agree to reliably serve a set of content together. That's all about reliability, not about privacy.
@scanlime I've seen this repository showing how to setup a private IPFS swarm to host private files. Apparently this is possible by removing the default bootstrap nodes and isolating your peers inside a private network. See also ipfs-inactive/faq#4 where the creator of IPFS himself says that this is part of their goals. So I guess choosing IPFS for privacy usage is possible and not an anti-feature.
See also ipfs-inactive/faq#4 where the creator of IPFS himself says that this is part of their goals. So I guess choosing IPFS for privacy usage is possible and not an anti-feature.
That is 5 years old, there is a much better way now, you can create private IPFS networks by restricting the p2p connection with a pre-shared key. This is only really practical for situations where you are using ipfs in a data center or in a corporate environment, I don't see how that would apply to something like PeerTube intended to be an open network.
If you are using ipfs and you care about privacy you have to understand it at least. Everyone your node connects with can easily find out what content you have and what content you're looking for. Like most P2P systems.
Everyone your node connects with can easily find out what content you have and what content you're looking for. Like most P2P systems.
Yes, that's the point of a swarm, this shouldn't be a problem if you control each nodes. But from what I see, using IPFS with a private network is an experimental feature that's not yet supported but which is not against the IPFS's design & goals. But you're right, this is not a feature that PeerTube would need.
This is a bit off topic but let me say that the Internet is not anonymous on a lower level of the stack and there is no way in general to share data publicly on a P2P network and don't make other nodes aware of what other data you are sharing. When we say PeerTube is privacy-friendly we mean privacy, not being anonymous. This means you are not tracked across the Web and nobody is using your browsing history on PeerTube to profile you. This is about the relation between the user and the tech giants providing Web services.
The problem of being anonymous is totally about how the Internet infrastructure is managed in the nation you live in. If you don't trust private ISPs you should politically pretend a public Internet service. If you don't trust your governament, you need to change it. No technology is going to fix that.
If you don't trust your governament, you need to change it.
This is true, but I'll also add that this is the whole point of technical privacy systems like Tor.
This whole thread seems to be missing a demo, so here is one, it is complete with P2P, loads extremely quickly, and automatically switches between 5 different bit rates:
https://bafybeiazt45dboknwnwwrekot7eenfr62sr6vmxhrwobr4p3cymfmorx5y.ipfs.dweb.link/
You can get the peers by running this in the console:
for await (const peer of await node.swarm.peers()) { console.log(peer.addr.toString()) }
Actually there's even another distributed solution which is SIA Skynet.
If you look as this example: https://siasky.net/EAC6AsZovYp4aIN-FLj1mFEi43WSrGtF7IBZU1T8BzCGfg it loads faster than 90% of Peertube instances....(the video in the link is the documentary "The Internet's Own Boy: The Story of Aaron Swartz" licensed under Creative Commons "Attribution-NonCommercial-ShareAlike 4.0 International" ( http://creativecommons.org/licenses/by-nc-sa/4.0/ ) )
There a lot of distributed solutions popping up right now, however only IPFS is not tied to a block chain. You could argue filecoin, but IPFS does not depend on filecoin in any way. This gives value to IPFS for all sorts of applications,
Sia requires the use a token to download or upload files. Sia's slash page hides this, and the link you sent is going though their public gateway that they are paying for. 100% of the traffic is going though their webservers. These web portals will need to make money some how eventually as bandwidth isn't free. Similar to Sia, the people who own the bittorrent trademark released btfs. However it is also just a mechanism to give value to some coin. It is impossible to use these networks directly without jumping though many hoops.
IPFS, like bittorrent (the protocol), is much more true P2P. The peertube instances could run IPFS nodes without having to figure out how to get coins in and out of it.
Tying PeerTube to a storage token, or any specific coin for that matter, would be a bad move IMHO.
It's not about tying PeerTube to anything, I think that having more options to choose between is always a good idea. You want to use IPFS? Amazon AWS? SIA? should be your choice cuz at the end someone has to pay for bandwith and/or storage so i don't see the point.
Perhaps i'm missing something obivous, but if the uploader goes offline for some reason wouldn't the video be inaccessible (sorry if this was already replied)?
should be your choice cuz at the end someone has to pay for bandwith and/or storage so i don't see the point.
@nukelr Yeah the instance operator should be able to configure storage backends however they want.
Perhaps i'm missing something obivous, but if the uploader goes offline for some reason wouldn't the video be inaccessible (sorry if this was already replied)?
@alex9099 If PeerTube blindly ripped out WebTorrents and replaced it with just the browser js IPFS distribution and then didn't host it locally, maybe. But that's a ridiculous idea.
Perhaps i'm missing something obivous, but if the uploader goes offline for some reason wouldn't the video be inaccessible (sorry if this was already replied)?
that is relative, anyway if you remove the webtorrent it will not be available only in the peers, if the ipfs content is pinned it will federate between other instances it will always be available somewhere on the p2p network
Would be great for a peertube server to be able to talk with IPFS and allow anyone with a peertube account on the clear web to watch and connect to a video from IPFS or an unstoppable domain site.
Being the peertube server, each instance would run a different IPFS repository where the pinned videos would be stored, but the storage is local so the videos would have to be read from the IPFS repository, once the hash of the video blocks is shared between the instances are shared and the content is replicated. I have seen how the current storage of peertube is programmed, it is not modular so it would have to be adapted in case a plugin was used. Also comment on the option of using IPNS as DNS resolution of the instances. You would have two P2P networks running on the clients (webtorrent) and another one between instances (IPFS). Yes it would be unstoppable.
Good to get an update on mediating the storage issue with p2p solution on peertube? Are the any paths to this, is it on the agender?
The fact is that we use WebTorrent (BitTorrent/WebRTC) on the client side to watch videos. It provides a handy pool of content seeders and direct browser connection. Watching a video via IPFS would mean to replace entirely that component with an IPFS client in the browser. So it's not just thinking of a different storage/upload mechanism.
I think they can both exist in parallel.
I got it IPNS in the player 🤔
https://manalejandro.com/~/Art%C3%ADculos/reproductor-p2p-con-enlace-ipns
We could choose IPFS or P2P player loader 😉 unstoppable
It's hardly "unstoppable" if it's strictly less reliable than a single server and a single DNS entry. If you really want to avoid "censorship" (whatever that means to you) you actually need multiple reliable paths to serve the content. My experience with ipfs is that it is strictly less reliable than the network you're overlaying it on top of.
For what it's worth I'm not too optimistic about IPFS even after all the time I spent experimenting with video distribution. Ultimately, IPFS doesn't have a great content routing mechanism and it's extremely memory intensive, because it's not magic and all content-addressed storage is extremely memory intensive. Folks get optimistic about IPFS when it works well in small-scale tests, but that's mostly because in small tests you'll be able to keep enough open connections to just gossip your entire working set between hosts that already know about the data you are requesting.
None of it is private or efficient or hard to stop. I'd love to find a more efficient and reliable way to cluster storage together from multiple non-trusted hosts, but I've been thinking there does need to be at least a minimal trust relationship built around quality of service. I like the idea of peertube servers getting a "buddy" or two, with each server in each group capable of providing failover for the others.
It's hardly "unstoppable" if it's strictly less reliable than a single server and a single DNS entry. If you really want to avoid "censorship" (whatever that means to you) you actually need multiple reliable paths to serve the content. My experience with ipfs is that it is strictly less reliable than the network you're overlaying it on top of.
For what it's worth I'm not too optimistic about IPFS even after all the time I spent experimenting with video distribution. Ultimately, IPFS doesn't have a great content routing mechanism and it's extremely memory intensive, because it's not magic and all content-addressed storage is extremely memory intensive. Folks get optimistic about IPFS when it works well in small-scale tests, but that's mostly because in small tests you'll be able to keep enough open connections to just gossip your entire working set between hosts that already know about the data you are requesting.
None of it is private or efficient or hard to stop. I'd love to find a more efficient and reliable way to cluster storage together from multiple non-trusted hosts, but I've been thinking there does need to be at least a minimal trust relationship built around quality of service. I like the idea of peertube servers getting a "buddy" or two, with each server in each group capable of providing failover for the others.
Only if you use DNS with IPNS it is not necessary 😃 i mean, we have the player that works at half power and can give a lot more to choose from ⏯️ IPFS has more advantages than disadvantages.
I think IPFS is really cool, but I am not so sure it is ready for this use case, at least with native js-ipfs. It is simply not fast or efficient enough.
Consider your demo (or my demo at https://bbb.fu.io), it take multiple seconds before the video even starts playing. My demo has the video seeded on dozen of ipfs nodes and the javascript is bootstrapped with an ipfs node that has the video on it; and it is still the case that it takes multiple seconds to load. Significantly longer on lower power devices, such as a mobile device.
My demo has multiple bit rates, and the javascript can not load fast enough to play 1080p without pausing to buffer every few seconds. The loader is smart enough to choose a lower bitrate so a user might not notice. But if the user chooses higher bitrate, they will not have a good time.
IPFS is getting there, but it is not ready for browser video playback just yet.
I think IPFS is really cool, but I am not so sure it is ready for this use case, at least with native js-ipfs. It is simply not fast or efficient enough.
Consider your demo (or my demo at https://bbb.fu.io), it take multiple seconds before the video even starts playing. My demo has the video seeded on dozen of ipfs nodes and the javascript is bootstrapped with an ipfs node that has the video on it; and it is still the case that it takes multiple seconds to load. Significantly longer on lower power devices, such as a mobile device.
My demo has multiple bit rates, and the javascript can not load fast enough to play 1080p without pausing to buffer every few seconds. The loader is smart enough to choose a lower bitrate so a user might not notice. But if the user chooses higher bitrate, they will not have a good time.
IPFS is getting there, but it is not ready for browser video playback just yet.
It takes a while to connect and download the data but it works fine on my device 👍
A use case I am thinking is for users to be able to imp import videos from the ipfs network. I don't think ipfs should be used for sending video to the client, but it might work well for server side storage. Expessially if the peertube server caches the video.
A use case I am thinking is for users to be able to imp import videos from the ipfs network. I don't think ipfs should be used for sending video to the client, but it might work well for server side storage. Expessially if the peertube server caches the video.
Another idea would be to use IPLD through IPNS, as this your IPFS settings will be from your DNS server so you won't have to code anything since PT won't see any URL changes. I talk about if you have your own IPFS cluster of course.
[I’ve made some corrections since first posting this]
I’ve read through the entire issue and I think I’ve come up with a solution for this IPFS usage debacle. Here it is:
- Each instance should (well, “could” is a better term) host its own local IPFS node and set up a public IPFS gateway to the node;
- Keep the video uploading to an instance as it is (no need for IPFS at this point);
- Once a video is uploaded and converted into different files of different qualities, import each of these files (but don’t duplicate them) to a folder on the local IPFS node, and, by doing so, get a public IPFS gateway URL to each file;
- Make the IPNS name(s) of the folder(s) known to everyone (these are the “big folders”, not just a folder for different files for one video), much like is already being done for Library Genesis and Z-Library (see more links below), and tell people: “Go and host these videos on your own IPFS node”. Doing so will create a new kind of redundancy for instances which doesn’t rely on maintaining a PeerTube instance or even knowing anything about servers and command line;
- Create a torrent for each video file and then use the public IPFS gateway URL to each file as a webseed;
- By properly handling the URLs used as webseeds, a user who has the IPFS desktop software installed will also be able to get the video from the IPFS network and not just through WebTorrent (the companion extension isn’t needed for this). If they don’t have it installed, no problem, everything will continue as usual;
- If P2P Media Loader is used instead of WebTorrent, it should first try to get the file from the IPFS node. If that doesn’t workout, then go get the file from the “normal” storage;
- The IPFS URL to the file could also be injected in the src attribute of the video HTML element. For this step to really work users would need the IPFS companion extension, though, otherwise the instance would be relying solely on the public gateway.
In short: each instance should host a local IPFS node, use IPFS as webseeds and/or try to load videos first from the IPFS and make it really easy for people to import the instance’s videos into their own IPFS nodes.
Plus: Look at IPFS Free Library and Pirate Library Mirror’s efforts and at all the work needed to get files from a BitTorrent client to a local IPFS node (without duplicating files or the need to mess with command line) and help me get BitTorrent clients to make this process really easy. Links below.
Putting 5,998,794 books on IPFS
Make WebTorrent more IPFS-friendly by better handling IPFS URLs used as webseeds
Make Transmission more IPFS-friendly by better handling IPFS URLs used as webseeds
Make qBittorrent more IPFS-friendly by better handling IPFS URLs used as webseeds
Make Frostwire more IPFS-friendly by better handling IPFS URLs used as webseeds
Make BiglyBT more IPFS-friendly by better handling IPFS URLs used as webseeds
@Chocobozzz Do you think it could work as I propose?
I thougt also about IPFS and fediverse.
But more for images.
Now every image must be spread to each instance, where users should get the image.
A big amount if redundant space all over the fediverse instances...
If image-uploads go to an ipfs-storage, clients can retrieve them from ipfs, not from the own instance only...
If you find a way, to implemrnt it to this service, would you spend some thoughts on other fedi-services too?
maybe add option section for ipfs starting with the ipfs gateway selection. then I think this is more a server owner to manage than PeerTube project.
It would be great if some of the IPFS fans here could take some more time to understand the overheads and the limitations in the protocol.
There's nothing magical or even good about IPFS. It's an overlay network that makes your average content take longer than https would in the worst case. It doesn't store anything for free, and it doesn't store anything automatically. The caching that does happen is fairly ad-hoc and doesn't relate to the actual usage patterns of the content.
If you took the PeerTube as it currently exists and simply bolted IPFS onto it, memory and network usage would increase dramatically, and there would be no immediate benefit to users. It would be theoretically possible for other users to pin and re-host content that's addressable over IPFS, but in practice that content would always travel through an existing peertube server as a gateway. Each server would need to keep a local copy of content that must be delivered reliably/quickly, like video segments or local video thumbnails.
I can see maybe replacing the existing mechanisms for caching remote server thumbnails, but I really doubt it would be more efficient. There's a really astounding amount of memory overhead for indexing a fully content-addressable storage of any type, and the network protocol relies on a huge amount of spurious communication between nodes. It's not going to realistically allow third parties to contribute hosting resources.
Here's a useful subproblem to think about if someone wants it:
To make any realistic use of P2P for primary data storage (i.e. having an IPFS gateway which doesn't have local content pinned locally) the other layers of the stack will need to account for content taking a highly variable amount of time to retrieve.
This piece would be useful for a lot of different kinds of storage! Even in my current PeerTube setup, without object storage or anything, some content will take a different amount of time to retrieve because of block-level caching. It would help every PeerTube server run smoother, but it would open the door to truly latency-variable storages, where some items would be local and some would take an unknown amount of time to retrieve. This would require changes to the backend and UI software as well as probably the UI interaction itself.
@scanline
there is one point about IPFS logic today I'm not sure is working like you say. what I know about IPFS is as long as the local chunk exist (the original chunk from the original server) the replication will exist. if it does not, after a while it disappear from the network unless some servers configured IPFS as persistent, creating as result huge latency or timeout.
unless some servers configured IPFS as persistent
Not sure i understand your point. I'll try to respond anyway?
You can host data permanently if you want, in IPFS terminology this is "pinning". All this does is, as you suggest, gets the data and prevents the data expiring from a cache that it would otherwise expire from.
IPFS relies on gossiping the state of this local cache to nodes that you have a direct (tcp/udp) connection with. If one server happens to still hold onto a chunk because it was fetched recently, its neighbors will find out about this chunk. If it stays cached for long enough, unconnected servers could learn about it through the DHT.
This pinned-ness is a flag that has to be set per chunk. (Chunks are MAX about a megabyte, usually half that.) It can be applied recursively to all the chunks necessary for a file or directory, via IPLD links, but the server still fundamentally is just gossiping about which specific <1MB pieces of data are in the cache.
This is the only replication you get: either someone explicitly chooses to host the file by pinning it, or you get lucky and someone still has a piece cached. That latter strategy can make IPFS appear pretty fast when you're dealing with data that isn't too large or too uncommon or too deeply nested. Every time you have to discover a new layer of blocks that you need but don't have, it takes considerable time to broadcast that request to neighbors, search for it in the DHT, make new connections, gossip block lists, and eventually find someone who has a copy of that data.
Hello, to clarify the matter a bit, each instance of PeerTube would have its own IPFS repository, the repositories share the information with each other, but i can have a repository that does not accept sharing only the content that i have locally, in that case it could be decide which instances to share my pinned blocks with the others. I was looking at the current implementation and it is not modular, so the PeerTube storage layer would have to be modularized first to integrate this P2P storage.
@manalejandro
your suggestion seems to be interesting indeed. why not build a plugin and when it's reaching maturity why not to integrate it in peertube core.... It would be interesting to study how archive.org uses IPFS......
Peertube doesn't even need to host its own IPFS store, it can just start streaming directly from ipfs.io or dweb.link (both ran by protocol labs) to the browser.
For extra authenticity / trustlessness, peertube can download the video in car format (like .tar but for IPFS content, and self-verifying), and reconstruct it locally to verify its authenticity, or do that to speed up downloads / archive the video via traditional storage means.
In the end, running an active IPFS node, or even caching it locally via IPFS, is only needed if the peertube instance is going to fetch remote content over the network, and not via gateways.
The only problem is that this would make the job of pinning the data nebulous, as the video is just a reference to a CID (IPFS file hash/reference, "Content IDentifier"), and the backing data to that CID can be either alive or not.
For this, pinning outsourcing can be done via the Pinning Services API a REST Spec that some commercial pinning services follow, but more interestingly, ipfs-cluster - a self-hostable pinning 'orchestrator' - also uses. A peertube instance can be paired up with a ipfs-cluster (running an IPFS node), and pin locally-saved videos to this cluster (or those other pinning services) via this API.
All in all, there are plenty of options to make this more speedy.
What I will not recommend is defaulting to adding an IPFS node to every peertube instance, this might work if it is put into dhtclient mode (or in other words: it does not participate in the DHT exchange comprehensively, and doesn't announce it's presence to the network, as exchanging and refreshing DHT records is a major idle load), so that the peertube node can at least attempt to find a video on the network without relying on public gateways, but this requires investigation.
What I will also warn against is putting the ipfs client directly in the browser. IPFS runs best when it's been warming up for a while, as then it has a healthy cache of known closest-distance peers on the network, and it can resolve instances relatively quickly. A cold-started IPFS client on a video page will not be able to serve content (quickly) for at least half a minute or so, even more if the content is behind NAT, and finding addresses for that peer is tricky.
A service worker could work, maybe. If one is connected to the Peertube instance's "home ipfs node" by default, and/or other "known peers that host the content", that federated peertubes could hint at, retrieval could start immediately. Though this would simply turn the IPFS node into a 'workhorse' as a fancy CDN, giving it non-distributed load scaling.
Finally, Kubo (aka go-ipfs) isn't the only implementation out there, recently i found out about iroh, a WIP rust-powered IPFS client, which could maybe illustrate that not everything is about the golang impl.
All-in-all, here's the summary of IPFS-in-peertube as I see it;
Pros:
- Disconnect of data and addressing with CIDs
- Videos can be simple links to CIDs, or can be the CIDs
- These CIDs and their content will be immutable, and self-verifying
- Hosting can be done else-where, or can even be done by goodwill of a community.
- (as long as one node has the content and is reproviding it to the network, it has a high chance of being found and retrieved)
- Less load on the peertube instance, especially in a distributed setting
- Assuming that other nodes grab the data from public gateways or with their own IPFS nodes, they can also re-provide (as long as it is cached and reproviding is turned on for cached content)
- Public gateways at least have a level of caching, lessening load and need to re-fetch content constantly
- Users who pinned the content may also be able to re-provide
- Load may be avoided from the peertube instance entirely if the client loads directly from a public gateway
Cons:
- Unpredictable loading times
- DHT search can be slow
- Content can be missing, never completing a search
- Providers may be offline, or unaccessible
- Public gateways time out, as the content does not meet their need of relative spread and knowledge of that content (i.e. content has been reprovided a bunch of times, and is well-known in the DHT, this isn't the case for hours after the video is first "seen" by the network, especially if it's a big file.)
carformat cannot be downloaded in chunks- I found nowhere that it is possible to do this, and i read that
cardata is given opportunistically in a "first seen in strict dag order". The CARv1 specification also notes that there needs to be an additional spec for determinism, basically waiving this problem. - This means that a
carfile needs to be downloaded whole, in one HTTP session, or otherwise restarted- This makes it relatively unsuitable for client-side reconstruction of the IPFS file DAG
- (directed a-cyclical graph, basically a self-verifying set of blocks in a strict order and relationship, chunks of data that're all verifiable by one hash, which is inside the CID)
- This means that the client needs to download a direct video file from somewhere, and trust that they're sending the right data according to the CID
- This makes it relatively unsuitable for client-side reconstruction of the IPFS file DAG
- I found nowhere that it is possible to do this, and i read that
- Some gateways explicitly forbid video streaming and downloading
- Cloudflare's IPFS gateway is one of these, I don't know if they've reversed their policy or if there are any others, but this is worth noting
- IPFS (at least in a server profile) has high overhead and high idle load
- This has already been covered above, but yes, Kubo constantly connects and gets connected by other peers exchanging DHT records, constantly reproviding them. Needing to do a full cycle every 12h.
- It is worth noting that with Kubo v0.18.0 this period is increases to 22h, applying less pressure on the network
- This has already been covered above, but yes, Kubo constantly connects and gets connected by other peers exchanging DHT records, constantly reproviding them. Needing to do a full cycle every 12h.
- Managing pins can be a hassle.
- Running and maintaining an ipfs-cluster service is extra system administration.
- Buying a subscription at IPFS pinning services, to essentially upload data and keep it uploaded. This might not be very attractive as many of these services are branding themselves around cryptocurrencies and related.
- Data can simply be gone one day.
- This can be circumvented by having a
.carbackup of the data locally, and reviving the data somewhere on-demand (as some/most pinning services, and kubo itself, allows uploading.cardirectly), though this would be a last-resort case.- Additionally this still incurs the cost of having the data somewhere in the first place, which adds cost.
- In lieu of having it archived, this means the video will essentially rot after a long while if nobody is interested in pinning it and keeping the records reprovided to the network, which can be the case, as running an IPFS node isn't cheap, and it doesn't get any cheaper the more data you want to store on it, the longer.
- This can be circumvented by having a
Essentially, IPFS is a loud, hyperactive, social butterfly cousin of torrent, where everyone can download anything they want with a magical key of that data, but that that data difficult to retrieve, 'inconvenient' to cache, or simply nonexistent, outside of people-with-that-key's control.
The biggest argument CIDs have (imo) is Portability and Self-verification, that anyone and everyone can take the hash and 'just download' the content, wherever it exists. Else it doesn't have much going for it.
No, I am not going to talk about Filecoin.
Peertube doesn't even need to host its own IPFS store, it can just start streaming directly from
ipfs.ioordweb.link(both ran by protocol labs) to the browser.
This is a terrible idea, we have a decentralized network now even if it's mostly just https. And you'd suggest that folks move to funneling all traffic through Protocol Labs in order to switch to a more decentralization-flavored protocol?
For IPFS to be an improvement at all, it would need to retain our ability to self-host video. Relying on an external HTTP gateway is really a no-go, and the gateways are a major limitation in deploying IPFS for general browser use. As others have noted, there's more opportunity for practical use as a server-to-server synchronization mechanism than for delivering client video. But as you and I have both noted, IPFS is quite resource intensive on the server.
The IPFS + PeerTube experiments I did some time ago were focusing on the case where you'd bundle an IPFS server (WebRTC gateway) on the same machine as PeerTube, so browsers could get a quick and reliable connection to the content while still allowing IPFS to serve files from elsewhere once we can warm up the connection. I put that project on hold though because of the high cost in latency and RAM.
And you'd suggest that folks move to funneling all traffic through Protocol Labs in order to switch to a more decentralization-flavored protocol?
Please read the rest of my comment, that's just one of the options I suggested.
I forgot to enumerate it, but here's roughly the options as i see it:
- Download directly from gateways, like i said above
- Puts a lot of trust in the gateway to get the correct file.
- Download from gateways, but via
.carformats- This'd allow the gateways to not need to be trusted, buuuut...
- ...the content can be slow to load, or time out in loading, in which case the HTTP request needs to be started over from the beginning.
- And the client needs to reconstruct the proper file (stream) from this format, needing buffering in the browser itself.
- Download from a sysadmin-ran gateway
- This would add implicit trust in that gateway to fetch it, but;
- Its extra sysadmin overhead
- Running a gateway like that isn't cheap, as it's essentially running an IPFS node
- Download through a browser-based IPFS client
- This needs to have the IPFS client ran in service workers, else it doesn't have time to warm up.
- It still needs to warm up, so content isn't availible instantly
- It might consume a bunch of resources on the client, just to find and fetch content
- Download through a browser-based IPFS client, peering instantly with an admin-ran IPFS node storing the content
- Same problem as above; Sysadmin overhead and resource usage
All but the last option still needs the data pinned somewhere, either on user's computers, commercial pinning services, or a sysadmin-ran pinning service, so that comes on top of those.
One trick that peertube could probably deploy is the following;
IPFS has a high overhead and idle resource mostly of two things: Acting as a DHT server, and reproviding constantly.
The first one adds a lot of inbound connections and related handling, the second one adds a lot of outbound connections and related handling.
Both can be turned off, leaving the client in a relatively-low idle state, but turning of reproviding makes it not possible to discover content properly on the network.
However, Kubo has an option to only reprovide the "roots" of pins, i.e. the head block, the first block, the one that points to all the others.
It's generally not recommended this way, as something like the following could happen;
- Alice wants to find X, Bob has X
- X points to Y and Z, these aren't announced by bob, but X is.
- Alice finds out Bob has X, and connencts, and asks for X, bob provides it.
- Alice now sees that it needs to find Y and Z, it starts "looking" for this, possibly taking the time to ask Bob, but;
- Alice can possibly have disconnected from Bob before she can ask him if he has Y and Z, this is a problem, because since Alice has X cached, she wont need to look for it anymore, and instead keeps trying to find Y and Z, which nobody tells Alice where it is, because Bob hasn't told anyone he has Y and Z.
This makes this option less favourable, even if it has extremely low overhead (it's easier to announce 100 records than it is to announce 100_000), because of its fallibility. Normally IPFS recovers from this by finding the same node over and over again when DHT-querying the remaining content, and more-or-less stays connected to it while fetching new blocks.
What Peertube could do in this instance is something like this;
- It sees that a client wants to play X
- It queries through its local IPFS node which nodes have X. (
ipfs dht findprovs) - It says Bob has it, Peertube then instructs the local node to add a peering to Bob, i.e. "Stay connected to Bob, and reconnect if things go wrong"
- (We could take the fact that Bob has provided this CID as a signal that it has the entire file)
- Then, Peertube instructs the local node to download X, and it starts fetching the blocks from Bob, and caches them locally.
However, this still has a few fallacies;
Kubo only lists up to 20 peers, any of these peers might not be a Peertube-ran peer, and could only have cached the root node, not any other.
(Other implementations, such as js-ipfs, give an iterable with all found-peers that have this content, possibly allowing someone to mostly-exhaustively map every node currently providing it.)
There is no real way of knowing if a peer is a peertube-ran ipfs node or not, on the surface.
However, this could be solved by providing a custom protocol through the IPFS node. kubo gives an experimental API for this with ipfs p2p, essentially injecting a new protocol in an existing node.
This could be used to simply say "hi" to another node, verifying if it is a peertube instance, and/or asking "hey, i saw you had X, do you have it all pinned as well?" for reliability, to know that "this is the peer to connect to". Through this protocol it could even be signalled that the peer would like to not be downloaded from, if possible.
Through this it might be possible to have an extremely-low-overhead IPFS peer running inside peertube, one that is mostly dedicated to storing and exchanging IPFS content for peertube primarily.
@manalejandro your suggestion seems to be interesting indeed. why not build a plugin and when it's reaching maturity why not to integrate it in peertube core.... It would be interesting to study how archive.org uses IPFS......
Hi, i'm trying to implement the plugin but i don't think it's the best idea to start a js-ipfs instance in the client side or as a server addon, i would like to do an implementation like uses OBJECT_STORAGE with S3 storage, I think it is the most economical solution in this case. I will use the latest release and we will discuss when i have something, regards.
I don't have much time to dedicate, but i think this is a good start
let ipfs: IPFS.IPFS
async function getClient () {
if (ipfs) return ipfs
const IPFS_STORAGE = CONFIG.IPFS_STORAGE
ipfs = await IPFS.create({
repo: IPFS_STORAGE.REPO_PATH || undefined,
repoAutoMigrate: IPFS_STORAGE.REPO_AUTO_MIGRATE || false,
peerStoreCacheSize: IPFS_STORAGE.PEER_STORE_CACHE_SIZE || 1024,
config: {
Profiles: IPFS_STORAGE.PROFILE || 'server'
}
})
logger.info('Initialized IPFS repo path: %s.', IPFS_STORAGE.REPO_PATH, lTags())
return ipfs
}
Hi, i'm trying to implement the plugin but i don't think it's the best idea to start a js-ipfs instance in the client side or as a server addon, i would like to do an implementation like uses OBJECT_STORAGE with S3 storage, I think it is the most economical solution in this case.
@manalejandro Yes, if I got your idea right, that's also how I see it: as a middle ground between totally local storage and totally remote storage (S3). Each instance should have its own local IPFS node (and a public gateway for it) and the files locally stored would then be imported (but not duplicated) to its local IPFS node.
As others have noted, there's more opportunity for practical use as a server-to-server synchronization mechanism than for delivering client video.
@scanlime Yes, that's why I talked about sharing IPNS names of the folders stored by an instance: right now, there is no easy to contribute to an instance by re-hosting its videos. If PeerTube wasn't about to drop WebTorrent, this could be easily done by creating dumps of torrent files and making them available to the users, but WebTorrent support is going to be dropped, so something else must be done so that real decentralization can be achieved.
right now, there is no easy to contribute to an instance by re-hosting its videos. If PeerTube wasn't about to drop WebTorrent, this could be easily done by creating dumps of torrent files and making them available to the users, but WebTorrent support is going to be dropped, so something else must be done so that real decentralization can be achieved.
Of the systems we have implemented or discussed here, few of them actually let a random contributor provide bandwidth assistance to an instance or video they want to support.
We have the redundancy system, which comes the closest. Another peertube server can follow, download content, and provide regular HTTPS URLs that will actually help clients download the video faster. This works pretty well, but it does require that the server allows redundancy, and the features for robustness and abuse prevention are currently quite limited.
WebTorrent was never especially useful for providing bandwidth assistance like this, because so few torrent clients were actually compatible with WebTorrent. The bandwidth helper would have to use one of the few torrent clients that supports WebRTC transport, and each client would still take some time to discover the available helpers rather than getting a list right away like the redundancy system provides.
The IPFS scheme you're presenting, as i understand it, would let anyone provide long-term storage with no reliability claims attached. So, not a replacement for primary storage but an optional source of additional bandwidth. Problem is, that bandwidth is only directly useful to IPFS peers, not to regular web clients. The helpfulness of this scheme depends pretty much entirely on gateway bandwidth. You'd either need more widespread adoption of the WebRTC bridge so js-ipfs can connect to helpers directly, or you'd need people to also (separately?) volunteer to provide IPFS gateway services. This might work but it seems like a lot of extra RAM and bandwidth to spend for roughly the same benefit we had in the existing redundancy setup.
WebTorrent was never especially useful for providing bandwidth assistance like this, because so few torrent clients were actually compatible with WebTorrent. The bandwidth helper would have to use one of the few torrent clients that supports WebRTC transport
Yes, it's a shame that so few clients offer support for WebTorrent, but there is a WebTorrent client and it can be run on a server, so technically that could still be achieved...
The IPFS scheme you're presenting, as i understand it, would let anyone provide long-term storage with no reliability claims attached. So, not a replacement for primary storage but an optional source of additional bandwidth.
That's right.
Problem is, that bandwidth is only directly useful to IPFS peers, not to regular web clients. The helpfulness of this scheme depends pretty much entirely on gateway bandwidth. You'd either need more widespread adoption of the WebRTC bridge so js-ipfs can connect to helpers directly, or you'd need people to also (separately?) volunteer to provide IPFS gateway services. This might work but it seems like a lot of extra RAM and bandwidth to spend for roughly the same benefit we had in the existing redundancy setup.
I do understand the problems around this scenario, and so I pose this question: how could someone that is familiar with using seedboxes help a PeerTube instance? After all, someone who pays for a seedbox service isn't (necessarily) someone that know how to setup a sever, and as such the existing redundancy setup isn't of real use to such an user. How can this gap be bridged so that your P2P-heavy user can contribute in an easy way?
how could someone that is familiar with using seedboxes help a PeerTube instance?
Seems like we need social infrastructure for this not technical, no? Anyone can install peertube and offer redundancy, if they have a computer that's reachable over https. If their computer is not reachable over https they aren't going to be able to provide data quickly to clients, so it's of limited help.
Seems like we need social infrastructure for this not technical, no?
That's like saying someone should go for trade school to learn how to install solar panels in order to contribute to the grid.
Anyone can install peertube and offer redundancy, if they have a computer that's reachable over https.
I agree. Is there an .exe for that?
One line in the command line is one line too many.
I agree. Is there an .exe for that?
i get that a lot of folks would like there to be an approximately zero-step way to contribute bandwidth, but the reality is just more complicated than that and we don't have magic solutions...
if you want to help today right now using technology that exists, you need to be reachable over https OR you need to be able to very reliably do NAT hole punching. neither of these are compatible with being a completely clueless user who wants to understand nothing about the network.
as i said, this isn't fundamentally technical it's social. people need to interact socially and trust each other to some extent. if we tried to do video streaming over a fully trustless network that's not going to be a performance improvement over the status quo.
i get that a lot of folks would like there to be an approximately zero-step way to contribute bandwidth, but the reality is just more complicated than that and we don't have magic solutions...
if you want to help today right now using technology that exists, you need to be reachable over https OR you need to be able to very reliably do NAT hole punching. neither of these are compatible with being a completely clueless user who wants to understand nothing about the network.
I ask for an .exe more to create some discussion than to actually get an .exe. I get it, things just are not that simple. But if today someone who know their way around command line and networks would be to start a seedbox service for PeerTube's instances, would they be able to do it or they would need to host an entire PeerTube instance? Could they run a bunch of "PeerTube's sharing-is-caring instances" much like a seedbox runs a bunch of BitTorrent clients and then let users select which instances they would like to help (much like a user do when they add a bunch of torrents to a seedbox)?
Seems like we need social infrastructure for this not technical, no?
That's like saying someone should go for trade school to learn how to install solar panels in order to contribute to the grid.
Anyone can install peertube and offer redundancy, if they have a computer that's reachable over https.
I agree. Is there an .exe for that?
One line in the command line is one line too many.
i get that a lot of folks would like there to be an approximately zero-step way to contribute bandwidth, but the reality is just more complicated than that and we don't have magic solutions...
if you want to help today right now using technology that exists, you need to be reachable over https OR you need to be able to very reliably do NAT hole punching. neither of these are compatible with being a completely clueless user who wants to understand nothing about the network.I ask for an .exe more to create some discussion than to actually get an .exe. I get it, things just are not that simple. But if today someone who know their way around command line and networks would be to start a seedbox service for PeerTube's instances, would they be able to do it or they would need to host an entire PeerTube instance? Could they run a bunch of "PeerTube's sharing-is-caring instances" much like a seedbox runs a bunch of BitTorrent clients and then let users select which instances they would like to help (much like a user do when they add a bunch of torrents to a seedbox)?
This is basically how the redundancy feature already operates. The server can automatically select popular videos to mirror or an admin can select them. It's done via the peertube UI, so you'd either leave that enabled or write something simpler that's special purpose. But the simplest thing is often whatever you've already got.
@scanlime I have the impression that you have forgotten what my original proposal was.
@scanlime I have the impression that you have forgotten what my original proposal was.
No, the problem here is that when we try to examine the details of who is providing what over what channels, the magic-ness evaporates and we are stuck talking about servers and clients and recurring administration duties.
As rigelk responded nearly a hundred comments ago, the data needs we have on the client side and the import side are different, so offering uploads over IPFS doesn't help in most use cases. So, that's basically a non starter unless we have the desire and infrastructure to repeat the transcodes in many places.
There are surely places we could use IPFS but I'm just trying to bring this conversation down to earth and work through the actual problems we are trying to solve and understand the limitations of the tech stacks available.
Hi, i'm trying to implement the plugin but i don't think it's the best idea to start a js-ipfs instance in the client side or as a server addon, i would like to do an implementation like uses OBJECT_STORAGE with S3 storage, I think it is the most economical solution in this case.
@manalejandro Yes, if I got your idea right, that's also how I see it: as a middle ground between totally local storage and totally remote storage (S3). Each instance should have its own local IPFS node (and a public gateway for it) and the files locally stored would then be imported (but not duplicated) to its local IPFS node.
As others have noted, there's more opportunity for practical use as a server-to-server synchronization mechanism than for delivering client video.
@scanlime Yes, that's why I talked about sharing IPNS names of the folders stored by an instance: right now, there is no easy to contribute to an instance by re-hosting its videos. If PeerTube wasn't about to drop WebTorrent, this could be easily done by creating dumps of torrent files and making them available to the users, but WebTorrent support is going to be dropped, so something else must be done so that real decentralization can be achieved.
Hi, i'm trying to implement the complete backend with IPFS, otherwise it wouldn't make sense to duplicate the data, i think an API like the one used by S3 is the cheapest way to start it, i'm not saying it's the only one or the most efficient one, besides IPFS information can not be duplicated because all have a unique hash, regards.
https://github.com/Chocobozzz/PeerTube/blob/develop/shared/models/videos/video-storage.enum.ts
export const enum VideoStorage {
FILE_SYSTEM,
OBJECT_STORAGE,
IPFS_STORAGE
}
the best would be that every server starting a peertube instance become a node from a peertube gateway or whatever cluster gateway specified in the config....
Storing videos in IPFS prevents administrators from deleting my videos,
PeerTube instance administrators from maliciously deleting videos or closing registration/shutting down the server.