cdn.media.ccc.de mirror
opened this issue · comments
cdn.media.ccc.de is the archive of all video and audio from Chaos Communication Congresses and other events by CCC. Including the 32c3 material, it's about 3 TB.
I made a little test download earlier today from a mirror which is geographically very close to biham (same city!) and got a steady 90 MB/s.
IPFS is in there too :):)
Started mirroring to biham a moment ago: screen bash -c "wget --mirror --no-parent -nv https://ftp.fau.de/cdn.media.ccc.de/ 2>&1 | tee ccc.log"
So fast <3 The mirror I mentioned above is https://ftp.fau.de -- it has plenty more interesting stuff @davidar @achin
while i am achin on IRC, i think maybe you meant @eminence ? 😄
Indeed -- totally forgot :)
@lgierth Don't tempt me, I will fill the disk on biham if you're not careful :p
Whoa who initiated that lead? :P
I've asked media.ccc.de for rsync access to get this going
I've asked media.ccc.de for rsync access to get this going
that was super fast -- I got rsync access and will start mirroring in the next couple of days
I finally did an rsync, took about a day at 50 MB/s and it's 3.68 TB in total. Started adding:
> screen sh -c 'go-ipfs/ipfs add -r cdn.media.ccc.de/ | tee ipfs.log'
It might be wise to add it file by file, keep track of the added hashes. From the past ipfs add is bound it fail.
It's at 53 % added now, has started really fast with >100 MB/s, and has slowed down over time to ~ 2 GB / min. CPU time, goroutines, and file descriptors are stable, memory has slightly increased over time to 2.3 GB, but that's the same ballpark as e.g. the gateways at the moment.
It's at 66 % now, so it took 4 hours for 13 %.
Looks pretty good so far, it's still getting gradually slower, but it looks like it'll complete some time tonight. It's at 82 % now and likely not run out of memory.
It's finished: https://ipfs.io/ipfs/QmXkqTFfTeZHTiMxPSBwcLWem5cQ8MsmbYfQRQd8Jydyg5
Symlinks don't work yet, we need a patch for ipfs add for that.
New hash, from a sync two days ago: QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4
https://ipfs.io/ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4
Symlinks are worked on by @kevina at the moment in ipfs/kubo#3508
So given that https://ipfs.io/ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4/congress/33c3 doesn't work due to not being able to read symlinks, where do I go to find the actual 33c3 videos not through a symlink?
▲ ipfs cat /ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4/congress/2016/dummy.txt
Obviously, there are no recordings yet.
This is a just dummy file to help the mirror admins verify that their
synchronization setup is working as intended. It will disappear before 33c3.@jedahan this mirror isn't contiuously updating itself yet, that's why there's no files for 33c3 yet.
Did an rsync run today, it's about 4.5 TiB now, and in the process of adding:
20:17:24 I: Started ./ipfs add -r cdn.media.ccc.de/
20:17:25 O: added QmSeCGfgvjdU28X4k4WJ484cNiBKeczk9uRNw5U2FVJfqF cdn.media.ccc.de/INDEX
New up-to-date mirror, now 5,4 TiB: ipfs://QmW84mqTYnCkRTy6VeRJebPWuuk8b27PJ4bWm2bL4nrEWb -- see #165
Should also grab https://cdn.media.freifunk.net
@lgierth I downloaded https://cdn.media.freifunk.net, actually I downloaded it from https://media-mirror.hamburg.freifunk.net/ as it was faster for me.
As I do not have a stable 24/7 IPFS node running at the moment the files is only on my laptop so if someone else can pin them that would be great.
Or even better might be if someone else also downloaded the files and verified my hash, total size is around 26 GB.
https://ipfs.io/ipfs/QmVbm2ZXeZYFVTojsjbsSyELQfpSiWscWaRsgs6s4kcyNj