Revival?
karbowiak opened this issue · comments
Any chance of a revival of this project @thejinx0r ? :)
TL;DR Maybe, but not soon. A C++ Version of AmazonFS with EncFS baked in might be coming. If that looks good, than a GDrive might come later.
Not anytime soon.
Out of curiosity, how does this project compare to others?
I've switched to Amazon for now for my data storage, but I haven't published the code for it yet, plus I'm also rewriting that, and moving the code away from node and straight in to c++ with encfs possibly baked in.
I had tried encfs over my amazon fs and it was slow, and i had baked in some form of it in to it, but i'm starting to get limited with the lack of threads in node.
Hey man!
After some fiddling about it's running stable and fast for me, managing to peg a 1/1Gbit to the limit in either direction (Had some issues with it crashing because of memory tho, but some --memory_reducer --optimize_for_size --zero_fill_buffers --max_old_space_size=4096 --expose-gc and global.gc(); added to file to force garbage collection, it's behaving quite nicely.
Sadly i tried the acdcli route as well for Amazon Cloud Drive, but the speed was terrible for me here in the EU, could do ~200Mbit/s up, but barely 20Mbit/s down :(
But it sounds interesting to write it in c++ with EncFS built in - altho i've personally stopped using EncFS and switched to CryFS ( https://www.cryfs.org/ ) works just the same tho, so no big difference there.
As for threads in Node, you could use something like RxJS ( https://github.com/Reactive-Extensions/RxJS ) to provide threading / promises / async / ?? - tho i it might not be fast enough i guess :P
Not gonna lie, your setup sounds pretty sweet.
For Acd speeds, it really depends on which datacenter your data is stored in. Generally, it's been in the US. I don't know if the European ACD accounts also use the US East datacenter or if they use one in Europe.
But apparently, if you have a European account, you can request to have your data moved to the European datacenter. I did get decent speeds from Europe, maybe 200mbit up, ~100mbit down to and from amazon US, but I can see why switching to Amazon would be a pain for you.
Javascript doesn't have the concept of threads unfortunately. There's no library that can add it in.
Just to be clear, what I mean is that a library implemented in C++ can use threads, but javascript code being executed cannot be run concurrently. For example, if I get 10 "getattr" requests, each one of them will have to be processed serially, because javascript can only process them serially. From the C++ side, fuse can be multithreaded, but what happens is that the "getattr" events are added to the javascript queue, and this queue can only be processed one event at a time. In Node, you can use multiple processes, like in python, but it's not good enough.
And is cryfs compatible with encfs?
Or did you need to reencrpyt everything?
Sadly my ACD account was an American account ( Some fiddling with Amazon back in the day to get Prime, but that didn't work out, ¯_(ツ)_/¯ ) so havn't been able to test the German/UK account types, with ACD in the EU, tho i bet they're quite a bit faster there. Google Drive however still blows it out of the water - shits crazy fast!
As for CryFS/EncFS, sadly no, they're not compatible - but in my case it wasn't so bad - since i didn't have much more than 1TB uploaded, re-uploading that didn't take too long. Plus blocks in CryFS are 8MB a size, meaning they can be downloaded pretty fast ( That's only really interesting if you have a load of small files tho )
Oh I see why the download slow.
That was one of the things that I didn't like about encfs, was that it couldn't download chunks bigger than the blocksize. That's why I had started to implement my own version in node.
And the other crappy part for Acd, is that you can't update a part of a file. So if you want to change just a small section, you have to download the whole file and then reupload the modified file.
There's a new api for it coming out soon, but it's only available by invitation only. So who knows, things might change.
@thejinx0r I would recommend for you to look going back to Google Drive. After many months of testing both ACD and Google Drive, the performance of Google Drive significant outperform ACD. ACD is one of the most unreliable Cloud storage out there. When you upload a large file, it can take 10-15min before the file shows up. You can't resume upload. And the files are stored in specific datacenter. Google Drive you can move anywhere around the world and still have decent speed. I've tried multiple servers in EU, East Coast USA, and West Coast USA. Most cities you get 10ms ping to Google Drive's server, probably because they have more of them. I also believe performance is better because of "egde caching". With ACD you either have to pick Ireland for EU accounts or Virginia for US/CA accounts. Plus Unlimited Google Drive is completely free with many work/school emails accounts.