More info on active projects and modules at dat-ecosystem.org
Build a home server that adds high speed storage and upload capacity to a global volunteer science network
Work in progress!
If you want to build a DataSilo, you can use your own hardware or use our recommended hardware below!
If you are ready to run the software, hold on for a little while and keep an eye on svalbard, which is currently being worked on!
You can use any hardware but these are ones that we've tested and work great:
- Server: Intel Compute Stick V2 (~$125)
- Hard Drive: WD MyBook 4TB (~$110)
- Hard Drive: WD MyBook Duo 16TB (~$499)
- (Optional) Anker USB3/Gigabit Ethernet Hub ($25)
If you're in Canada here's the best link we've found for the compute stick: http://www.ncix.com/detail/intel-compute-stick-stk1aw32sc-atom-6e-134117.htm
Additionally to set it up you will need:
- USB Thumb Drive (Compute Stick can't boot from SD Card Slot so must be USB)
- Second computer to flash the USB thumb drive with Ubuntu
- USB or Bluetooth Keyboard + Mouse
- Display with HDMI input
The Compute Stick comes with Windows. You can choose to use Windows if you prefer, the Svalbard software should run on it (check the Svalbard readme first, work in progress). You can also dual-boot Ubuntu pretty easily.
- Sandisk branded SD cards don't work under Ubuntu
- Sometimes randonly when booting the ubuntu bootloader hangs on a purple screen and requires a power cycle
- Download Image: http://www.linuxium.com.au/how-tos/runningubuntuontheintelcomputestick
- Use Etcher to put it on a USB Flash Drive: http://etcher.io/
- Boot from USB Stick on Compute Stick using F10 at startup. (You may need to configure the compute stick to boot in Windows 64 mode, which is done by configuring the BIOS using F2 at startup)
- Choose 'Install Ubuntu' and follow instructions to dual boot Windows with a 10GB split partition.
Helpful Youtube: https://www.youtube.com/watch?v=KAMcr2vbOsA
These instructions are for MyBook or MyBook DUO
- install WD Drive Utilities on Mac OS
- convert to JBOD/EXFAT and turn off Drive Sleep
- plug into linux
The next phase uses parted to remove the existing JBOD/ExFAT partitions on the new drive and create an ext4 partition. First we'll find out what the block device name is (usually /dev/sda or /dev/sdb, /dev/sd1, /dev/sd2, or the like), then we'll remove the existing partitions.
lsblk # to view drives and get the /dev/sd<num> entries
sudo parted /dev/sd<num>
print
rm 1 (and others)
mklabel gpt
unit TB
mkpart primary 0.00TB 8.00TB
print
quit
Change 8.00TB above to match the drive size depending on which drive you got
Now we create a new ext4 filesystem: sudo mkfs.ext4 /dev/sd<num>.
Create a mount point for the drive, e.g.
mkdir -p /media/hd1
Find the UUID of the newly created filesystem with ls -l /dev/disk/by-uuid/ and use it to add the appropriate entry in /etc/fstab:
UUID=<UUIDHERE> /media/hd1 ext4 defaults 0 0
finally sudo mount -a to reload fstab and mount the new disk. df -H should now show you a new big drive.
On mac, we recommend using Homebrew: brew install node
On linux use the package manager you prefer
On Windows use the MSI installer
NOTE Svalbard is still under construction. If you are an early adopter and here already, try seeding this dat (9GB of White House Open Datasets):
# install node https://github.com/nodesource/distributions#deb
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs
# set npm prefix https://docs.npmjs.com/getting-started/fixing-npm-permissions
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'
# add to ~/.profile: export PATH=~/.npm-global/bin:$PATH
# install dat and download the data
npm install dat -g
dat clone 6fa91405f280c30cedd461dfcd3b4fffffb27759e26f8135b7cbdfe08870ccb2 eop-gov
cd eop-gov
dat sync # to seed
See https://github.com/datproject/svalbard for instructions when svalbard is ready
Here are devices we tested but for various reasons weren't as good of a fit as the Compute Stick, though YMMV and suggestions are always welcome as the DataSilo is a very hackable, DIY concept.
If you have a NAS you can use it! But buying a full NAS might be a little overkill if you are just using your computer solely as a DataSilo.
Requirements are:
- Has gigabit ethernet
- Runs Linux and Dat (
npm install dat -g), we use libsodium and leveldb native modules - Allows JBOD (RAID is not needed with Svalbard/DataSilo as Dat acts as an internet-level RAID)
These are a great way to get a 4 drive JBOD bay (non-RAID). I actually have 2 of them (8 drives total, with 4TB drives). But compared to the 16TB MyBook Duo the price for these wasn't worth it! But could be a great option if you have existing drives you want to use.
$40
MicroSD http://www.banana-pi.org/m1-download.html Ubuntu Mate 16.04 Limitations: Slow CPU encryption
$80
http://odroid.in/ubuntu_16.04lts/ubuntu-16.04-mate-odroid-x2-20160920.img.xz Ubuntu Mate 16.04
Limitations: Loud Fan, slow CPU encryption
$30
Limitations: 100Mb Ethernet, USB2, slow CPU encryption
~$200 and up
Limitations: Many kits barebones (requires more setup), expensive, but speed/performance was good. A great option for a more general purpose machine, but Compute Stick is same thing in smaller, more limited but cheaper package.