Ghost Tools
Transparently turns all of your existing servers, laptops, S3 buckets and dropboxes into one big de-duplicating content aware virtual filesystem that can archive, backup, replicate, synchronize and restore. GFS is zero-infrastucture, and respects all of your existing file locations and usage patterns. Inspired by the power of merkle trees and git.
Imagine all of your EXISING, as-is filesystems and organization schemes that you use on a daily basis were magically, natively content-aware and you had drop in replacement tools for things you do now that leveraged content awareness for backup, archive, snapshots, synchronization, etc. This is the premise of "ghost FS".
Why? Recently, there have been some awesome proliferation of de-duplicating, content-based storage tools like bup, attic, camilstore, git, git-annex, etc. But these tools are usually confined to backup or version control even though the same principals can easily work for archiving, file search, distributed storage, caching, and even just regular syncing files.
So why not build a general purpose tool that does all of these things? And lets make it work without any hard requirement for metadata servers, SQL databases, redis, etc. In other words, just a simple command line utility.
Along the way, we'll also get rid of the annoying assumptions and conventions that these other tools sometimes require (like .git directories or backup descriptors) so you're always free to break storage conventions, make your own backups, and abandon, ignore (or return to) GFS at any time with minimum breakage.
GFS's philosphy is "Go ahead without me. I'm smart -- I can catch up later."
gfs rsync workdir remote-dir
gfs rsync workdir jgilbert@server.com:remote-dir
<mess around with stuff on remote-dir using cp/mv>
# just works
gfs rsync workdir jgilbert@server.com:remote-dir
In a familiar, UNIX-like way, GFS can:
- Rsync files and directories without ever transmitting a single unnecessary byte, even when files and directories are renamed or re-ordered behind its back
- Create online and offline archives that can be partially or entirely locally cached
- Instantly tell you if a file is mirrored in other places and the last time its bytes were checksumed, even on offline storage
- Handle complex copies and file restores by sourcing files from the most efficient and promixal locations (think "mini CDN for my stuff")
- Backup files with super-efficient incremental, delta and baselines
- Snapshot versions of files and directories for recovery, reconstruction or even just bitrot detection
- Instantly search and extract files from zip files, tar files, backups
- Cache nearly any deterministic file operation
- Respect all of your existing file locations and usage patterns
GFS pretends your existing as-is filesystems, cloud storage buckets and servers are all one large content aware file system inside a universal (but very familiar) namespace. This remapping is done transparently without any effort or new filenames to remember. In fact, you won't even be aware of the remapping most of the time.
Inside this "ghost FS", all of the awesome properties of a content-aware file store are available.
- Instantly locate any file or directory by checksum
- Rapidly detect filesystem changes
- Easily remove (or restore) duplicate files
Inside GhostFS, we've implemented a set of general purpose tools to handle synchronization, backups, archival, snapshots, etc. Also inside this "pretend" name space are low level primatives that support deduplication, chunking, merges, splits, compression, encryption in a generalized and pluggable way.
But its just pretend. On the outside of gfs, your files are exactly what they've always been, and don't have to move to new locations or new names.
GFS's game of "pretend" is fast and quite resiliant. It transforms your filesystem into a content-aware semantics on a on-demand basis. Files are rarely checksumed more than once. And the metadata about GFS transforms is spread around in a highly recoverable (but always deletable) fashion. GFS is principaled that correctness of operations is invarient to metadata -- the correctness is verifiable with actual files on disk. Metadata is used to cache.
Best of all: GFS imposes no conventions - by default your backups and files, repositories and archives stay as 100% files and directories you can recognize, search and recover at any time. They don't even have to be renamed. That means you can walk away from GFS -- destory some or all of its metadata -- and your backups stay valid and recoverable on a per-file (or even per-directory) basis.
GFS gives you nearly instant, hyper-efficient incremental backups. It lets you spread your files over many drives with and without redudancy. GFS is indifferent if you use gfs(1) or other more familiar tools (Finder, cp(1), rsync(1)) to move files around. And it does this all without reliance on opaque databases, demons, or persistent processes. Nor does it require creation of "repositories" or "volumes" - if its a writeable directory, it can contain blobs.
Sound too good to be true? The downsides:
- GFS is not complete
- Its primary audience is individuals (maybe small workgroups) without complex permissions and <20M files
- To some, GFS's internals will seem overly complicated (most likely because it's design goals differ from similiar projects.)
- GFS's test suite is not complete (about 5% done)
- GFS is not built on git (even though we admire and love git, its not right for our use cases)
- GFS is fast because it is highly reliant on lazy caching, which doesn't suit some software developers who want formal deterministic behavior from their caching
- Remote stuff not implemented yet (but already designed and easy to implement)
- Its open source, so you'll get what you pay for (or are willing to fork)
- Its internal concepts could be hard to grok since they emphasize generality
- Some of the more esoteric functions like chunking, rdiffs, etc are not implemented yet.
- There is no FUSE / local mount support yet (but implementing this is trivial)
- GFS is written in a portable version of perl
- Because the metadata file formats are not yet stable, new versions of GFS may run with degraded performance for a while
- GFS can be RAM intensive for really massive file operations. (300-2000 bytes per file)
Swiss Army Knife Description
GFS is an overlay snapshoting and directory reconstruction tool. Its not really meant for version control, but managing someone's life which may be scattered across many different asset types and locations. GFS is entirely built around command line tools. It doesn't watch the filesystem for changes.
key features
-
highly accelerated content-aware rsync - a file is never retransmitted unless needed
-
worried you still wantt o eb sure you have a backup of a wedding photos? this has tools that let you flag and manage bit-rot and verify your stuff is where you think it is
-
work with large file trees both on a desktop and laptop? GFS can tell you instantly which chnages you've made on your laptop that are not yet on your desktop and push them to the desktop efficiently
-
in the field and want to be sure files you are working on are protected? GFS can make sure any file is either in your dropbox, on your machine at home
-
trying to free up space on your laptop? do you have a copy of that big 10 GB ISO file at home? find out quickly if you can ditch it.
Rationale
Introduction
Ghost Tools (gfs) is a maximally unobtrusive utility that manages universal state and locations of files you care about. This means that as files are scattered about in your life, across different drives or storage providers, GFS knows where things are gone and how to get them back. It has been especially designed to avoid any damons, persistent state, databases or cloud systems. Everything it does will continue to work no matter what kind of large axe you take to it’s “private” files.
The first rule of GFS is that every file or thing you want to protect continues to live exactly where you have it now.
Lets take a simple example. I store my main photos in the following directory:
apollo:~/PhotoFSX/LR-PhotoFS-LT $ ls -l
total 0
drwxr-xr-x+ 20 jgilbert staff 680 Jun 13 2015 2015-04-18
drwxr-xr-x+ 4 jgilbert staff 136 Oct 10 2015 2015-10-09
drwxr-xr-x+ 28 jgilbert staff 952 Feb 27 10:41 2016-02-27
drwxr-xr-x+ 31 jgilbert staff 1054 May 21 09:44 Isabella 2014 Part 05
drwxr-xr-x+ 16 jgilbert staff 544 May 21 09:49 Isabella 2015 Part 06
drwxr-xr-x+ 5 jgilbert staff 170 May 29 21:06 New Hampshire Drone May 2016
drwxr-xr-x+ 3 jgilbert staff 102 May 21 10:02 Rejects
drwxr-xr-x+ 8 jgilbert staff 272 Apr 20 2015 Rejects to move 2015April
drwxr-xr-x+ 556 jgilbert staff 18904 May 21 09:48 San Deigo Family 2015
drwxr-xr-x+ 92 jgilbert staff 3128 Nov 8 2015 Vermont 2015 Unsorted
drwxr-xr-x+ 13 jgilbert staff 442 Jun 15 2014 cambridge
drwxr-xr-x+ 10132 jgilbert staff 344488 Nov 11 2015 iPhone MASTER
drwxr-xr-x+ 2 jgilbert staff 68 Aug 17 2013 iphone
As you can see
gfs chvolumes
rule #2 - GFS still works even if you blow away its state and still loves you
gfs addvol sdfsdfsdf gfs checkpoint
gfs verify X, Y # tells you what has changed since last checkpoints
Motivation
If you're anything like me, you've got files scattered in an annoying number of different places. Much of this is compounded by the fact that i am both a tech geek (with lots of old programs, projects etc) and a photo-geek who inisists on saving raw photos and photoshop originals.
In short, I am a data hoarder. I don't keep everything, but I keep a lot.
In 2016, I have important data scattered over a variety of places because it is impractical to unify it into one place due to mobility:
- My main mac laptop (800GB)
- My main mac desktop (3TB internal) plus a SATA RAID-8TB and a DroboPro-16Tb
- An old Drobo that sometimes I connect to keep spare copies of things
- A few old hard drives on my desk with video render files I probably can delete
- An archive of my files before 2012 at my parent's house (safety net backup)
- An archive of some more recent things at my office across town
- A dropbox folder (using only 100 GB)
- A google drive folder
- An old laptop that I may or may not have fully moved out from 2013
- A Win7/10 gaming machine that may or may not have other files
How do you manage all of this stuff? How do you figure out what is a copy, what is a backup, and what is a primary storage for something? How do you make sure you have a DR plan? What do you do if you have tens of terrabytes of files? And what would happen if you main machines died -- how would you reconstruct your digital life scattered over so many places?
There are a few possible soluitons
- Buy an insanely large cloud storage or backup solution and a cable connection fast enough to upload it
- Be extremely organized and personally disciplined and never make a mistake
- Run your own server farm of owncloud or camilstore or infinite and become your own infrastructure provider
But there is another way. The power of technology!
Design Criteria
Ghost tool's design critera are unusual and probably won't work for everyone. The key principals that have worked well for my own digital packrat nature and govorn the use of this tool
- Be close to zero-install - everything is a perl script with base CPAN libraries
- Use checksums and careful design to NEVER send anything twice
- Familiar - syntax nearly identical to rsync
- Largely ignore filesystem metadata (tags and permissions) - most people don't care about these things, or use EXIF or XMP
- Require no databases or even meta-files (but if you can stand them, they'll make everything go much faster!)
- Always keep copies of things in organized directories similiar to the way they were found when backed up. That way, if things blow up you can manually recover everything and see your stuff in a familiar way - no proprietary tools needed!
- Everything is driven from command line like git - no servers!
- Paranoia - nothing is presumed to last forever, all storage has provisions for re-verification
- Archival occurs at the file and directory level because those are the units of things that matter to people - data is not divided and scattered to the wind
Pros and Cons
Advantages
- Feels a lot like rsync and copy tools but smarter
- Tiny footprint and low maintanance
Disadvantages
- Not going to easily give you any sort of GUI or web server
- Doesn't provide peer-to-peer trusted storage
History - Why this tool, why now?
Originally, I created a tool called powercut in 2005-6, back when I was a window's junkie. It did many of these same things but was a design nightmare. I was uninformed by much cleaner examples of content aware file storage like GIT. The design was based off of code I wrote at the beginning of my career to manage file collections for a knowledge management solution called KnowledgeRe@ch. (That old system had to be cool -- it had an "@" sign in the name!). Back before Carbonite and Mozy launched, I wrote a whole distributed backup tool called DSB in Java that I planned to launch as a commercial service. It had a nifty design that would have probably made me some money if I had continued the process of patenting it (I abandoned the pending patent in 2007.) DSB was great because it was self-organizing and fully distributed like Crashplan, and had a zero-trust model for internal replication. (Think a lightweight version Tahoe-FS.) Fast forward a few years while I got my MBA, and I gave up trying to maintain Java code and the servers needed to run it for myself and went back to simple command line tools.
Now its 2016 and time for a rewrite. I've learned what works and doesn't work in managing file replication and archival, and what the bare minimum of the principals needed to make such things work well. The design principals are clearer to me. And I've had the opportunity to learn what I like and don't like about newer technologies like git, Infinit, Dropbox, etc.
example
make a copy of some file you care about here is a simple example of how to back it up and manage it
also make a simple video / GIF tutorial