===========================
dedup: a deduplication tool
===========================
Find and delete duplicated files.
Use cases
---------
Find duplicates in a specific folder and create a script to delete them:
dedup find -o byedups.sh /your/folder
Find duplicated mp3 in a folder (and subfolders) and create a script to delete them:
dedup find --include *.mp3 -r -o byedups.sh /your/folder
Find duplicates except jpg in two folders and create a script to delete them:
dedup find --exclude *.jpg -o byedups.sh /your/folder /your/other/folder
How does it work?
-----------------
Finding duplicates is a two steps process:
1. Indexing: The full path of each file is stored in a database together
with a hash of the file. Currently the following hashes are implemented:
a. size: size of the file in bytes.
b. md5partial: md5 hex digest of the first 8Kb of the file.
c. md5: md5 hex diggest of the file.
Hashing allows narrowing down the list of potetial duplicates.
While is very likely that two different files have the same size, it is
unlikely that they have the same hash (`birthday problem`_).
.. birthday problem : http://en.wikipedia.org/wiki/Birthday_problem
2. Comparing: the potential duplicates (i.e. files with the same hash) are
compared to detect true duplicates.
Which is the right hash method depends on the number of files to check, their
size and the expected number of duplicates. Remember that is necessary to read
the complete file to generate a full hash and therefore it is time consuming to
do it for files in which the first 8k are already different.
The default choice is md5partial which provides a good tradeoff between speed
and reliability.
Why do you generate a script instead of directly deleting the files?
--------------------------------------------------------------------
Is good to have a way to check (and double-check) before deleting lots of files,
and being dedup a command line tool generating a script which you can open, edit
and check was the simplest and yet most powerful.
In addition this tools was concieved to delete duplicates across computers.
Generating scripts allows you to do this easily.
Deduplicating in different computers
------------------------------------
First create an index for each computer that you want to deduplicate::
# In computer 1
dedup index -o c1.sqlite -r /your/folder
# In computer 2
dedup index -o c2.sqlite -r /your/other/folder
Then transfer the two databases (c1.sqlite and c2.sqlite) to a single computer
and generate the script::
dedup script --pot c1.sqlite c2.sqlite
You will see two files (c1.sqlite.sh and c2.sqlite.sh, extensions are .bat if
you are in a Windows computer). ** WARNING **: As the files are in different
computer they cannot be compared. Therefore the script show the potential
duplicates acording to md5partial hash.
You can get less false positives by doing a full hashing but this might be
* VERY * time consuming if you have a lot of large files. An alternative is to
hash by md5partial, remove singles (those which we are sure are not duplicated),
and rehash the rest::
# In computer 1
dedup index -o c1.sqlite -r /your/folder
# In computer 2
dedup index -o c2.sqlite -r /your/other/folder
# Bring the two files to the same computer
dedup nosingle c1.sqlite c2.sqlite
# Copy back the files to the originating computer
# In computer 1
dedup index -o refined1.sqlite --by-md5 --db c1.sqlite
# In computer 2
dedup index -o refined2.sqlite --by-md5 --db c2.sqlite
# Bring the two refined files to the same computer
dedup script --pot refined1.sqlite refined2.sqlite
Hopefully there will be now only a few potential duplicates which you can
transfer from computer to the other and do a real comparison (or you can take
your chances and delete them!).
From a group of duplicates, which file is kept?
-----------------------------------------------
The first file appearing in the index is kept. You may force from which folder
are these files by specifiying it first in the list of folders::
dedup find -o byedups.sh /your/main/folder /your/other/folder
If you are deduplicating from multiple computers, put first in the list the
index generated in the computer where you want to keep the files.
Other commands
--------------
You can inspect a index database file with the following commands::
- list: Tab separated list the content of an index.
- info: Display indexing method and duplication statistics.
- doc: Print this help