=========================== dedup: a deduplication tool =========================== Find and delete duplicated files. Use cases --------- Find duplicates in a specific folder and create a script to delete them: dedup find -o byedups.sh /your/folder Find duplicated mp3 in a folder (and subfolders) and create a script to delete them: dedup find --include *.mp3 -r -o byedups.sh /your/folder Find duplicates except jpg in two folders and create a script to delete them: dedup find --exclude *.jpg -o byedups.sh /your/folder /your/other/folder How does it work? ----------------- Finding duplicates is a two steps process: 1. Indexing: The full path of each file is stored in a database together with a hash of the file. Currently the following hashes are implemented: a. size: size of the file in bytes. b. md5partial: md5 hex digest of the first 8Kb of the file. c. md5: md5 hex diggest of the file. Hashing allows narrowing down the list of potetial duplicates. While is very likely that two different files have the same size, it is unlikely that they have the same hash (`birthday problem`_). .. birthday problem : http://en.wikipedia.org/wiki/Birthday_problem 2. Comparing: the potential duplicates (i.e. files with the same hash) are compared to detect true duplicates. Which is the right hash method depends on the number of files to check, their size and the expected number of duplicates. Remember that is necessary to read the complete file to generate a full hash and therefore it is time consuming to do it for files in which the first 8k are already different. The default choice is md5partial which provides a good tradeoff between speed and reliability. Why do you generate a script instead of directly deleting the files? -------------------------------------------------------------------- Is good to have a way to check (and double-check) before deleting lots of files, and being dedup a command line tool generating a script which you can open, edit and check was the simplest and yet most powerful. In addition this tools was concieved to delete duplicates across computers. Generating scripts allows you to do this easily. Deduplicating in different computers ------------------------------------ First create an index for each computer that you want to deduplicate:: # In computer 1 dedup index -o c1.sqlite -r /your/folder # In computer 2 dedup index -o c2.sqlite -r /your/other/folder Then transfer the two databases (c1.sqlite and c2.sqlite) to a single computer and generate the script:: dedup script --pot c1.sqlite c2.sqlite You will see two files (c1.sqlite.sh and c2.sqlite.sh, extensions are .bat if you are in a Windows computer). ** WARNING **: As the files are in different computer they cannot be compared. Therefore the script show the potential duplicates acording to md5partial hash. You can get less false positives by doing a full hashing but this might be * VERY * time consuming if you have a lot of large files. An alternative is to hash by md5partial, remove singles (those which we are sure are not duplicated), and rehash the rest:: # In computer 1 dedup index -o c1.sqlite -r /your/folder # In computer 2 dedup index -o c2.sqlite -r /your/other/folder # Bring the two files to the same computer dedup nosingle c1.sqlite c2.sqlite # Copy back the files to the originating computer # In computer 1 dedup index -o refined1.sqlite --by-md5 --db c1.sqlite # In computer 2 dedup index -o refined2.sqlite --by-md5 --db c2.sqlite # Bring the two refined files to the same computer dedup script --pot refined1.sqlite refined2.sqlite Hopefully there will be now only a few potential duplicates which you can transfer from computer to the other and do a real comparison (or you can take your chances and delete them!). From a group of duplicates, which file is kept? ----------------------------------------------- The first file appearing in the index is kept. You may force from which folder are these files by specifiying it first in the list of folders:: dedup find -o byedups.sh /your/main/folder /your/other/folder If you are deduplicating from multiple computers, put first in the list the index generated in the computer where you want to keep the files. Other commands -------------- You can inspect a index database file with the following commands:: - list: Tab separated list the content of an index. - info: Display indexing method and duplication statistics. - doc: Print this help