Fork to fix warnings and change in location of glib.
dtreetrawl
dtreetrawl trawls/traverses directory tree or file path to collect stats of every entry in the tree.
Usage:
dtreetrawl [OPTION...] "/trawl/me" [path2,...]
Help Options:
-h, --help Show help options
Application Options:
-t, --terse Produce a terse output; parsable.
-j, --json Output as JSON
-d, --delim=: Character or string delimiter/separator for terse output(default ':')
-l, --max-level=N Do not traverse tree beyond N level(s)
-f, --follow-symlink Follow symbolic links
-T, --no-tent Output only the summary(dstat), no other entries
--hash Enable hashing(default is MD5).
-c, --checksum=md5 Valid hashing algorithms: md5, sha1, sha256, sha512.
-R, --only-root-hash Output only the root hash. Blank line if --hash is not set
-N, --no-name-hash Exclude path name while calculating the root checksum
-F, --no-content-hash Do not hash the contents of the file
-s, --hash-symlink Include symbolic links' referent name while calculating the root checksum
-e, --hash-dirent Include hash of directory entries while calculating root checksum
Stats of all paths passed as arguments to dtreetrawl is printed to the standard output, errors to standard error.
After the stats of trawled entries, a summary of that run for the root path follows. When multiple paths are given as arguments, stats continue to print to the standard output.
tentry fields:
- entry/path name
- base name of the path
- level from root
- file type
- referent name if it's a symlink
- file size in bytes
- inode number
- number of directory entries
- permission in octal notation
- hard link count
- ownership: user and group id
- I/O block size
- allocated blocks
- last status change (ctime)
- last file access (atime)
- last file modification (mtime)
- hash
dstat fields:
- root path name of this run
- elapsed time in seconds
- start time
- follow symlink
- hash exclude name
- hash exclude content
- hash symlink ref name
- hash directory entry names
- root hash
- hash type
- depth
- total size
- total entries
- total directories
- total regular files
- total symlinks
- total block devices
- total char devices
- total sockets
- total FIFOs/pipes
Human readable output
dtreetrawl -f -F --hash .
...
... //clipped
...
/home/lab/dtreetrawl/README.md
Base name : README.md
Level : 1
Type : regular file
Referent name :
File size : 7483 bytes
I-node number : 305946
No. directory entries : 0
Permission (octal) : 0644
Link count : 1
Ownership : UID=0, GID=0
Preferred I/O block size : 4096 bytes
Blocks allocated : 16
Last status change : Sat, 20 Jan 18 14:33:11 +0530
Last file access : Sun, 21 Jan 18 21:42:24 +0530
Last file modification : Sat, 20 Jan 18 14:33:11 +0530
Hash :
Stats for /home/lab/dtreetrawl:
Elapsed time : 0.063725 s
Start time : Mon, 22 Jan 18 17:07:57 +0530
Follow Symlink : yes
Exclude name hash : no
Exclude content hash : yes
Hash symlink ref name : no
Hash directory entry names : no
Root hash : 6ddf338bc4fabe53a5eadf8bf8ff26d8
Hash type : md5
Depth : 6
Size : 851355 bytes
Entries : 241
Directories : 103
Regular files : 138
Symlinks : 0
Block devices : 0
Char devices : 0
Sockets : 0
FIFOs/pipes : 0
JSON output
dtreetrawl -f -F --hash -j .
...
... //clipped
...
},
{
"path": "/home/lab/dtreetrawl/README.md",
"basename": "README.md",
"level": "1",
"type": "regular file",
"refname": "",
"size": "6680 bytes",
"inode": "304693",
"ndirent": "0",
"permission": "0644",
"nlink": "1",
"ownership": "uid:0, gid:0",
"blksize": "4096 bytes",
"nblocks": "16",
"ctime": "Mon, 22 Jan 18 17:12:25 +0530",
"atime": "Mon, 22 Jan 18 17:12:25 +0530",
"mtime": "Mon, 22 Jan 18 17:12:25 +0530",
"hash": ""
},
{}
],
"dstat": [
{
"root_path": "/home/lab/dtreetrawl",
"elapsed": "0.090811",
"start_time": "Mon, 22 Jan 18 17:17:45 +0530",
"follow_symlink": "yes",
"hash_exclude_name": "no",
"hash_exclude_content": "yes",
"hash_symlink": "no",
"hash_dirent": "no",
"hash": "6ddf338bc4fabe53a5eadf8bf8ff26d8",
"hash_type": "md5",
"nlevel": "6",
"nsize": "850712 bytes",
"nentry": "241",
"ndir": "103",
"nreg": "138",
"nlnk": "0",
"nblk": "0",
"nchr": "0",
"nsock": "0",
"nfifo": "0"
},
...
... //clipped
...
Parsable terse output
In the terse output, each trawlentry/tentry/tent is surrounded by the set delimiter/separator twice. That is, say if the delimiter is :, then, at the begining and end of the tentry, :: will be placed; fields within the tentry will be separated by :. A successful run is followed by a summary(dstat), to differentiate it from the tentries, it will have only a single delimiter at the begining and ending. trawlentry starts with a double delimiter, dstat starts with a single delimiter.
Fields appear in the same order as listed. Fields that are not applicable are left blank, which leads to adjacent delimiters.
After each run, a new line is printed after dstat(summary entry) to help differentiate the next run for another root path.
Units are not attached to the field values in terse output.
Time values are in UTC as epoch for terse output(local string form for human readable output).
Sizes are all in bytes.
File type notation:
SOCK socket
LNK symbolic link
REG regular file
BLK block device
DIR directory
CHR character device
FIFO FIFO/pipes
dtreetrawl -f -F -t -d ":" --hash .
...
... //clipped
...
::/home/lab/dtreetrawl/Makefile:Makefile:1:REG::498:305832:0:0644:1:0,0:4096:8:1515853283:1516553956:1515853283:::
::/home/lab/dtreetrawl/dtreetrawl:dtreetrawl:1:REG::44424:304696:0:0755:1:0,0:4096:88:1516620913:1516620917:1516620913:::
::/home/lab/dtreetrawl/README.md:README.md:1:REG::7463:304693:0:0644:1:0,0:4096:16:1516621215:1516621215:1516621215:::
:/home/lab/dtreetrawl:0.013780:1516621221:yes:no:yes:no:no:6ddf338bc4fabe53a5eadf8bf8ff26d8:md5:6:851335:241:103:138:0:0:0:0:0:
Dependencies
Linux, GCC & GLib-2.0
To compile and install from source(preferred):
make clean
make
make install
# To uninstall
make uninstall
To install from RPM(centos):
# Pull the RPM
wget https://github.com/six-k/dtreetrawl/releases/download/dtreetrawl-1.0/dtreetrawl-1.0-1.el7.centos.x86_64.rpm
# Pull the public key
wget https://github.com/six-k/dtreetrawl/releases/download/dtreetrawl-1.0/RPM-GPG-KEY-dtreetrawl
# Verify signature
rpm --import ./RPM-GPG-KEY-dtreetrawl
rpm -K -v ./dtreetrawl-1.0-1.el7.centos.x86_64.rpm
# should get: dtreetrawl-1.0-1.el7.centos.x86_64.rpm: rsa sha1 (md5) pgp md5 OK
# Install
yum localinstall dtreetrawl-1.0-1.el7.centos.x86_64.rpm
To run the pre compiled binary, clone or fork this repo and cd to the directory:
./dtreetrawl -h
./dtreetrawl --hash -d ":" -c "sha1" -- /stairway/to/heaven /eastblue/southblue/allblue
To run this by name, symlink this to one of the shell's search PATH:
ln -s /path/to/dtreetrawl/dtreetrawl /usr/local/bin/dtreetrawl
Acknowledgements
Folks at Fractalio have been kind enough to test this tool at production sites and reporting bugs. Purushotham & Gugan have been a huge help.