ameenmaali / urldedupe

Pass in a list of URLs with query strings, get back a unique list of URLs and query string combinations

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Can't remove duplicate files

pwnedDesal opened this issue · comments

jsfiles.txt

https://www.test.com/js/0-0c6e5e47ca6a3f3f7243.js
https://www.test.com/js/0-249c4f63764b90e95f29.js
https://www.test.com/js/0-356c7b1d95f2143f6cd2.js
https://www.test.com/js/0-5adfe0ed1f01b27b5f5f.js
https://www.test.com/js/0-6553d716c12f03bb710d.js
cat jsfiles.txt | urldedupe -s

expected output should be: https://www.test.com/js/0-0c6e5e47ca6a3f3f7243.js