Kadetron / Exticz

A scrapping tool which is used for crawling and extracting href links from a website

Exticz

Exticz is a scrapping tool which is used to extract href links from a website and is written in shell scripting, can be used for examining websites for redirection and finding sub domains

This is the initial build and can crawl up to a depth of 2.

Usage

$ exticz

$ exticz
Enter the URL: <[domain].[TLD]> | <[domain].[TLD]/directory> | <[domain].[TLD]/directory?key=value>

$ exticz <[domain].[TLD]> | <[domain].[TLD]/directory> | <[domain].[TLD]/directory?key=value>

While using arguments as input it is recommended to enclose the url in quotes

Installation

  git clone https://github.com/Kadetron/Exticz  
  cd Exticz
  sudo mv exticz /usr/local/bin
  cd ..
  rm -r ./Exticz

Features

• Uses https scheme by default when no scheme is specified in input
• Highlight the base urls to have a consistent format. The sub urls come under the highlighted base urls
• Accepts any kind of valid links
• Can be used in a command chain
• Removes successive repeated urls from the result
• Prints the number of links at the end

Dependencies

• awk
• curl
• grep
• sed
• uniq
• tee
• wc