failed to build dictionary: dictionary

[D4rth R3v4n]

Hello @stefanoj3 ,

When running the below command, specifically placing single quotes in the 2nd to last flag I got the aforementioned error., Interestingly if I removed the quotes it worked and later in testing i was able to get it to successfully run with the single quotes. Its not a major issue but am raising it so you know it exists

docker run stefanoj3/dirstalk dirstalk scan http://example.com -d "https://raw.githubusercontent.com/daviddias/node-dirbuster/master/lists/directory-list-1.0.txt" --no-check-certificate --http-statuses-to-ignore '404,301' -t 10
Error: failed to build dictionary: dictionary: failed to get `https://raw.githubusercontent.com/daviddias/node-dirbuster/master/lists/directory-list-1.0.txt`: Get https://raw.githubusercontent.com/daviddias/node-dirbuster/master/lists/directory-list-1.0.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Usage:
  dirstalk scan [url] [flags]

Flags:
      --cookie stringArray             cookie to add to each request; eg name=value (can be specified multiple times)
  -d, --dictionary string              dictionary to use for the scan (path to local file or remote url)
      --header stringArray             header to add to each request; eg name=value (can be specified multiple times)
  -h, --help                           help for scan
      --http-cache-requests            cache requests to avoid performing the same request multiple times within the same scan (EG if the server reply with the same redirect location multiple times, dirstalk will follow it only once) (default true)
      --http-methods strings           comma separated list of http methods to use; eg: GET,POST,PUT (default [GET])
      --http-statuses-to-ignore ints   comma separated list of http statuses to ignore when showing and processing results; eg: 404,301 (default [404])
      --http-timeout int               timeout in milliseconds (default 5000)
      --no-check-certificate           to skip checking the validity of SSL certificates
      --out string                     path where to store result output
      --scan-depth int                 scan depth (default 3)
      --socks5 string                  socks5 host to use
  -t, --threads int                    amount of threads for concurrent requests (default 3)
      --use-cookie-jar                 enables the use of a cookie jar: it will retain any cookie sent from the server and send them for the following requests
      --user-agent string              user agent to use for http requests

Global Flags:
  -v, --verbose   verbose mode

level=fatal msg="Execution error" err="failed to build dictionary: dictionary: failed to get `https://raw.githubusercontent.com/daviddias/node-dirbuster/master/lists/directory-list-1.0.txt`: Get https://raw.githubusercontent.com/daviddias/node-dirbuster/master/lists/directory-list-1.0.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

[D4rth R3v4n]

I played with it a bit more but it does seem its not exclusive to that flag, it just some times fails to build at random times and I cant deduce why

[Stefano Gabryel]

net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Seems like it has to do with timeout, do you think it could relate to your connection or a temporary downtime of the server?

I could add a retry mechanism to make it less likely to fail

[Stefano Gabryel]

Hey @gbiagomba , I realized where the issue might have been and merged a PR that should fix the issue you are experiencing.

Here you can read about what was changed.

I've tagged a new release, you can find it here.

[Stefano Gabryel]

Closing now because I believe it should be solved, if that is not the case please let me know.

[D4rth R3v4n]

@stefanoj3 Thank you and sorry for the late response I will check it out and let you know if i find anything else!