URL Extraction - Build incomplete URLs

Question

URL Extraction - Build incomplete URLs

jjackson37 opened this issue 5 years ago · comments

Need a way to convert incomplete URLs to usable ones, e.g. "/Foo/Bar" to "example.com/Foo/Bar".
The process needs to identify these URLs and convert them without affecting the currently complete URLs.
Possibly think about seperating out classes with RegEx that will return these URL types?

BoggartBear · Answer 1 · Mon Mar 04 2019 18:41:00 GMT+0800 (China Standard Time)

We also need to handle URLs that are missing HTTP or HTTPS.

This is mainly for QoL reasons, but also we can extend this to examine and fix URLs for errors such as mistypes; "htp//" for example.

BoggartBear · Answer 2 · Mon Mar 04 2019 20:58:10 GMT+0800 (China Standard Time)

Need to check the input has a top level domain (.com, .uk, .gov, etc).

@jjackson37 how should we store these, new domains are added often, so it would need to in a config or eventual DB.

BoggartBear · Answer 3 · Thu Mar 07 2019 22:09:30 GMT+0800 (China Standard Time)

RetrieveFile method has been added to the WebClientPageRetriever. This takes a file URL and the directory in which to store the file and downloads it.