As a subscriber of magazine called dTest, I have access to online archive of all published articles/reviews. I want to make it accessible for offline usage for my parents. As there is no provided option to download articles on dTest website and there are too many articles to download it manually I decided to download it programmatically. For each article/test I want full copy of page with images and everything visible in form of html page. I do not want it as PDF as I am not satisfied with broken layout.
Disclaimer: There isn't any intend to illegally share downloaded content or any other bad intent.
The very first and most straighforward steps to get the job done that comes to my mind are like this:
- Iterate over paginator on page with list of articles for store all links to article.
- Iterate over all articles page bodies and find links to comparision pages (article can have one or more links to sub-article with products comparision) and store them.
- Persist parsed tree hierarchy of links (and titles). Structure is going to be used as list of tasks for download stage.
- Copy cookies from a browser after authentication on a website to use it later.
- Download all the links using provided cookies to access content behind paywall.
- Create simple html page in root directory for navigation across downloaded content.
- The easiest way I found to webscrap webpage using
wgetutility, inspired by this gist (thanks Dannguyen). - Kotlin/Gradle for the rest, because as a Java/Maven guy and wanna taste a different kind of coffee.
- jsoup: HTML Parser lib and gson
- I am running on Windows so there is usage of WSL to easily reach
wget - It's a freetime micro project for one-time usage with no future plans