- Before you download the Console App, make sure you have DotNet 6 installed.
- Go to release section of this repository and download latest version as zip (e.g. ExpertPriceCrawler-0.5.zip)
- Extract the zip archive to any directory on your computer
- Open the directory to which you extracted the archive with explorer
- Open a Terminal Window (Powershell, Bash, ZSH or whatever) and execute
dotnet ExpertPriceCrawler.Shell.dll- When Windows warns you that the file might hurt your computer, you must proceed the execution. If you don't trust the Application, you can't use it. Sorry :)
- The Application will ask you for expert product url
- On first usage, the App will download a chromium browser. Therefore the first crawling might take a few minutes. Just be patient and wait until you see many messages in the Console window.
If you want to run the WebApp on your computer or server(preferred), there are three ways:
- Simply run the dotnet app using cli
- Run the webapp hosted inside IIS on Windows machines
- Let the WebApp run inside a Container using Docker
- Create a file named
expertpricecrawler.envat any desired folder The content of that file might look like thisASPNETCORE_ENVIRONMENT=Production PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable Serilog__MinimumLevel=Verbose SmtpServer__From=noreply@domain.tld SmtpServer__Hostname=mailserver.domain.tld SmtpServer__Password=anyawesomepassword SmtpServer__Port=587 SmtpServer__Username=anyusernameavailableonyourmailserver SmtpServer__UseSsl=true Logging__Console__FormatterName=json Options__CrawlerType=BrowserCrawler Options__MaxParallelRequests=5 Options__MemoryCacheMinutes=5 Options__RateLimitInMinutes=1 - In that folder, open a command line (Powershell, Bash, ...)
- RUN
docker run -p 80:80 --env-file expertpricecrawler.env awinnen/expertpricecrawler:latest - Open any Browser and navigate to http://localhost:80 (http is important! https will not work. If you can't reach that site, make sure your browser is not redirecting to https due to htst!)