Abrupt exit or crash

[Anmol Varshney]

Iteration 9
A possible issue of crashing was identified and the following plan was thought:

• In case the crawler shuts down abruptly, it should start from the last fetched URL in order to avoid doing the same work again.
• Log File(s) for the already seen and already processed URL(s) can be maintained
• System upon restarting can be initialised from the point of exit from these log files.

The work should be completed by 24 June.