Sub-page requesting - Keep track of already requested URLs

[Jake Jackson]

For any pages we request we should make sure that we have not already requested it, we need a quick way to keep track of something like this.
I would suggest maybe looking towards dictionaries or hashmaps.