oduwsdl / hypercane

A toolkit for developing algorithms that sample mementos from a web archive collection.

Home Page:https://oduwsdl.github.io/hypercane

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Test that Hypercane works effectively with the proxy servers specified in HTTP*_PROXY environment variables

shawnmjones opened this issue · comments

Most *nix commands honor the HTTP_PROXY and HTTPS_PROXY environment variables. Hypercane processes these variables and applies them in hypercane/utils.py as part of get_web_session. We need to test this with Squid or Varnish to ensure the system will actually use a proxy server as a datastore.

We use the proxy as a storage cache for content. Many proxies just provide a pass-thru for HTTPS traffic and do not cache it. This is not sufficient for our purposes. We truly need to save mementos so we can process them multiple times, hence our existing MongoDB dependency. Merely supporting these environment variables is insufficient for Hypercane's needs.

This no longer makes sense when considering #65.