POCP scraper
djhume opened this issue · comments
Add the pocp scraper - from existing code for a start
How do we deal with mappings/meta data? The pocp scraper uses a call to our DW that grabs a generation/type mapping... I also add to this mapping manually (not complete enough). I guess for now could push up a json dict or similar??
Ok, think I'll stick with a raw data grab for now - keep the munging bit separate...
At the moment I'm working on a general Database class.
Just been working with escape characters and file permissions, been a major pain in the...
Just got a working energy offer database up and working.
I think I may set these up to work on a year by year basis in order to improve the query performance.
Touch slow at the moment with all of the energy offer data dumped into PostGres
Shit, didn't even answer the question.
I'm happy to have general purpose mappings in the static folder
Ok - will whack some of the pocp mappings in there. cool.