Generate new places to watch data when GLAD is updated
This code intersects GLAD points with a grid that has been scored for the amount of IFL and WDPA area within each cell. After this intersection, it multiplies the count of GLAD points by the importance score (determined by IFL and WDPA area) to help us identify important places of forest change.
We select the top 10 high scoring grid cells for each region (Central America, South America, Africa, etc) and push them to a CARTO table. These are then surfaced on the staging map under the Uncurated Places to Watch toggle on the GLAD dataset.
These areas are then vetted by our staff to determine what stories we'll share as our offical Places to Watch. With the help of Vizzuality, these get added to a separate CARTO table, and are viewable on the flagship.
How goes this work? We use hadoop_pip to start a hadoop cluster and run the GLAD / grid intersection for us. We then grab the output table (CSV of alerts per grid and the total grid score) and then do the math to figure out the top 10 per region. This data is pushed to the CARTO ptw_top_10 table, then should appear in the uncurated table on the staging map.
- Clone hadoop_pip in the same directory that this repo is sitting in
- Create a
tokensdirectory in this repo, with a file calledwri-01@cartodbin it that contains our CARTO API token. - Check the
processing/hadoop_ptw.inifile to be sure it's up to date. This file contains:
- Grid polygon TSV:
s3://gfw2-data/alerts-tsv/ptw/ptw_grid.tsv - Path to GLAD point directory:
s3://gfw2-data/alerts-tsv/glad/ - SQL query to sum alert area_ha and count alerts, grouping by grid ID:
SELECT _c3, _c4, sum(_c5), sum(_c6), _c8, _c9, count(*) FROM my_table GROUP BY _c3, _c4, _c8, _c9
python update_ptw.py --min-date 2018-07-01 --max-date 2018-09-30