Use an alternative geocoding API for News Channel
larsenv opened this issue · comments
On June 11, Google is changing their API to have new pricing and be integrated in Google Cloud API.
You get $200 in free credit per month, which can give us 40,000 geocoding calls (every thousand requests after that are $5 each). I checked out Datadog dashboard and we have used 66K locations in the past month. That means we'd have to pay over $100 for the rest of the 26K requests which I'm not paying for. That's disappointing.
Either I can make a dictionary of the most common locations so we don't have to geocode them (Chicago, Paris, Sydney, etc.) or use a different API. (For Google Maps API you will now have to set up billing and I'd have to get my dad to pay it since I don't have a credit card, and he wouldn't be happy if the bill is high.)
For the Associated Press, here's the locations not requiring a state or country. I think we can make all these locations be in a local dictionary so we don't have to geocode them, that would save a lot of requests.
I took a quick look at alternative APIs and looked at geopy's documentation, here's some quick looks at APIs:
- OpenStreetMap Nominatim - @spotlightishere recommended this, but you can only make one request per second. I'm not interested in it for that reason since it'd slow down the script.
- Yahoo BOSS - I don't think this is still up, but I don't like Yahoo in general so I don't want to use it
- There's other APIs I didn't look out, but they don't seem to support multiple languages.
- Bing API - The prices aren't public for this one, but they support multiple languages and looks pretty good. Hopefully this one can be used.
Maybe you could write a script and let it parse everything?
Why would we want to manually hardcode all of our locations? Places change.
It'd be easiest to cache and update said cache every so often.
Yeah, basically my idea was to cache them.
My idea isn't to hardcode all locations, I'm just planning to hardcode the most commonly used cities (basically all the ones that AP doesn't write the state or country for in their byline, I would predict that's about 50 cities) and that should save thousands of requests.
Remember that the script is only requesting the city, and if New York changed places overnight I'd be surprised.
suprise
I found that the locations were being requested twice per build and the actual number of locations were doubled. This was because it was doing it once to make the file for Wii, and one for Wii U (there's support to make files for Wii U with different timestamps but it's not used yet). So it went through 33K locations.
Ultimately the most common locations will still be hard-coded.
Got the billing thing worked out, we're now prepared for the API change.