New policy for excluding locations from our Mapbox and API exports

[Simon Willison]

Follow-on from #702 in which we learned that our logic that excludes locations from our API export and website if their last recorded non-skip report used certain availability tags ("May be a vaccination site in the future" etc).

We should reconsider this policy now that we rely much more heavily on data scraped and imported from other sources.

[Simon Willison]

Here's the current code:

vial/vaccinate/api/search.py

Lines 347 to 361 in 3aca574

 def filter_for_export(qs): 
     # Filter down to locations that we think should be exported 
     # to the public map on www.vaccinatethestates.com 
     return qs.exclude( 
         dn_latest_non_skip_report__planned_closure__lt=datetime.date.today() 
     ).exclude( 
         dn_latest_non_skip_report__availability_tags__slug__in=( 
             "incorrect_contact_information", 
             "location_permanently_closed", 
             "may_be_a_vaccination_site_in_the_future", 
             "not_open_to_the_public", 
             "will_never_be_a_vaccination_site", 
             "only_staff", 
         ) 
     ) 

[Simon Willison]

Could we do this based on the date of that report?

select location.public_id, location.name, location.full_address, report.created_at, json_agg(availability_tag.name)
from location join report on location.dn_latest_non_skip_report_id = report.id
join call_report_availability_tag on call_report_availability_tag.report_id = report.id
join availability_tag on call_report_availability_tag.availabilitytag_id = availability_tag.id
where location. soft_deleted = false and location.id in (select location_id from concordance_location where concordanceidentifier_id in (select id from concordance_identifier where authority = '_issue' and identifier = '701')) group by location.public_id, location.name, location.full_address, report.created_at order by report.created_at desc

https://vial.calltheshots.us/dashboard/?sql=select+location.public_id%2C+location.name%2C+location.full_address%2C+report.created_at%2C+json_agg%28availability_tag.name%29%0D%0Afrom+location+join+report+on+location.dn_latest_non_skip_report_id+%3D+report.id%0D%0Ajoin+call_report_availability_tag+on+call_report_availability_tag.report_id+%3D+report.id%0D%0Ajoin+availability_tag+on+call_report_availability_tag.availabilitytag_id+%3D+availability_tag.id%0D%0Awhere+location.+soft_deleted+%3D+false+and+location.id+in+%28select+location_id+from+concordance_location+where+concordanceidentifier_id+in+%28select+id+from+concordance_identifier+where+authority+%3D+%27_issue%27+and+identifier+%3D+%27701%27%29%29+group+by+location.public_id%2C+location.name%2C+location.full_address%2C+report.created_at+order+by+report.created_at+desc%3ALdkKfys_atKjhtWiSqijkWvPIErh7ZTDLwjOqwc5cJA

That shows the dates on the reports, for the locations we investigated in #702 - sorted by most recent at the top. 2021-06-19 16:23 is the most recent one, the oldest is 2021-03-30 and it looks like about a third of them are June, almost all of the rest are May, a few are April and March.

[Simon Willison]

The issue here is actually a pretty big one: at what point do we consider a location "closed"? Do we stop showing it on our mapbox-powered vaccinatethestates.com site? Do we also exclude it from our API export?

[Simon Willison]

A plan for this is forming in #712.