Giters

opencivicdata / scrapers-us-municipal

Scrapers for US municipal governments.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Chicago: Handle events without a location

hancush opened this issue · comments

commented

Per the OCD standard, event locations can be an object or null. The pupa schema doesn't seem to agree with this. This is causing scrapes of events (specifically, Chicago events) without a location to fail.

We should either:

  • Update the pupa schema to allow for null event locations and make sure we're setting event location to null appropriately in the scaper
  • Add a placeholder event location to events without a location
  • Skip events without a location

Sentry link: https://sentry.io/organizations/datamade/issues/1605194328/?project=56420

Stacktrace:

ScrapeValueError: validation of Event e5ce52d6-b48b-11ea-b15d-122a3d729da3 failed: 
	None is not of type 'string'

Failed validating 'type' in schema['properties']['location']['properties']['name']:
    {'minLength': 1, 'type': 'string'}

On instance['location']['name']:
    None
  File "bin/pupa", line 8, in <module>
    sys.exit(main())
  File "pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "pupa/cli/commands/update.py", line 278, in handle
    return self.do_handle(args, other, juris)
  File "pupa/cli/commands/update.py", line 327, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "pupa/cli/commands/update.py", line 175, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "pupa/scrape/base.py", line 117, in do_scrape
    self.save_object(obj)
  File "pupa/scrape/base.py", line 100, in save_object
    raise ve
  File "pupa/scrape/base.py", line 97, in save_object
    obj.validate()
  File "pupa/scrape/base.py", line 200, in validate
    self.__class__.__name__, self._id, '\n\t'+'\n\t'.join(errors)

Event:

{
    "_id": "e5ce52d6-b48b-11ea-b15d-122a3d729da3",
    "agenda": [],
    "all_day": false,
    "classification": "event",
    "description": "tentative",
    "documents": [],
    "end_date": "",
    "extras": {},
    "links": [],
    "location": {
        "coordinates": null,
        "note": "",
        "name": null
    },
    "media": [],
    "name": "City Council",
    "participants": [
        {
            "organization_id": "~{\"name\": \"Chicago City Council\"}",
            "entity_type": "organization",
            "note": "participant",
            "name": "Chicago City Council"
        }
    ],
    "pupa_id": "5663",
    "sources": [
        {
            "url": "http://webapi.legistar.com/v1/chicago/events/5663",
            "note": "api"
        },
        {
            "url": "https://chicago.legistar.com/DepartmentDetail.aspx?ID=12357&GUID=4B24D5A9-FED0-4015-9154-6BFFFB2A8CB4&G=503964DF-290E-455B-86CC-7B03A231E984",
            "note": "web"
        }
    ],
    "start_date": "2020-09-09T15:00
}