codeforboston / maple

MAPLE makes it easy for anyone to view and submit testimony to the Massachusetts Legislature about the bills that will shape our future.

Home Page:https://mapletestimony.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Modify eventScraper to pull in DocumentHistoryActions instead of Hearings

mvictor55 opened this issue · comments

Modify scrapeEvents function to pull in DocumentHistoryActions instead of Hearings.

Currently, the scrapeEvents function is making use of the listEvents function from the malegislature file.

See:

https://github.com/codeforboston/maple/blob/main/functions/src/malegislature.ts
https://github.com/codeforboston/maple/blob/main/functions/src/events/scrapeEvents.ts

The backbone for publishing notifications currently already exists in the publishNotifications cloud function. However, no notifications are published because the events it looks for are in the form bill-example-1 or org-example-1.

Right now, the events scraper (scrapeEvents) only scrapes hearings (scrapeHearings) and sessions (scrapeSessions) which are saved in the form hearing-1 and session-1-1, respectively. This needs to be expanded to scrape all DocumentHistoryActions and store them in a form that can be understood by publishNotifications.

  1. Create a new scraper class (DocumentHistoryActionsScraper) that is a child of the EventsScraper abstract class that all of the other scrapers inherit from (see: scrapeEvents.ts). Use the HearingScraper as reference.
  2. Add the scraper to the existing list of scrapers (see: runScrapers.ts)

the documenthistoryactions are called the "bill history" and it is scraped and saved as part of the bill info, used on the bill page, right now in the bill history modal. hopefully this is helpful--

https://github.com/codeforboston/maple/blob/main/functions/src/bills/bills.ts

image