fgregg / legistar-scrape

Legistar Scraper is a python library for scraping Legistar sites -- legislation management sites hosted by by Granicus.

THIS REPOSITORY HAS BEEN DEPRECATED AND IS NO LONGER BEING MAINTAINED. IT HAS BEEN SUPERSEDED BY https://github.com/opencivicdata/python-legistar-scraper AND https://github.com/opencivicdata/scrapers-us-municipal

Legistar Scraper is a python library for scraping Legistar sites -- legislation management sites hosted by by Granicus.

Legistar sites include

• Chicago
• Philadelphia
• Oakland
• and many other cities

This software is under active development. It is currently known to work for Chicago and Philadelphia.

Installation

> pip install -r requirements.txt
> python setup.py install 

Note: The current stable branch of mechanize has a bug in it. If you are installing the dependencies by hand, use https://github.com/abielr/mechanize.

Tests

To run all unit tests:

nosetests

To run a single test:

nosetests tests/test_legistar_scraper.py:name_of_test

For example, if you wanted to test just the council members pagination

nosetests tests/test_legistar_scraper.py:paging_through_council_members

Example usage

• powering Councilmatic
• saving council agendas in to MongoDB
• saving councilmembers to a CSV

Sample script

from legistar.scraper import LegistarScraper
from legistar.config import Config, DEFAULT_CONFIG

#__________________________________________________________________________
#
# Configure and create a scraper

config = Config(
  hostname = 'chicago.legistar.com',
).defaults(DEFAULT_CONFIG)
scraper = LegistarScraper(config)

#__________________________________________________________________________
#
# Get a summary listing of all of the legislation

all_legislation = scraper.searchLegislation('')

#__________________________________________________________________________
#
# Get more detail for a particular piece of legislation

for legislation_summary in all_legislation:
  (legislation_attrs, legislation_history) = \
    scraper.expandLegislationSummary(legislation_summary)
  break
# NOTE: searchLegislation returns an iterator; you may not use subscript
# indexing (e.g., all_legislation[0]). You may, however, achieve the same
# thing with all_legislation.next()

#__________________________________________________________________________
#
# Get details about legislation history, such as voting results

for history_summary in legislation_history:
  (history_detail, votes) = scraper.expandHistorySummary(history_summary)

#__________________________________________________________________________
#
# Get a list of all council members


councilmembers = scraper.councilMembers()

#__________________________________________________________________________
#
# Get a list of all upcoming agendas

councilmembers = scraper.councilCalendar()

#__________________________________________________________________________
#
# Get a list of all agendas (including past ones)

councilmembers = scraper.councilCalendar('all')