brunowego / scraping-criminal-statistics

Extraction of criminal statistics from the Goiás state government portal.

Scraping Criminal Statistics

Extraction of criminal statistics from the Goiás state government portal.

References

• Goiás State Statistics Portal
• tabula-py
• Camelot
• Excalibur

Examples

• tabula-py example notebook

Tools

tabula-py

Dependencies

pip install tabula-py

REPL

>>> import tabula
>>>
>>> file = 'https://www.seguranca.go.gov.br/wp-content/uploads/2020/11/relatorio-2020-jan-e-set.pdf'
>>>
>>> tables = tabula.read_pdf(file, pages='all', multiple_tables=True, stream=True)
>>>
>>> tabula.convert_into(file, './tabula_tables.csv', output_format='csv', pages='all')
>>> exit()

Camelot

Dependencies

• Ghostscript

pip install camelot-py opencv-python

REPL

>>> import camelot
>>>
>>> file = 'https://www.seguranca.go.gov.br/wp-content/uploads/2020/11/relatorio-2020-jan-e-set.pdf'
>>>
>>> tables = camelot.read_pdf(file, pages='1-end', flavor='stream')
>>>
>>> tables[0].df
>>> tables[0].parsing_report
>>>
>>> tables.export('./camelot_tables.csv', f='csv')
>>> exit()

Excalibur

Dependencies

• Ghostscript

Running Web Server

pip install excalibur-py

# Change to use '~/.excalibur' folder instead of '~/excalibur'
export EXCALIBUR_HOME="$HOME/.excalibur"

# Initialize locally database
excalibur initdb

# Run web server
excalibur webserver

# Open on web browser
echo -e '[INFO]\thttp://127.0.0.1:5000'