Brunopaes / sigef-crawler

This project aims in crawling through sigef pages, extract information and parses it into a csv file.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Sigef Crawler

Optimized for python 3.6+

This project aims in crawling through sigef pages, extract information and parses it into a csv file.


Dependencies

For installing the requirements, in your venv or anaconda env, just run the following command:

pip install -r requirements.txt

Project's Structure

.
└── sigef-crawler
    ├── data
    │   ├── links.txt
    │   └── outputs
    │       ├── sigef-2020-07-05.csv
    │       ├── sigef-2020-07-06.csv
    │       └── sigef-2020-07-07.csv
    ├── docs
    │   └── CREDITS
    ├── src
    │   ├── __init__.py
    │   └── crawler.py
    ├── tests
    │   └── unittests
    │       └── __init__.py
    ├── .gitignore
    ├── LICENSE
    ├── README.md
    └── requirements.txt

Directory description

  • data: The data dir. Group of non-script support files.
  • docs: The documentation dir.
  • src: The scripts & source code dir.
  • tests: The unittests dir.

Usage Notes

Section aimed on clarifying some running issues.

Running

For running it, at the ~/src directory just run:

python crawler.py

or, if importing it as a module, just run:

from crawler import SigefRequests

if __name__ == '__main__':
    SigefRequests('path/to/file').__call__()

About

This project aims in crawling through sigef pages, extract information and parses it into a csv file.

License:MIT License


Languages

Language:Python 100.0%