Brunopaes / sigef-crawler

This project aims in crawling through sigef pages, extract information and parses it into a csv file.

Sigef Crawler

Optimized for python 3.6+

This project aims in crawling through sigef pages, extract information and parses it into a csv file.

---

Dependencies

For installing the requirements, in your venv or anaconda env, just run the following command:

pip install -r requirements.txt

---

Project's Structure

.
└── sigef-crawler
    ├── data
    │   ├── links.txt
    │   └── outputs
    │       ├── sigef-2020-07-05.csv
    │       ├── sigef-2020-07-06.csv
    │       └── sigef-2020-07-07.csv
    ├── docs
    │   └── CREDITS
    ├── src
    │   ├── __init__.py
    │   └── crawler.py
    ├── tests
    │   └── unittests
    │       └── __init__.py
    ├── .gitignore
    ├── LICENSE
    ├── README.md
    └── requirements.txt

Directory description

• data: The data dir. Group of non-script support files.
• docs: The documentation dir.
• src: The scripts & source code dir.
• tests: The unittests dir.

---

Usage Notes

Section aimed on clarifying some running issues.

Running

For running it, at the ~/src directory just run:

python crawler.py

or, if importing it as a module, just run:

from crawler import SigefRequests

if __name__ == '__main__':
    SigefRequests('path/to/file').__call__()

---