rodp63 / yams

A simple Python scraper to collect data from the most popular Peruvian news sites

Yet Another Media Scraper

$ yams
Usage: yams [OPTIONS] COMMAND [ARGS]...

Options:
  --version   Show the version and exit.
  -h, --help  Show this message and exit.

Commands:
  info   Display useful information
  start  Start a crawling process

Installation

$ python setup.py install

Examples

$ yams start newspaper --help 
Usage: yams start newspaper [OPTIONS] SOURCE

Options:
  -k, --keyword TEXT      Set one keyword for post retrieval  [required]
  -o, --output FILE       Save output to json FILE instead of stdout
  -s, --since [%Y-%m-%d]  Set the lower date of the posts to retrieve
  -t, --to [%Y-%m-%d]     Set the upper date of the posts to retrieve
  --exact-match           Look for the exact match of the keywords
  -h, --help              Show this message and exit.

# Get all the posts containing the word 'peru' in the last month (by default).
$ yams start newspaper elcomercio -k peru
# Get all the post containing the exact word 'congreso' between two dates and save it in a json file.
$ yams start newspaper elcomercio -k congreso -s '2023-01-01' -t '2023-06-30' -o data --exact-match

Deployment

$ make build-image
$ make upload-image

Formatting

$ make lint