andreformento / querido-diario-data-processing

querido-diario-data-processing

Setup

• Install podman
• execute setup stage:

make setup

Populate data

Populate data following this instructions.

• you can see created data inside storage using local credentials
• you can see gazettes not processed yet connecting on database
• open database console in a new terminal

make shell-database

• and run a query to see gazettes not processed

select processed, count(1) from gazettes g group by processed;

Run

• execute processing stage:

make re-run

• and see gazettes processed running the query above
• you can search using ElasticSearch

curl 'http://localhost:9200/querido-diario/_search' \
  -H 'Content-Type: application/json' \
  --data-raw '{"query":{"query_string":{"query":"*"}},"size":2}'