tattle-made / factchecking-sites-scraper

A repo to store helper functions for scraping + experiments/visualisations

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Index factchecking articles into Tattle search services

variablenerd opened this issue · comments

commented

Background: The data collected by Tattle is indexed into its search services (Simple Search and Tattle Search, whose descriptions can be found https://github.com/tattle-made/simple-rt-search and https://github.com/tattle-made/tattle-api). Tattle's search services enable misinformation research and power user-facing solutions such as the Khoj app.

Issue: The factchecking scraper in this repository collects data from factchecking websites and stores it in a MongoDB. This data needs to be indexed in both the search services mentioned above, and the indexing status reports added back to the MongoDB. Reference scripts for indexing and reporting can be found in the Sharechat scraper repository here - https://github.com/tattle-made/sharechat-scraper/tree/master/workers. Note that the reporter is a RabbitMQ consumer.

Task summary: To create indexer and reporter scripts for the factchecking database, similar to what has been done for the Sharechat db. Please reach out to @dennyabrain for more information.