medialab / minet

A webmining CLI tool & library for python.

medialab/minet Issues

Forward SelectionError to minet.scrape
Updated 6 days ago
CrawlJob data type should not be wrapped in an optional by default
Updated 8 days ago
Command to add jobs to a crawler's queue
Updated 8 days ago
Adjust twitter scraper retryer and rate limit (again)
Updated 15 days ago
Add more automatic context when Spider.process raises
Updated 22 days ago
KeyError: 'expanded_url' with minet twitter scrape tweets
Closed a month ago5
Issues (core dump or cannot unpack non-iterable FocusCrawlInfo object)
Updated a month ago1
potential changes in rate limit of twitter public API
Updated a month ago3
tiktok search-videos error
Updated 2 months ago3
When -c is not specified, we should default to test all available browsers instead of only firefox
Updated 2 months ago1
-I should default to "downloaded" in scrape and extract
Updated 2 months ago1
"Invalid Twitter cookie!" error (possibly due to migration from twitter.com to x.com ?)
Closed 2 months ago3
Spider process exceptions should at least be raised with some context around them
Updated 2 months ago
Retrieve videos from instagram hashtag function
Updated 3 months ago9
Scrapping 1000's of comments on Instagram
Updated 3 months ago3
Add FORWARD_SPIDER option
Updated 3 months ago
Error on wikipedia pageviews
Updated 3 months ago1
Draw edges kwarg
Updated 3 months ago
Improve ThreadsafeBrowser.request stability by retrying content acquisition if needed
Closed 3 months ago
ThreadsafeBrowser enhancements
Updated 4 months ago
Add LoadingBar.track
Updated 4 months ago
instagram post-infos should have line parity in the output and increase a stat rather than log
Closed 4 months ago
/usr/local/bin does not exist on recent mac installs
Closed 4 months ago
Add a playwright version of the crawler
Closed 4 months ago
Refactor Crawler request_args as inheritance
Updated 4 months ago
Spider process error should lead to errorred crawl?
Closed 4 months ago1
User friendly error message when spider returns a non-2-tuple
Closed 4 months ago
Multithreaded API clients connection pools should allow more connections
Closed 4 months ago
Upgrade trafilatura and deal with lxml_html_clean
Closed 4 months ago
Move away from lxml as default soup engine?
Closed 4 months ago
Method of CrawlJob to get an identical CrawlTarget to retry
Closed 4 months ago
Upgrade rich and other deps
Closed 4 months ago
Upgrade to min version py3.8
Closed 4 months ago
os.makedirs is already threadsafe
Closed 4 months ago
There should be a Crawler side global callback for each job
Updated 4 months ago
Crawler should lazily open data files to be written until there is actually something to write
Updated 4 months ago
Add some crawler level job filter
Updated 4 months ago
Make minet installable without playwright
Updated 4 months ago
Youtube improvements
Closed 4 months ago
A challenges challenge
Updated 4 months ago
Add a "minet hal" module?
Updated 4 months ago
ErroredCrawlReponse should de facto None response attributes
Closed 4 months ago
Command tw tweets should not require API key
Closed 4 months ago
x.com urls are not usually recognized
Closed 4 months ago
path column is not very useful when using --glob file on scrape/extract
Updated 5 months ago
Return None if TwitterGuestAPIScraper.tweet does not find tweet
Closed 6 months ago
Stop condition of twitter scraper is not working anymore
Closed 6 months ago
lzma issues
Closed 6 months ago
Telegram channel-messages crashes on string parsing
Closed 6 months ago
Try to scrape twitter embed for tweets rather than using the Guest scraper
Updated 6 months ago