dqsdatalabs's repositories
openapi-generator
OpenAPI Generator allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Spec (v2, v3)
burplist
Web crawlers for Burplist, a search engine or craft beers in Singapore
scrapy-boilerplate
Scrapy project boilerplate done right
flower
Real-time monitor and web admin for Celery distributed task queue
aio-scrapy
将基于twisted的scrapy/scrapy-redis改成基于asyncio,使用aiohttp发送请求
scrapy_utils
scrapy_utils is configuration template project, Contains the extraction of scrapy configuration
GerapyAutoExtractor
Auto Extractor Module
X-news
Pipeline data use scrapy, kafka, spark streaming, spark ML and elasticsearch, Kibana
ScrapyDouban
豆瓣电影/豆瓣读书 Scarpy 爬虫
scrapy-poet
Page Object pattern for Scrapy
Gerapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
scrapy_demo
all kinds of scrapy demo
scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Darkweb-search-engine
Dark Web & Deep Web Search Engine. Data Crawler and indexer for Darkweb , OSINT Tools for the Dark Web
apachecn-python-zh
:books: ApacheCN Python 译文集
libcloud
Apache Libcloud is a Python library which hides differences between different cloud provider APIs and allows you to manage different cloud resources through a unified and easy to use API
advertools
advertools - online marketing productivity and analysis tools
httpbin
HTTP Request & Response Service, written in Python + Flask.
CrawlerX
CrawlerX - Develop Extensible, Distributed, Scalable Crawler System which is a web platform that can be used to crawl URLs in different kind of protocols in a distributed way.
tenacity
Retrying library for Python
newspaper
News, full-text, and article metadata extraction in Python 3. Advanced docs:
requests-ip-rotator
A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.
scrapyrt
HTTP API for Scrapy spiders
docker-scrapyd
🕷️ Scrapyd is an application for deploying and running Scrapy spiders.
parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
almaren-framework
The Almaren Framework provides a simplified consistent minimalistic layer over Apache Spark. While still allowing you to take advantage of native Apache Spark features. You can still combine it with standard Spark code.
SparkPipelineFramework
Framework for simpler Spark Pipelines
scrapy-distributed
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.