LuomingXu / bilibiliSpider

Graduation Project

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

bilibili spider

graduation project repo

Require

dependencies
db sqlalchemy, postgreSQL
http urllib3
s3 boto3, minio
xml bs4, lxml
util selfusepy
cache redis
tips
不再记录danmaku的信息
tables(av_info, av_stat, user)
multi_danmaku_v3_java 已弃用
ids_spliced = [ids[i:i + size] for i in range(0, len(ids), size)]
list.sort(key = lambda k: k["id"])
注意引用传递
use map.pop(key, None)
multiprocess Pool要用Manage().Queue()

About

Graduation Project


Languages

Language:Python 71.1%Language:Java 28.8%Language:Dockerfile 0.2%