corpus

There are 20 repositories under corpus topic.

brightmart / nlp_chinese_corpus
大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP
chinese-dataset chinese-corpus pretrain word2vec nlp bert language-model wiki news question-answering chinese corpus chinese-nlp dataset text-classification
9204
dariusk / corpora
A collection of small corpuses of interesting data for the creation of bots and similar stuff.
language words bots corpus
Language:JavaScript 4861
CLUEbenchmark / CLUEDatasetSearch
搜索所有中文NLP数据集，附常用英文NLP数据集
nlp datasets chinese ner qa match text-classification machine-translation knowledge-graph corpus machine-reading-comprehension sentiment-analysis text-similarity text-summarization
Language:Python 3916
CLUEbenchmark / CLUE
中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard
nlu benchmark chinese corpus dataset bert albert chineseglue glue roberta language-model pretrained-models transformers tensorflow pytorch
Language:Python 3848
wainshine / Chinese-Names-Corpus
中文人名语料库。人名生成器。中文姓名,姓氏,名字,称呼,日本人名,翻译人名,英文人名。可用于中文分词、人名实体识别。
corpus dataset dict names ner
3845
adbar / trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
web-scraping text-extraction nlp html2text news text-mining crawler text-cleaning text-preprocessing article-extractor readability scraping news-crawler tei html-to-markdown corpus corpus-builder corpus-tools rss-feed news-aggregator
Language:Python 2929
endymecy / awesome-deeplearning-resources
Deep Learning and deep reinforcement learning research papers and some codes
deep-learning reinforcement-learning neural-network paper code video modelzoo corpus nlp
2824
lucasjinreal / weibo_terminater
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
scraper weibo sina corpus chinese chatbot
Language:Python 2312
candlewill / Dialog_Corpus
用于训练中英文对话系统的语料库 Datasets for Training Chatbot System
chatbot corpus dataset dialog system
Language:Python 2011
fendouai / Awesome-Chatbot
Awesome Chatbot Projects,Corpus,Papers,Tutorials.Chinese Chatbot =>:
chatbot tensorflow awesome seq2seq seq2seq-chatbot seq2seq-model corpus tutorial
Language:Python 2008
gunthercox / chatterbot-corpus
A multilingual dialog corpus
chatterbot corpus dialog language yaml
Language:Python 1346
wainshine / Company-Names-Corpus
公司名语料库。机构名语料库。公司简称,缩写,品牌词,企业名。可用于中文分词、机构名实体识别。
company corpus dict ner dataset
1203
chatopera / insuranceqa-corpus-zh
:helicopter: 保险行业语料库，聊天机器人
corpus chatbot qasystem natural-language-processing natural-language-understanding machine-learning dataset question-answering insurance insuranceqa-corpus-zh
Language:Python 997
NiuTrans / Classical-Modern
非常全的文言文（古文）-现代文平行语料
corpus parallel-corpus traditional-chinese traditional-and-simplified-chinese
Language:Python 909
CLUEbenchmark / CLUECorpus2020
Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料
albert bert chinese chinese-corpus corpus datasets nlp pretrain roberta
886
OYE93 / Chinese-NLP-Corpus
Collections of Chinese NLP corpus
chinese-nlp corpus datasets
Language:Python 851
tensorlayer / seq2seq-chatbot
Chatbot in 200 lines of code using TensorLayer
tensorlayer tensorflow chatbot rnn lstm bot nlp chat corpus python
Language:Python 836
quanteda / quanteda
An R package for the Quantitative Analysis of Textual Data
corpus natural-language-processing quanteda r text-analytics
Language:R 826
CLUEbenchmark / CLUEPretrainedModels
高质量中文预训练模型集合：最先进大模型、最快小模型、相似度专门模型
pretrained-models chinese bert roberta albert sentence-classification sentence-pairs semantic-similarity sentence-analysis text-classification distillation corpus dataset
Language:Python 788
soskek / bookcorpus
Crawl BookCorpus
bookcorpus corpus crawler nlp scraper
Language:Python 780
PlexPt / chatgpt-corpus
ChatGPT 中文语料库对话语料小说语料客服语料用于训练大模型
awesome corpus corpus-data question-answering
767
mhbashari / awesome-persian-nlp-ir
Curated List of Persian Natural Language Processing and Information Retrieval Tools and Resources
persian-language natural-language-processing information-retrieval language-detection persian-nlp corpus part-of-speech-tagger normalizer named-entity-recognition embeddings morphological-analysis stemmer dependency-parser spell-check persian-stemmer shallow-parser
700
CBLUEbenchmark / CBLUE
中文医疗信息处理基准CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
benchmark chineseblue corpus dataset chinese biomedical-tasks evaluation acl2022
Language:Python 677
BLKSerene / Wordless
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
corpus corpus-analysis corpus-linguistics corpus-processing corpus-search corpus-statistics corpus-tools dependency-parser lemmatizer linguistics literature stopword tagger tokenizer translation
Language:Python 672
ko-nlp / Korpora
Korean corpus repository
corpus korean-natural-language
Language:Python 653
nonamestreet / weixin_public_corpus
微信公众号语料库
chinese-nlp corpora corpus linguistics natural-language-processing nlp wei-xin weixin weixin-data yu-liao yu-liao-ku
563
chatopera / efaqa-corpus-zh
❤️Emotional First Aid Dataset, 心理咨询问答、聊天机器人语料库
natural-language-processing natural-language-understanding corpus psychology
Language:Python 557
crownpku / Small-Chinese-Corpus
Some useful Chinese corpus datasets 中文语料小数据
chinese-nlp corpus
526
louisowen6 / NLP_bahasa_resources
A Curated List of Dataset and Usable Library Resources for NLP in Bahasa Indonesia
natural-language-processing bahasa-indonesia dataset library packages nlp sentiment-analysis nlp-bahasa-resources indonesian indonesian-language sentiment-analysis-dataset corpus corpus-linguistics
438
several27 / FakeNewsCorpus
A dataset of millions of news articles scraped from a curated list of data sources.
artificial-intelligence corpus database dataset fakenews machine-learning natural-language-processing nlp
375
GAIR-NLP / MathPile
Generative AI for Math: MathPile
corpus language-model large-language-models math pre-training
Language:JavaScript 347
guhhhhaa / 4675-scifi
chinese NLP corpus of chinese science fiction,chinese science fiction corpus : About 4675 Chinese science fiction novels 大约有4675本科幻小说，中文科幻小说自然语言处理语料库，中文科幻小说文本语料库，中文科幻小说文本数据库，科幻小说语料
scifi corpus corpus-data nlp nlp-datasets nlp-machine-learning nlp-resources science-fiction chinese-nlp datasets
325
ko-ichi-h / khcoder
KH Coder: for Quantitative Content Analysis or Text Mining
content-analysis text-mining visualization kwic corpus
Language:Perl 305
malaysian-dataset
mesolitica / malaysian-dataset
We gather Malaysian dataset! https://malaysian-dataset.readthedocs.io/
text-mining corpus malaysia malay-dataset bahasa-melayu manglish
Language:Jupyter Notebook 284
grammarly / ua-gec
UA-GEC: Grammatical Error Correction and Fluency Corpus for the Ukrainian Language
dataset corpus gec grammatical-error-correction ukrainian-language corpus-data corpus-tools natural-language-processing nlp-datasets
Language:Macaulay2 254
lil-lab / nlvr
Cornell NLVR and NLVR2 are natural language grounding datasets. Each example shows a visual input and a sentence describing it, and is annotated with the truth-value of the sentence.
natural-language-processing computer-vision machine-learning corpus
Language:HTML 251

corpus

brightmart / nlp_chinese_corpus

dariusk / corpora

CLUEbenchmark / CLUEDatasetSearch

CLUEbenchmark / CLUE

wainshine / Chinese-Names-Corpus

adbar / trafilatura

endymecy / awesome-deeplearning-resources

lucasjinreal / weibo_terminater

candlewill / Dialog_Corpus

fendouai / Awesome-Chatbot

gunthercox / chatterbot-corpus

wainshine / Company-Names-Corpus

chatopera / insuranceqa-corpus-zh

NiuTrans / Classical-Modern

CLUEbenchmark / CLUECorpus2020

OYE93 / Chinese-NLP-Corpus

tensorlayer / seq2seq-chatbot

quanteda / quanteda

CLUEbenchmark / CLUEPretrainedModels

soskek / bookcorpus

PlexPt / chatgpt-corpus

mhbashari / awesome-persian-nlp-ir

CBLUEbenchmark / CBLUE

BLKSerene / Wordless

ko-nlp / Korpora

nonamestreet / weixin_public_corpus

chatopera / efaqa-corpus-zh

crownpku / Small-Chinese-Corpus

louisowen6 / NLP_bahasa_resources

several27 / FakeNewsCorpus

GAIR-NLP / MathPile

guhhhhaa / 4675-scifi

ko-ichi-h / khcoder

mesolitica / malaysian-dataset

grammarly / ua-gec

lil-lab / nlvr