text-processing

There are 23 repositories under text-processing topic.

learnbyexample / Command-line-text-processing
:zap: From finding text to search and replace, from sorting to beautifying text and more :art:
awk command-line ebook grep linux perl regex ruby sed text-processing
Language:Shell 10142
google / diff-match-patch
Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.
diff difference match patch text-processing
Language:Python 7125
chmln / sd
Intuitive find & replace CLI (sed alternative)
cli command-line regex rust terminal text-processing
Language:Rust 5390
pymupdf / PyMuPDF
PyMuPDF is a high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.
mupdf xps pdf-documents epub ocr pdf font python data-science extract-data table-extraction pymupdf tesseract text-processing text-shaping
Language:Python 4111
fastnlp / fastNLP
fastNLP: A Modularized and Extensible NLP Framework. Currently still in incubation.
chinese-nlp deep-learning natural-language-processing nlp-library nlp-parsing text-classification text-processing
Language:Python 3034
pyparsing / pyparsing
Python library for creating PEG parsers
parser-combinators parsing parsing-expression-grammar parsing-library peg-parsers python python-2 python-3 python2 python3 text-processing
Language:Python 2103
kk7nc / Text_Classification
Text Classification Algorithms: A Survey
text-classification nlp-machine-learning document-classification text-processing dimensionality-reduction rocchio-algorithm boosting-algorithms logistic-regression naive-bayes-classifier k-nearest-neighbours support-vector-machines decision-trees random-forest conditional-random-fields deep-learning deep-neural-network recurrent-neural-networks convolutional-neural-networks deep-belief-network hierarchical-attention-networks
Language:Python 1779
birchb1024 / frangipanni
Program to convert lines of text into a tree structure.
go golang text-processing tree-structure
Language:Go 1193
roshan-research / hazm
Persian NLP Toolkit
natural-language-processing python persian nlp persian-nlp dependency-parser embeddings text-processing tokenizer lemmatization farsi normalization pos-tagging
Language:Python 1112
lingua-go
pemistahl / lingua-go
The most accurate natural language detection library for Go, suitable for short text and mixed-language text
natural-language-processing language-detection language-recognition language-classification language-identification language-processing nlp nlp-machine-learning golang-library go language-modeling text-processing
Language:Go 1103
BurntSushi / aho-corasick
A fast implementation of Aho-Corasick in Rust.
aho-corasick finite-state-machine search substring-matching text-processing
Language:Rust 953
helix-editor / nucleo
A fast and convenient fuzzy matcher library for rust
fuzzy-matching fuzzy-search performance rust text-processing
Language:Rust 716
sstadick / hck
A sharp cut(1) clone.
command-line rust text-processing
Language:Rust 681
cbaziotis / ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp text-processing nlp-library spelling-correction tokenizer tokenization word-segmentation word-normalization spell-corrector text-segmentation semeval
Language:Python 658
derek73 / python-nameparser
A simple Python module for parsing human names into their individual components
python python-module text-parser text-processing
Language:Python 637
abadojack / whatlanggo
Natural language detection library for Go
language go nlp text-processing
Language:Go 630
open-korean-text / open-korean-text
Open Korean Text Processor - An Open-source Korean Text Processor
korean korean-text-processing natural-language-processing text-processing tokenizer korean-tokenizer
Language:Scala 598
ChenghaoMou / text-dedup
All-in-one text de-duplication
data-processing de-duplication nlp text-processing
Language:Python 495
proycon / pynlpl
PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).
nlp python computational-linguistics linguistics library folia machine-learning language-modelling search-algorithms evaluation-metrics text-processing nlp-library natural-language-processing
Language:Python 476
pyarabic
linuxscout / pyarabic
pyarabic
nlp-library arabic-language text-processing
Language:Python 422
andrewbihl / bsed
Simple SQL-like syntax on top of Perl text processing.
awk csv domain-specific-language grep perl python sed text-processing
Language:Python 408
airbnb / artificial-adversary
🗣️ Tool to generate adversarial text examples and test machine learning models against them
machine-learning classification python python3 python2 text text-mining adversarial-examples spam spam-filtering spam-detection spam-classification text-classification text-analysis data-science data-mining text-processing black-box-benchmarking black-box-attacks metrics
Language:Python 391
PyKoSpacing
haven-jeon / PyKoSpacing
Automatic Korean word spacing with Python
korean-nlp nlp spacing text-processing
Language:Python 370
wenet-e2e / WeTextProcessing
Text Normalization & Inverse Text Normalization
normalization production-ready text-processing
Language:Python 365
BurntSushi / regex-automata
A low level regular expression library that uses deterministic finite automata.
regex dfa automata text-processing regexp regex-engine rust nfa automaton
Language:Rust 354
textpipe / textpipe
Textpipe: clean and extract metadata from text
nlp named-entities named-entity-recognition text-processing text-analysis language-identification
Language:Python 299
gagolews / stringi
Fast and portable character string processing in R (with the Unicode ICU)
stringi icu icu4c r regex regexp string-manipulation unicode natural-language-processing text-processing text stringr nlp tidy-data
Language:C++ 292
ikegami-yukino / jaconv
Pure-Python Japanese character interconverter for Hiragana, Katakana, Hankaku, and Zenkaku
japanese-language preprocessing text-processing japanese-kana pure-python character-converter julius transliteration
Language:Python 289
RandyPen / TextCluster
短文本聚类预处理模块 Short text cluster
nlp text-processing text-mining cluster text-cluster text-clustering
Language:Python 262
rust-unic
open-i18n / rust-unic
UNIC: Unicode and Internationalization Crates for Rust
unicode internationalization text-processing crates rust cldr locale-data unic unicode-characters unicode-algorithms
Language:Rust 234
himkt / konoha
🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.
nlp text-processing mecab kytea sudachi sentencepiece natural-language-processing japanese janome
Language:Python 209
larrykollar / Unix-Text-Processing
Recreated sources for the book "UNIX Text Processing," published in 1987.
formatting gnu-troff groff publishing text-processing unix utp utp-revival
Language:Roff 202
catatsuy / purl
Streamlining Text Processing
grep-like regexp sed text-processing
Language:Go 199
aappleby / matcheroni
A minimalist single-header library for building pattern-matchers, lexers, and parsers.
lexer lexing parser parsing parsing-expression-grammar parsing-expression-grammars regex text-processing c cplusplus-20 pattern-matching regular-expression regular-expression-engine regular-expressions
Language:C++ 192
bytesparadise / libasciidoc
A Golang library for processing Asciidoc files.
golang library asciidoc text-processing
Language:Go 192
textvec / textvec
Text vectorization tool to outperform TFIDF for classification tasks
python nlp machine-learning text-analysis text-classification text-processing tf-idf natural-language-processing
Language:Python 191