tokenize

There are 2 repositories under tokenize topic.

micromark / micromark
small, safe, and great commonmark (optionally gfm) compliant markdown parser
markdown unified ast cst commonmark parse compile tokenize render gfm
Language:JavaScript 1655
wink-nlp
winkjs / wink-nlp
Developer friendly Natural Language Processing ✨
natural-language-processing nlp tokenize sbd sentence-boundary-detection negation-handling sentiment-analysis pos-tagging ner named-entity-extraction custom-entity-detection word-vectors visualization pattern-matching wink wink-nlp chatbot hacktoberfest bm25 vectorizer
Language:JavaScript 1143
wooorm / markdown-rs
CommonMark compliant markdown parser in Rust with ASTs and extensions
commonmark compiler gfm markdown parse render rust tokenize
Language:Rust 748
alasdairforsythe / tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder text-tokenization vocabulary-generator
Language:Go 493
here-be / snapdragon
snapdragon is an extremely pluggable, powerful and easy-to-use parser-renderer factory.
parser compiler ast parse compile tokenize render lexer token source-map lex node nodejs javascript
Language:JavaScript 218
syntax-tree / mdast-util-from-markdown
mdast utility to parse markdown
unist mdast mdast-util parse tokenize markdown
Language:JavaScript 180
winkjs / wink-nlp-utils
NLP Functions for amplifying negations, managing elisions, creating ngrams, stems, phonetic codes to tokens and more.
tokenize stem ngrams bag-of-words phonetize stop-words sentence-boundary-detection nlp natural-language-processing
Language:JavaScript 111
sina-al / pynlp
A pythonic wrapper for Stanford CoreNLP.
nlp natural-language-processing core-nlp corenlp stanford stanford-corenlp wrapper sentiment-analysis coreference-resolution parser dependency-parser lemmatizer tokenize ssplit part-of-speech-tagger named-entity-recognition
Language:Python 107
dragonofmercy / Tokenize2
Tokenize2 is a plugin which allows your users to select multiple items from a predefined list or ajax, using autocompletion as they type to find each item. You may have seen a similar type of text entry when filling in the recipients field sending messages on facebook or tags on tumblr.
ajax autocompletion jquery-plugin multiselect-elements tokeniser tokenize
Language:JavaScript 83
privateai / deid-examples
Examples scripts that showcase how to use Private AI Text to de-identify, redact, hash, tokenize, mask and synthesize PII in text.
redaction redact de-identification de-identify tokenize masking synthetic-data synthetic-dataset-generation synthetic-data-generator deidentification hipaa anonymization data-anonymization dlp data-loss-prevention text-anonymization cpra gdpr data-masking privacy-tools
Language:Jupyter Notebook 65
jonschlinkert / extract-comments
Extract JavaScript code comments from a string or glob of files.
extract comments parse tokenize javascript esprima code-comments jonschlinkert nodejs
Language:JavaScript 46
chayanforyou / bkash-pgwclient-demo-flutter
bKash payment gateway integration in flutter
bkash bkash-payment bkash-payment-gateway bkash-pgw flutter tokenize
Language:Dart 39
begin / parsers-compilers
Lexers, tokenizers, parsers, compilers, renderers, stringifiers... What's the difference, and how do they work?
parse lexer ast syntax-tree parsers-compilers compiler token tokenize node token-stream guide
22
hexydec / htmldoc
A token based HTML Document parser and minifier written in PHP. Extract attribute values and text using CSS selectors.
html html-dom-parser html-parser html5 minification minify minify-html php simplehtmldom svg tokenize tokenizer
Language:PHP 21
YongWookHa / kor-text-preprocess
Korean text data preprocess toolkit for NLP
korean preprocessing tokenize text-cleaning mecab nlp tool text-processing
Language:Python 16
jonschlinkert / babel-extract-comments
Uses babel to extract JavaScript code comments from a string. Returns an array of comment objects, with line, column, index, comment type and comment string.
babel extract comments parse tokenize jonschlinkert nodejs javascript
Language:JavaScript 15
jonschlinkert / tokenize-comment
Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.
code code-comment comment comment-parser gfm javadoc javascript jonschlinkert js jsdoc node nodejs parse parse-comments tokenize
Language:JavaScript 15
fooSynaptic / transfromer_NN_Block
Implemented transformer NN block for Machine translation, text classfication, Natural language inference as well as Machine reading comprehension model.
jieba natural-language-inference nmt-model sentence-entailment text-classification tokenize transformer
Language:Python 11
akb89 / witokit
A Python toolkit to generate a tokenized dump of Wikipedia for NLP
dump multilingual nlp tokenize wikipedia wikipedia-dump
Language:Python 9
asmeurer / brown-water-python
More detailed documentation for the Python tokenize module
python tokenize python3 documentation
9
myint / untokenize
Transforms tokens into original source code (while preserving whitespace)
python tokenize untokennize
Language:Python 9
TI-Toolkit / tivars_lib_py
A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files
texas-instruments-calculators file-format python ez80 z80 calculators pil texas-instruments tokenize tokenization
Language:Python 9
here-be / snapdragon-scanner
Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.
scan scanner tokenizer tokenize token string parse lex lexer snapdragon
Language:JavaScript 7
carlosplanchon / tokenizesentences
Python3 module to tokenize english sentences.
tokenize sentences python3 python opensource carlosplanchon
Language:Python 6
rtflynn / NLP-Sentiment
Sentiment analysis for amazon product reviews using NLTK, Scikit-Learn, and Keras. Using hyperparameter search and LSTM, our best model achieves ~96% accuracy.
bag-of-words binary-classification hyperparameter-search keras lstm machine-learning natural-language-processing nltk scikit-learn sentiment-analysis sequence-models tokenize
Language:Python 6
paceaux / methodius
A utility for analyzing text on the web
bigram ngram parse split tokenize trigram
Language:TypeScript 5
queckezz / json-tokenize
Splits a JSON string into an annotated list of tokens
json lexer lexical-analysis tokenize tokenizer tokens
Language:JavaScript 5
here-be / snapdragon-stack
Snapdragon utility for creating a stack.
snapdragon stack array tokens nodes parser parse tokenize
Language:JavaScript 4
schemed-js / core
Modular TypeScript template engine
typescript javascript abstract modular string template engine load transpile token tokenize
Language:TypeScript 4
linguistic-dev / n-gram-extractor
A PHP Library to extract n-grams from a text. Simple preprocessing tools (cleaning, tokenizing) included.
php php-library php7 nlp tokenizer ngram ngrams ngram-analysis tokenization tokenize tokenized-sentences natural-language-processing
Language:PHP 3
BaseMax / go-lexer-token-simple
Simple Go lexer: Lex own syntax and read it's from file.
go golang lexer lexer-analyzer lexers lexer-example token tokenizer tokens tokenization tokenize scanner
Language:Go 2
blinky-z / REPL
A Read–Eval–Print Loop (REPL) and Better Bash Lang & Compiler written in C++
repl tokenize tokenizer lexing lexer parse parser eval evaluator ast compiler
Language:C++ 2
cygig / SerialConfigCommand
SerialConfigCommand allows user to issue commands, with or without values via the Serial Monitor easily. Example: "LED=255", "Lock=1", "Start". Compatible with Arduino String() class and character array.
arduino arduino-library command control parse serial setting tokenise tokenize
Language:C++ 2
hamedzarei / nlp-simple-punctuation-correction
simple regex for correcting punctuations
python regex nlp punctuation normalization tokenizer tokenize sentence-parser
Language:Python 2
here-be / snapdragon-location
Adds a location object to snapdragon token or AST node.
location position node token ast tokenize parse parser tokenizer snapdragon nodejs javascript
Language:JavaScript 2
stdlib-js / string-base-format-tokenize
Tokenize a string into an array of string parts and format identifier objects.
nodejs javascript stdlib node node-js utilities utility utils util string str format formatting fmt tokens tokenize token
Language:Makefile 2

tokenize

micromark / micromark

winkjs / wink-nlp

wooorm / markdown-rs

alasdairforsythe / tokenmonster

here-be / snapdragon

syntax-tree / mdast-util-from-markdown

winkjs / wink-nlp-utils

sina-al / pynlp

dragonofmercy / Tokenize2

privateai / deid-examples

jonschlinkert / extract-comments

chayanforyou / bkash-pgwclient-demo-flutter

begin / parsers-compilers

hexydec / htmldoc

YongWookHa / kor-text-preprocess

jonschlinkert / babel-extract-comments

jonschlinkert / tokenize-comment

fooSynaptic / transfromer_NN_Block

akb89 / witokit

asmeurer / brown-water-python

myint / untokenize

TI-Toolkit / tivars_lib_py

here-be / snapdragon-scanner

carlosplanchon / tokenizesentences

rtflynn / NLP-Sentiment

paceaux / methodius

queckezz / json-tokenize

here-be / snapdragon-stack

schemed-js / core

linguistic-dev / n-gram-extractor

BaseMax / go-lexer-token-simple

blinky-z / REPL

cygig / SerialConfigCommand

hamedzarei / nlp-simple-punctuation-correction

here-be / snapdragon-location

stdlib-js / string-base-format-tokenize