tokeniser

There are 0 repository under tokeniser topic.

• dragonofmercy / Tokenize2

  Tokenize2 is a plugin which allows your users to select multiple items from a predefined list or ajax, using autocompletion as they type to find each item. You may have seen a similar type of text entry when filling in the recipients field sending messages on facebook or tags on tumblr.

  autocompletionajaxtokenizetokenisermultiselect-elementsjquery-plugin
  Language:JavaScript 83

• LanguageMachines / ucto

  Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --

  natural-language-processinglanguagenlpcomputational-linguisticstokeniserpunctuationfolia
  Language:C++ 65

• jonsafari / tok-tok

  A fast, simple, multilingual tokenizer

  multilingualnlptokenisertokenizer
  Language:Python 28
• taibun

  andreihar / taibun

  Taiwanese Hokkien Transliterator and Tokeniser

  pojromanisationromanizationtaigitaiwanesetlzhuyintokenisationtokenisertokenizationtokenizertransliterationtransliteratorhokkiennatural-language-processingnlpnlp-librarypython
  Language:Python 22

• ben-sb / jisu

  JavaScript Parser

  javascriptparsertokeniser
  Language:TypeScript 9

• kuhumcst / rtfreader

  Text segmenter and tokeniser for Danish, English and other languages. Reads an RTF or flat text file and outputs the text, one line per sentence & optionally tokenized.

  segmenterrtfrtf-to-texttokenisertokenizersegmentationc-plus-plus
  Language:C++ 6

• phughesmcr / happynodetokenizer

  Javascript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz

  tokenizertokenizingtokenisertokenisingtwittertext-mininghappyfuntokenizerhappierfuntokenizing
  Language:TypeScript 5

• ztjhz / word-piece-tokenizer

  A Lightweight Word Piece Tokenizer

  bertnatural-language-processingnlptokenisertokenizerword-piece
  Language:Python 5
• taibun.js

  andreihar / taibun.js

  Taiwanese Hokkien Transliterator and Tokeniser

  hokkienjavascriptjsnatural-language-processingnlpnlp-librarypojromanisationromanizationtaigitaiwanesetltokenisationtokenisertokenizationtokenizertransliterationtransliteratorzhuyin
  Language:JavaScript 1