text-tokenization

There are 0 repository under text-tokenization topic.

alasdairforsythe / tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder text-tokenization vocabulary-generator
Language:Go 604
twardoch / split-markdown4gpt
A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.
data-preprocessing gpt gpt-3 gpt-35-turbo gpt-35-turbo-16k gpt-4 markdown markdown-processing mistletoe natural-language-processing nlp openai openai-gpt python split-text summarization text-analysis text-processing text-summarization text-tokenization
Language:Python 28
SayamAlt / Resume-Classification-using-fine-tuned-BERT
Successfully developed a resume classification model which can accurately classify the resume of any person into its corresponding job with a tremendously high accuracy of more than 99%.
bert-model exploratory-data-analysis fine-tuning-bert model-evaluation nlp text-preprocessing text-tokenization word-embeddings
Language:Jupyter Notebook 10
Mecanik / Modern-Text-Tokenizer
Modern UTF-8 aware C++ tokenizer with vocabulary support, ideal for NLP and transformer models. Header-only and zero-dependency.
ai artificial-intelligence bert deep-learning distilbert header-only high-performance machine-learning modern-cpp natural-language-processing nlp preprocessing text-analysis text-encoding text-processing text-tokenization tokenizer transformer vocabulary
Language:C++ 4
hikmatazimzade / azerbaijani-tokenizer
High-Performance Azerbaijani Tokenizers (30% fewer tokens, 40% faster than multilingual alternatives)
ai azerbaijani bpe huggingface nlp text-tokenization tokenization tokenizer transformers unigram azerbaijani-language azerbaijani-nlp
Language:Jupyter Notebook 3
SayamAlt / Financial-News-Sentiment-Analysis
Successfully developed a fine-tuned DistilBERT transformer model which can accurately predict the overall sentiment of a piece of financial news up to an accuracy of nearly 81.5%.
data-exploration-and-preprocessing distilbert-model fine-tune-bert-tensorflow hugging-face-transformers model-architecture-and-implementation model-inference model-training-and-evaluation multiclass-classification natural-language-processing sentiment-analysis text-preprocessing text-tokenization
Language:Jupyter Notebook 2
SayamAlt / Symptoms-Disease-Text-Classification
Successfully developed a fine-tuned BERT transformer model which can accurately classify symptoms to their corresponding diseases upto an accuracy of 89%.
bert-fine-tuning data-exploration-and-preprocessing exploratory-data-analysis fine-tune-bert-tensorflow hugging-face-transformers model-architecture-and-implementation model-inference model-training-and-evaluation multiclass-classification natural-language-processing text-classification text-preprocessing text-tokenization
Language:Jupyter Notebook 2
adilrasheed139 / AI-Powered-Resume-Screening-using-BERT
Successfully developed a resume classification model which can accurately classify the resume of any person into its corresponding job with a tremendously high accuracy of more than 99%.
bert-model deep-learning exploratory-data-analysis-eda fine-tuning-bert model-evaluation nlp nlp-machine-learning text-preprocessing text-tokenization word-embeddings word-embeddings-for-nlp
Language:Jupyter Notebook 1
cedrickchee / tokenizers
💥Fast State-of-the-Art Tokenizers optimized for Research and Production
nlp natural-language-processing natural-language-understanding language-model transformers bert gpt text-tokenization
Language:Rust 1
JustVugg / Biptoken
Fast BPE tokenizer with guaranteed perfect text reconstruction. 2x faster than tiktoken at encoding/decoding while preserving all whitespace and formatting.
bpe-tokenizer nlp open-source python python-library text-tokenization tokenizer fast-tokenizer llm-tooling perfect-reconstruction whitespace-preservation
Language:Python 1
katanabana / Nihotip
Nihotip is a web app that lets users explore Japanese text through interactive tokenization and detailed insights. Built with React and Python, it offers a dynamic way to analyze words and symbols with tooltips for deeper understanding.
japanese japanese-characters japanese-language japanese-learning jmdictfurigana language mecab nlp python react sudachipy text-analysis text-tokenization tokenization tooltips wanakana webapp
Language:JavaScript 1
markiskorova / Machine-Learning-NLP-Predict-Author
🧠 Machine Learning & Natural Language Processing: Predict the author of literary text snippets. Built with TensorFlow and Keras, this project trains an LSTM model on classic literature to identify writing style and authorship.
keras machine-learning natural-language-processing python tensorflow text-tokenization text-vectorization
Language:Python 1
muthu-kumar-u / Speech-Recognition-RNN
Deep learning-based subtitle generation model that processes audio datasets to generate accurate text transcriptions. Includes audio feature extraction, encoder-decoder architecture, training pipelines, and evaluation metrics for subtitle alignment.
audio-processing deep-learning encoder-decoder-architecture natural-language-processing rnn speech-recognition speech-to-text subtitle-generation text-tokenization transformer-models
1
SayamAlt / Cyberbullying-Classification-using-fine-tuned-DistilBERT
Successfully fine-tuned a pretrained DistilBERT transformer model that can classify social media text data into one of 4 cyberbullying labels i.e. ethnicity/race, gender/sexual, religion and not cyberbullying with a remarkable accuracy of 99%.
cyberbullying-detection data-exploration distilbert-model exploratory-data-analysis fine-tune-bert-tensorflow llm model-inference model-training-and-evaluation multiclass-classification natural-language-processing text-classification text-preprocessing text-tokenization
Language:Jupyter Notebook 1
SayamAlt / Fake-News-Classification-using-fine-tuned-BERT
Successfully developed a text classification model to predict whether a given news text is fake or not by fine-tuning a pretrained BERT transformed model imported from Hugging Face.
bert-embeddings bert-model data-analysis data-visualization deep-learning fine-tuning-bert model-evaluation model-training-and-evaluation text-classification text-preprocessing text-tokenization tokenizer-nlp wordcloud-visualization
Language:Jupyter Notebook 1
SayamAlt / News-Category-Classification
Successfully developed a news category classification model using fine-tuned BERT which can accurately classify any news text into its respective category i.e. Politics, Business, Technology and Entertainment.
bert-embeddings exploratory-data-analysis feature-engineering fine-tuning-bert model-evaluation nlp text-classification text-cleaning text-preprocessing text-tokenization
Language:Jupyter Notebook 1
Software-Research-Lab / dropsuit-tok
The tok function is a JavaScript and Node.js function that processes object instances and tokenizes text arrays. It returns tokenized words number, tokenized words array, and tokenized words concatenated string. It's part of the open-source DropSuit NLP library under the Apache License 2.0.
language-understanding text-analysis text-processing text-tokenization
Language:JavaScript 1
victoryosiobe / kingchop
Kingchop ⚔️ is a JavaScript English based library for tokenizing text (chopping text). It uses vast rules for tokenizing, and you can adjust them easily.
nodejs tokenizers javascript text-processing text-tokenization natural-language-processing sentence-tokenizer word-tokenizer paragraph-tokenizer
Language:JavaScript 1
LokeshKenche / ISP_ChatBot
ISPY ChatBot ISPY is a chatbot designed for ISP customer service, providing automated responses and assistance for various queries such as connection issues, payments, and service requests. Built using Python with libraries like nltk and newspaper3k, it simulates conversation and handles customer interactions effectively.
chatbot cosine-similarity customer-services machine-learning newspaper3k nlp-machine-learning nltk text-based-chatbot text-tokenization textanalysis webscraping article-parsing
Language:Jupyter Notebook 0
Aalaa4444 / Text_Processing-and-Unique_Word_Extraction_fromHTML
Extract text content from an HTML page, process it, and extract unique words from the processed text. This notebook utilizes various text processing techniques including cleaning, normalization, tokenization, lemmatization or stemming, and stop words removal.
lemmatization stemming text-processing tokenization tokenizer beautifulsoup data-extraction extract-html requests stopwords-removal text-cleaning text-extraction text-lemmatization text-normalization text-tokenization
Language:Jupyter Notebook
gudashashank / EqualEyes
The project aimed to push image captioning technology forward by combining recent advances in image recognition and language modeling to generate novel, descriptive captions that go beyond just naming objects and actions
bleu-score blip cnn-model cnn-rnn image-augmentation text-tokenization vectorization vision-transformer
SayamAlt / Customer-Support-Chatbot-using-NLTK
Successfully developed a chatbot model which can provide accurate and concise responses to a wide variety of customer queries regarding the services offered by a particular company as well as general topics.
chatbots deep-learning deep-neural-networks nlp nltk text-tokenization
Language:Python
SayamAlt / English-to-Spanish-Language-Translation-using-Seq2Seq-and-Attention
Successfully established a Seq2Seq with attention model which can perform English to Spanish language translation up to an accuracy of almost 97%.
attention-is-all-you-need attention-model bert-transformer exploratory-data-analysis fine-tuning-bert hugging-face-transformers language-translation luong-attention model-architecture-and-implementation model-inference model-training-and-evaluation natural-language-processing neural-machine-translation seq2seq-modeling text-generation text-preprocessing text-tokenization
Language:Jupyter Notebook
SayamAlt / Global-News-Headlines-Text-Summarization
Successfully established a text summarization model using Seq2Seq modeling with Luong Attention, which can give a short and concise summary of the global news headlines.
attention-mechanism data-exploration-and-preprocessing luong-attention model-architecture-and-implementation model-inference natural-language-processing seq2seq-model text-generation text-summarization text-tokenization
Language:Jupyter Notebook
SayamAlt / Luxury-Apparel-Product-Category-Classification-using-fine-tuned-DistilBERT
Successfully developed a multiclass text classification model by fine-tuning pretrained DistilBERT transformer model to classify various distinct types of luxury apparels into their respective categories i.e. pants, accessories, underwear, shoes, etc.
deep-learning distilbert-fine-tuning distilbert-model exploratory-data-analysis fine-tuning-bert model-inference model-training-and-evaluation multiclass-classification natural-language-processing text-classification text-preprocessing text-tokenization
Language:Jupyter Notebook
SayamAlt / Mental-Health-Classification-using-fine-tuned-DistilBERT
Successfully established a multiclass text classification model by fine-tuning pretrained DistilBERT transformer model to classify several distinct types of mental health statuses such as anxiety, stress, personality disorder, etc. with an accuracy of 77%.
data-visualization deep-learning distilbert-fine-tuning distilbert-model model-evaluation model-inference model-training-and-evaluation multiclass-text-classification natural-language-processing text-classification text-preprocessing text-tokenization
Language:Jupyter Notebook
Yeasir-Hossain / keycloak-authentication
A full-stack text analysis application with Keycloak authentication, implementing word tokenization, sentence boundary detection, and longest word extraction algorithms with Redis caching for performance optimization.
authentication caching containerization docker express jwt keycloak longest-word mern-stack mongodb nodejs paragraph-detection react redis sentence-boundary-detection text-analysis text-tokenization typescript word-frequency-analysis
Language:TypeScript

text-tokenization

alasdairforsythe / tokenmonster

twardoch / split-markdown4gpt

SayamAlt / Resume-Classification-using-fine-tuned-BERT

Mecanik / Modern-Text-Tokenizer

hikmatazimzade / azerbaijani-tokenizer

SayamAlt / Financial-News-Sentiment-Analysis

SayamAlt / Symptoms-Disease-Text-Classification

adilrasheed139 / AI-Powered-Resume-Screening-using-BERT

cedrickchee / tokenizers

JustVugg / Biptoken

katanabana / Nihotip

markiskorova / Machine-Learning-NLP-Predict-Author

muthu-kumar-u / Speech-Recognition-RNN

SayamAlt / Cyberbullying-Classification-using-fine-tuned-DistilBERT

SayamAlt / Fake-News-Classification-using-fine-tuned-BERT

SayamAlt / News-Category-Classification

Software-Research-Lab / dropsuit-tok

victoryosiobe / kingchop

LokeshKenche / ISP_ChatBot

Aalaa4444 / Text_Processing-and-Unique_Word_Extraction_fromHTML

gudashashank / EqualEyes

SayamAlt / Customer-Support-Chatbot-using-NLTK

SayamAlt / English-to-Spanish-Language-Translation-using-Seq2Seq-and-Attention

SayamAlt / Global-News-Headlines-Text-Summarization

SayamAlt / Luxury-Apparel-Product-Category-Classification-using-fine-tuned-DistilBERT

SayamAlt / Mental-Health-Classification-using-fine-tuned-DistilBERT

Yeasir-Hossain / keycloak-authentication