This repository contains homework assignments for an NLP course at [Your University's Name]. The assignments focus on tokenization and text processing using various Python libraries. In this README, we'll briefly explain the contents and provide an overview of tokenization, as well as the differences between lemmatization and stemming.
This repository contains the following NLP assignments, each implemented using different Python libraries:
- Assignment 1: Tokenization with Space Library
- Assignment 2: Tokenization with NLTK
- Assignment 3: [Add More Assignments If Applicable]
Each assignment includes the Python code, input data, and an explanation of the tokenization process.
Tokenization is the process of breaking text into individual words or tokens. It's a fundamental step in natural language processing. In each assignment, you'll implement tokenization using different libraries to gain hands-on experience with the variations in tokenization output.
Lemmatization is a text normalization technique that reduces words to their base or dictionary form. It considers the word's context and part of speech, providing more accurate results compared to stemming. For example, "running" would be lemmatized to "run."
Stemming, on the other hand, is a simpler text normalization technique that reduces words to their root form by removing prefixes or suffixes. It doesn't consider the word's context and may produce non-words or less accurate results. For example, "running" would be stemmed to "run."
In your assignments, you may explore both lemmatization and stemming to understand their differences and use cases in NLP.
To run the assignments, follow these steps:
- Clone this repository to your local machine.
git clone git@github.com:MohamadrezaKhalvati/Tokenization.git
cd your-nlp-assignments