- Abstractive summarization is a technique for creating a summary of a text based on its primary ideas rather than by directly reproducing its most important sentences. This work in natural language processing is crucial and difficult. For this project, the "Text to Text Transfer Transformer Model" popularly known as T5 transformers was used to train our custom dataset to enable it give us the abstractive summary of the text data.
- T5 transformers was used because it is flexible enough to be fine-tuned for quite a number of important tasks especially Abstractive Summarization. T5 has also achieved state-of-the-art result in this field.
- The framework used for this project is pytorch lightning and it was used because of its speed, efficiency, and reproducibility properties.
This project aims to sumamrize long text of 512 tokens or lesser, to tokens <= 128 without reproducing the words in the main text and also retaining context. The project was deployed using hugging face spaces with streamlit and this repo also contains a flask app which can be setup locally.
Project files/folders:
This project was started by getting a dataset that could work with the T5 transformer model since the model takes in texts and returns text(one of the main reason it is flexible). Data preprocessing and exploratory data analysis was carried out on the dataset before going ahead to use the tokenizer on the model. ##Add graph photo here##
Pytorch-lightning is the framework used in this project and that's majorly because it helps organize our pytorch code. Some of the functions created using pytorch lightning includes:
The section aims to give a walkthrough of the model training and the deployment aspect of the project.
- Setup and library imports: This is the first phase of any Machine learning data and it is at this point we switch to GPU(if necessary), install and import the libraries we need to get the project started(e.g numpy, torch, pytorch lightning). Some of these libraries can be imported/installed later in the project but I prefer to install mine at the beginning of my ipynb notebook.
- Getting the dataset from kaggle to google colab: Instead of manualing downloading the dataset this was done by using the kaggle library and the API command of the dataset on kaggle.
- Loading the dataset into a dataframe.
- Examining the dataframe: The dataframe consists of 6 columns, 4 of which are unnecessary for this task. A new dataframe containing just the text and summary columns was created.
- Data cleaning: The column names were changed to relevant names, rows containing null data was dropped as these rows are just few and filling them would be hard.
- Splitting the dataset into test and train data: The new dataframe created was splitted into train and test column with test_size of 0.1.
- NewsSummaryDataset: This class was created to encode, tokenize, pad and truncate the dataset.This is done with the t5 tokenizer from the T5 model installed.This class also specify the maximum token length for both the text column and the summary column.
- NewsSummaryDataModule: This class applies the function above to our dataset and takes into cognizance the train test split done above.
- NewsSummaryModel: This class contains functions that trains and tests the model.
- A checkpoint was created for the model and a trainer which was created from our pytorch lightning module was used to fit the data and the model.
- A function summarize was created which was used to test the model built. This function takes in the text to be summarized and returns the summary of length <= 128.
The T5-base model is a very large one and can't be deployed on free cloud platforms because of the size.
I found it hard building my flask APP with vscode, I am not sure why but I guess dependency was a major issue. Setting it on Pycharm was less challenging and I had to install RUST, CYTHON and one or two other dependencies in the git bash terminal.
References