Turkish Abstractive Text Summarization
Abstract
Text summarization can be defined as “is the task of producing a concise and fluent summary while preserving key information content and overall meaning”. There are bunch of studies about this task. Most of them are in English. We aimed to create a Turkish abstractive text summarization pipeline from scratch (Crawler to deployment).
Work Plan
Equal Contribution
| Action | Oguzhan Sahin | Nida Kapmaz |
|---|---|---|
| Crawler | x | x |
| Scraping | x | |
| Data Preprocessing | x | x |
| Labelling | x | |
| Encoder-Decoder Model | x | |
| Flask | x | x |
| Deployment | x |
Pipeline
- Crawler
- Web crawling is a component of web scraping, the crawler logic finds URLs to be processed by the scraper code.
- Used Scrapy library for this task.
- Built Scrapy crawler for Webtekno.com and collected about 18k news links.
- Scraping
- Scraped news text by using links that is obtained from crawler.
- Reques and bs4 libraries used for this task.
- Labelling
- Since text summarization task is supervised, the news needed to be labelled (summarized).
- Extracted summary for every single article by using TF-IDF method.
- Model
- Fine-tuned BERT model for this task for 3 epoch.
- Flask
- Created HTML, CSS files for this task.
- Created UI by implementing our model to these html files.
- Deployment
- As a future work, Heroku or Streamlit will be used.
How to run?
- If you do not have data, you can run crawler first. In scrapy_crawlers/spiders/, run below script:
scrapy crawl webtekno --logfile webktekno.log -o webtekno.json -t jsonlines
-
Once you run this script, you will have 2 files (webtekno.log, webtekno.json). In webtekno.json, you will have urls. If you change the urls, you need to adjust webtekno.py.
-
In scrapy_crawler/spiders/, there is parse_json.py for parsing json files and gives an .csv files as an output.
-
To get new text, scraping.py takes an input urls csv, and return urls and text csv file.
-
tf_idf.py here will be used for labelling news text.
-
Once you prepare your data for fine-tuning, you can run fine-tune.ipynb noteboook.
Results
This repository is created by Oguzhan Sahin and Nida Kapmaz