The repository contains dataset of Bull Bot from 5000+ websites and links of the University of South Florida
Checkout Bull Bot in action at BullBot.tech
- The program in main.ipynb file scrapes the links from the websites stored them in the form of a pickle file urls.pkl.
- After that, the program in data_webScrap.ipynb file scrapes the page contents from the links stored in the pickle file and stores them in the form of a json file data.json.
The dataset is a json file data.json. In another words, it is a list of python dictionary with the following structure:
[{
"page_content": {
"page_content of the pages.....",
"metadata": {
"source": "url of the page",
"title": "title of the page"
}
},
{
"page_content": {
"About USF. See .... "
}
"metadata": {
"source": "https://www.usf.edu/about-usf/index.aspx",
"title": "About USF | University of South Florida"
}
},
...
}
]
- Data Analyst only need to look at the data.json file.
Tasks:
- Remove and filter unrelevant data from the dataset. The data which is considered unrelevant when it is not related to the USF and contains unnecessary information.
- Clean the data. The current data contain a lo of new line characters (\n). They should be removed or cut off.
- Develop a python program to clean data, and filter out the unrelevant data. The program should be able to automatically clean the data and filter out the unrelevant data when the new data is added to the dataset.
- Develop a python program to filter out the data that relevant to
- Application process to USF
- Housing information at USF
- Dining at USF
- Student life at USF
After that, add the key to the metadata of those data objects. For example, the data object that contains the information about the application process to USF should have the key and vaue "group": "application" in the metadata. { "page_content": { "Apply. See .... " } "metadata": { "source": "https://admissions.usf.edu/application", "title": "About USF | University of South Florida", "group": "application" } },
{ "page_content": { "These are the dining options .... " } "metadata": { "source": "https://www.usf.edu/campus-life/housing-dining.aspx", "title": "Housing and Dining | University of South Florida", "group": "dining" } },
{ "page_content": { "Tampa Campus Student Life .... " } "metadata": { "source": "https://www.usf.edu/campus-life/", "title": "Student Life at Tampa Campus | University of South Florida", "group": "student-life" } },