I used datasets from amzon product review, I've choosed Arts, Crafts and Sewing (meta & core5 ) becuase the size is small
-
This Dataset is an updated version of the Amazon review dataset released in 2014. As in the previous version,
-
this dataset includes reviews (ratings, text, helpfulness votes), product metadata (descriptions, category information, price, brand, and image features),
-
I used datasets from amazon product review. I've chosen Arts, Crafts and Sewing (rating only, meta & core5) because the size is relatively small.
-
The datasets was zipped format and need to parse and clean.
- rating only dataset :9447882 records, 4 columns
- Meta: 302988 records, 18 columns.
the data sets after parsing and extractedfrom zipped format here!
- delete products with more than one product_id(asin)
with meta_titles(title,counts)
as
(
SELECT title,count(DISTINCT asin) counts from meta
GROUP by title
HAVING counts > 1
)
DELETE from meta where meta.title in (SELECT title from meta_titles)
- delete users with morethan one id
with core_names (reviewerName,counts)
as
(
SELECT reviewerName,count(DISTINCT reviewerID) counts from core5
GROUP by reviewerName
HAVING counts != 1
)
DELETE from core5 where reviewerName in (SELECT reviewerName from core_names)
- delete users with more than on name
with core_ids(reviewerID,counts)
as
(
SELECT reviewerID ,count(DISTINCT reviewerName) counts from core5
GROUP by reviewerID
HAVING counts != 1
)
DELETE from core5 WHERE reviewerID in (SELECT reviewerID from core_ids)
- create new table (final dataset) from joining two tables after cleaning
CREATE TABLE arts_crafts
as
SELECT ratings.asin,meta.title, core5.overall as rating,
meta.brand, meta.main_cat,meta.price,
meta.image, core5.reviewer ID as userId, core5.reviewer Name as username
from ratings
join meta on meta.asin = ratings.asin
join core5 on core5.asin = ratings.asin and core5.reviewerID = ratings.userid
- arts_craftss
- result after building th model arts_crafts_result
this plot shows how the density of the dataset , as we see most products have only few rating this means sparse issue and so that I used SVD algo