viki-challenge
http://www.dextra.sg/rakuten-viki-global-tv-recommender-challenge/
https://challenges.dextra.sg/challenge/43
Some preliminary analysis:
Presentation deck:
https://speakerdeck.com/lenguyenthedat/rakuten-viki-data-challenge-solution
Requirements:
This solution is 100% Python, below are a few libraries needed:
- Pandas
- Scikit-learn
Collaborative Filtering (Jaccard Index) plus feature similarity
$ python viki-videos-similarity.py # Pre-procesing `#videos x #videos` matrix
$ python viki-users-recommender.py # batch process
This is more practical since #videos x #videos matrix is much smaller.
Weights can be set manually:
top_videos_limit = 50
sim_features = ['sim_country', 'sim_language', 'sim_adult',
'sim_content_owner_id', 'sim_broadcast', 'sim_episode_count',
'sim_genres', 'sim_cast',
'jaccard_1_3', 'jaccard_2_3', 'jaccard_3_3',
'jaccard_high', 'sim_cosine_mv_ratio']
weight_features = [3,3,5,
0,0,0,
5,5,
0,0,15,
15,45]
weight_scores = [1,3,15]
Notes:
The reason why HOT VIDEOS dominated CF is because viki's homepage currently dominated by:
- Top banner
- Popular show
- Top Drama
- Gender filter for male / female
TODO:
- utilize ratio instead of score
- see if someone is into hot / fresh video or not
- KNN: cosine similarity for user => top 10 similar user => recommend top videos user havent watched
Tried implementing with cosinesimilarity - Killed 9
Tried Sklearn KNN - took 5h ++
Panns https://github.com/ryanrhymes/panns 2h++
Trying Spotify's Annoy https://github.com/spotify/annoy : 20mins with 10 trees, 12mins with 100 trees.
-> however it's taking too long to find k-NN for each users (more than 10s each to get a good enough result)
Submission history:
(Only those that worth documented or created when I am not too lazy):
https://github.com/lenguyenthedat/dextra-viki-2015/blob/master/submission_history.txt