pointing to issue I mentioned in mabwiser

Question

pointing to issue I mentioned in mabwiser

ayush488 opened this issue 2 years ago · comments

Please review this one.. By mistake I created it in mabwiser.. But it is related to mab2rec.
fidelity/mabwiser#59

Bernard Kleynhans · Answer 1 · Wed Jul 06 2022 00:06:56 GMT+0800 (China Standard Time)

Could you please try setting click_column="score" when defining the evaluation metrics as per below?

# Column names for the response, user, and item id columns
metric_params = {'click_column': 'score', 'user_id_column': 'ID', 'item_id_column':'MailerID'}

ayush488 · Answer 2 · Wed Jul 06 2022 01:48:26 GMT+0800 (China Standard Time)

metric_params = {'click_column': 'score', 'user_id_column': 'ID', 'item_id_column':'MailerID'}

Evaluate peformance at different k-recommendations

top_k_list = [4]

List of metrics to benchmark

metrics = []
for k in top_k_list:
metrics.append(BinaryRecoMetrics.AUC(**metric_params, k=k))
metrics.append(BinaryRecoMetrics.CTR(**metric_params, k=k))
metrics.append(RankingRecoMetrics.Precision(**metric_params, k=k))
metrics.append(RankingRecoMetrics.Recall(**metric_params, k=k))
metrics.append(RankingRecoMetrics.NDCG(**metric_params, k=k))
metrics.append(RankingRecoMetrics.MAP(**metric_params, k=k))

from mab2rec.pipeline import benchmark

Benchmark the set of recommenders for the list of metrics

using training data and user features scored on test data

reco_to_results, reco_to_metrics = benchmark(recommenders,
metrics=metrics,
train_data=X_train,
test_data=X_test,
user_features=df_users_X,
user_id_col= 'ID',
item_id_col= 'MailerID',
response_col = 'sales_net')
Traceback (most recent call last):

File "C:\Users\ayush\AppData\Local\Temp\2\ipykernel_5524\2509256242.py", line 27, in
response_col = 'sales_net')

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 417, in benchmark
return _bench(**args)

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 531, in _bench
recommendations[name])

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\combined.py", line 121, in get_score
return_extended_results)

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\auc.py", line 140, in get_score
return self._accumulate_and_return(results, batch_accumulate, return_extended_results)

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\base.py", line 121, in _accumulate_and_return
cur_result = self._get_results([results])

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\auc.py", line 146, in _get_results
return roc_auc_score(results[:, 0], results[:, 1])

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\sklearn\metrics_ranking.py", line 560, in roc_auc_score
raise ValueError("multi_class must be in ('ovo', 'ovr')")

ValueError: multi_class must be in ('ovo', 'ovr')

ayush488 · Answer 3 · Wed Jul 06 2022 01:49:00 GMT+0800 (China Standard Time)

getting this error now?

ayush488 · Answer 4 · Wed Jul 06 2022 01:50:49 GMT+0800 (China Standard Time)

seems like the evaluation can only be done with click 0 or 1?

ayush488 · Answer 5 · Wed Jul 06 2022 02:46:55 GMT+0800 (China Standard Time)

Could you please try setting click_column="score" when defining the evaluation metrics as per below?
# Column names for the response, user, and item id columns
metric_params = {'click_column': 'score', 'user_id_column': 'ID', 'item_id_column':'MailerID'}

now it is giving different error. Could you please help?

Bernard Kleynhans · Answer 6 · Wed Jul 06 2022 03:24:18 GMT+0800 (China Standard Time)

Yes, that's correct. All the recommendation metrics above are only well defined for binary (0 or 1) responses.

ayush488 · Answer 7 · Wed Jul 06 2022 05:23:42 GMT+0800 (China Standard Time)

so there is no way to evaluate the non binary responses?

Bernard Kleynhans · Answer 8 · Wed Jul 06 2022 05:41:55 GMT+0800 (China Standard Time)

Unfortunately, we don't have non-binary metrics implemented in the Jurity library. Common metrics include Mean Absolute Error (MAE) and Mean Squared Error (MSE) and are straightforward to compute.

An alternative strategy that is frequently used would be to binarize your response by setting the response to 1 if the continuous value is above some threshold and 0 otherwise.

ayush488 · Answer 9 · Wed Jul 06 2022 20:51:54 GMT+0800 (China Standard Time)

But the recommendation system as such works for non binary responses. Is that correct? Only evaluation is more geared towards binary responses.??

…

On Jul 6, 2022 at 6:29 AM, <Bernard Kleynhans ***@***.***)> wrote: Closed #14 (#14) as completed. — Reply to this email directly, view it on GitHub (#14 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/ABDPIETAPAPSCVINWJNOGD3VSVU3BANCNFSM52WXHAWQ). You are receiving this because you authored the thread.Message ID: ***@***.***>

Bernard Kleynhans · Answer 10 · Wed Jul 06 2022 21:24:26 GMT+0800 (China Standard Time)

Yes, that's correct.

ayush488 · Answer 11 · Wed Jul 06 2022 22:59:50 GMT+0800 (China Standard Time)

 File "C:\Users\ayush\AppData\Local\Temp\2\ipykernel_5524\2998262892.py", line 13, in <module>
    response_col = 'response')

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 417, in benchmark
    return _bench(**args)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 526, in _bench
    save_file=None)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 286, in score
    recs_of_batch, scores_of_batch = recommender.recommend(contexts, excluded_arms_batch, return_scores=True)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\rec.py", line 323, in recommend
    expectations = self.mab.predict_expectations(contexts)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\mab.py", line 1229, in predict_expectations
    return self._imp.predict_expectations(contexts)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\linear.py", line 151, in predict_expectations
    return self._parallel_predict(contexts, is_predict=False)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\base_mab.py", line 227, in _parallel_predict
    for i in range(n_jobs))

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 1043, in __call__
    if self.dispatch_one_batch(iterator):

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 861, in dispatch_one_batch
    self._dispatch(tasks)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 779, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\_parallel_backends.py", line 208, in apply_async
    result = ImmediateResult(func)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\_parallel_backends.py", line 572, in __init__
    self.results = batch()

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\linear.py", line 212, in _predict_contexts
    arm_to_expectation[arm] = arm_to_model[arm].predict(row)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\linear.py", line 93, in predict
    beta_sampled = self.rng.multivariate_normal(self.beta, np.square(self.alpha) * self.A_inv)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\utils.py", line 258, in multivariate_normal
    return np.squeeze(self.rng.multivariate_normal(mean, covariance, size=size, method='cholesky'))

  File "_generator.pyx", line 3625, in numpy.random._generator.Generator.multivariate_normal

  File "<__array_function__ internals>", line 6, in cholesky

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\numpy\linalg\linalg.py", line 763, in cholesky
    r = gufunc(a, signature=signature, extobj=extobj)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\numpy\linalg\linalg.py", line 91, in _raise_linalgerror_nonposdef
    raise LinAlgError("Matrix is not positive definite")

LinAlgError: Matrix is not positive definite

Still getting error after binarizing....