Giters

yuchenlin / LLM-Blender

[ACL2023] We introduce LLM-Blender, an innovative ensembling framework to attain consistently superior performance by leveraging the diverse strengths of multiple open-source LLMs. LLM-Blender cut the weaknesses through ranking and integrate the strengths through fusing generation to enhance the capability of LLMs.

Home Page:https://yuchenlin.xyz/LLM-Blender/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support for MPS device

fangyuan-ksgk opened this issue · comments

commented

Can you please indicate how I can use your wonderful package in a Mac machine with M2/3 chip?

Currently I tried:
blender.loadranker("llm-blender/PairRM", device='mps')
Which works fine, but
ranks = blender.rank(inputs, candidates_texts, return_scores=False, batch_size=1)
Gives me error as it expects CUDA...

Thanks in advance!

commented

This issues has been fixed in the latest github code. Please try reinstall the llm-blender package through

pip install git+https://github.com/yuchenlin/LLM-Blender.git

Then try the above code again