Cant import exllama
caseybasichis opened this issue · comments
import exllama
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\me\Documents\exllama.py", line 1, in
from model import ExLlama, ExLlamaCache, ExLlamaConfig
ModuleNotFoundError: No module named 'model'
from exllama.model import ExLlama, ExLlamaCache, ExLlamaConfig
Getting the same error.
Here is the successful install of exllama (maybe?) and the error attempting import.
(base) PS C:\Users\me\Documents> python -m pip install git+https://github.com/jllllll/exllama
Collecting git+https://github.com/jllllll/exllama
Cloning https://github.com/jllllll/exllama to c:\users\me\appdata\local\temp\pip-req-build-nzin51q7
Running command git clone --filter=blob:none --quiet https://github.com/jllllll/exllama 'C:\Users\me\AppData\Local\Temp\pip-req-build-nzin51q7'
Resolved https://github.com/jllllll/exllama to commit e25ee5a
Preparing metadata (setup.py) ... done
Requirement already satisfied: torch in c:\users\me\anaconda3\lib\site-packages (from exllama==0.0.17+cu118) (2.0.1+cu118)
Requirement already satisfied: filelock in c:\users\casRequirement already satisfied: typing-extensions in c:\users\me\anaconda3\lib\site-packages (from torch->exllama==0.0.17+cu118) (4.7.1)
Requirement already satisfied: sympy in c:\users\me\anaconda3\lib\site-packages (from torch->exllama==0.0.17+cu118) (1.11.1)
Requirement already satisfied: networkx in c:\users\caslama==0.0.17+cu118) (3.1)
Requirement already satisfied: jinja2 in c:\users\me\anaconda3\lib\site-packages (from torch->exllama==0.0.17+cu118) (3.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\me\anaconda3\lib\site-packages (from jinja2->torch->exllama==0.0.17+cu118) (2.1.1)
Requirement already satisfied: mpmath>=0.19 in c:\users\me\anaconda3\lib\site-packages (from sympy->torch->exllama==0.0.17+cu118) (1.3.0)
(base) PS C:\Users\me\Documents> python
Python 3.11.4 | packaged by Anaconda, Inc. | (main, Jul 5 2023, 13:38:37) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
from exllama.model import ExLlama, ExLlamaCache, ExLlamaConfig
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\me\Documents\exllama.py", line 1, in
from model import ExLlama, ExLlamaCache, ExLlamaConfig
ModuleNotFoundError: No module named 'model'
Your exllama.py script is using from model import ExLlama, ExLlamaCache, ExLlamaConfig to import the module.
This won't work as the model module is under the exllama namespace.
You need to use from exllama.model import ExLlama, ExLlamaCache, ExLlamaConfig in order to access that module.
Thank you for catching that mistake. Running OK.
The Transfomrs output on the same model seems more faithful to the prompt.
Any ideas why that might be?
EXLLAMA
generator.settings.token_repetition_penalty_max = 1.15
generator.settings.temperature = 0.7
generator.settings.top_p = 0.95
generator.settings.top_k = 100
generator.settings.typical = 0.5
Transformers
temperature=0.7,
top_p=0.95,
repetition_penalty=1.15
Transformers version
Once upon a time,
There was a little girl who lived in the country. She had no brothers or sisters and she was very lonely. One day her mother said to her: "Dear child, if you wish I will give you a little brother." The little girl clapped her hands with joy and cried out: "Oh! yes, dear Mother, please do!" So that night when the little girl went to bed, her mother gave her a tiny plant which looked like this:
[Illustration]
The next morning the little girl put it into a flower pot and placed it on the window sill where the sun would shine on it all day long. Every day she watered it carefully and kept the soil loose so that its roots could grow easily. In a few days there were two leaves; then four, eight, sixteen, thirty-two, sixty-four--and so on until at last there were one thousand leaves. Then the little girl's mother told her to pull up the plant by the roots and cut off the top of it. This she did and found that what remained looked just like this:
[Illustration]
Then her mother told her to go outside the house and dig a hole as deep as her arm was long. When she had done this, she put the plant into the hole and covered it over with earth. After that she took care to water it every day till the rainy season came. At last one day she saw something pushing through the ground. It grew bigger and bigger till it became a fine large tree. Its trunk was smooth and white like marble and its branches spread far away like clouds across the sky. On them hung golden fruits and silver flowers. All kinds of birds made their nests among the branches and sang sweet songs all day long.
Now the little girl loved her tree dearly and spent most of her time near it. But one day while she was sitting under it looking up into the sky, she heard a rustling sound above her head. Down fell a bird right at her feet. Poor thing! It lay quite still. The little girl picked it up gently and held it tenderly in her arms. Presently it opened its eyes and began to chirp. As soon as it felt strong enough to fly away it left her. But each day after that the same thing happened. Every afternoon a different bird flew down from the tree and dropped at her feet. Each time she picked it
---
<s> Once upon a time,
there was a little girl who lived in a village.
She was very pretty, and very good, and very poor.
She had no father or mother, and lived with a wicked woman who was her stepmother.
She had two stepsisters, who were very proud and very beautiful.
They had fine clothes, and petted and made much of by their mother, and they looked down on the little girl.
They would not let her go to the ball, but made her sit in the ashes and keep the fire.
She cried, and cried, and cried, and a fairy godmother came to her.
She touched her with her wand, and gave her a coach and horses, and a beautiful dress.
She went to the ball, and there she met a prince.
He fell in love with her, and she with him.
She had to go away at twelve o'clock, and in her haste she left her shoe.
The prince found it, and fell in love with the shoe.
He made it known that he would marry the lady who could wear the shoe.
The stepsisters tried, but they could not get their feet into the shoe.
The little girl tried, and her foot went in.
The prince took her to be his wife, and they lived happily ever after.
The End.
Exllama versions
Once upon a time, I thought the best way to be successful was to be better than everyone else. But that’s not how it works at all! It is much more effective and sustainable to help others get what they want by sharing your expertise.
You don’t have to give away everything you know for free but there are some things that will benefit both of us if we share them with each other (and sometimes even just one person). The most important thing here is being willing to take risks when necessary so long as those risks lead towards something greater down the road–which brings me back around again…
How To Get Yourself Into A Growth Mindset
A growth mindset is an attitude toward learning and improvement. It’s about seeing challenges as opportunities instead of obstacles, and believing that hard work can make you smarter or stronger. You may have heard this concept before in relation to business success: “The only difference between where you are now and where you want to be tomorrow is the effort put forth today.” This quote from Tony Robbins captures exactly why having a growth mindset matters so much for entrepreneurs—because it helps them see their goals as achievable through consistent action rather than wishful thinking or luck alone.
Having a growth mindset also means embracing failure because it allows us to learn from our mistakes without letting them define who we are or hold us back from future successes; instead, failures become stepping stones on which we build new ideas based off past experiences learned along the way!
Become A Learning Machine
...
---
Once upon a time, I had to deal with an employer who would make fun of me for not having the latest gadgets. She would constantly talk about how her iPhone was better than my Android and how she couldn’t understand why anyone wouldn’t have one. It was frustrating because I knew that I didn’t need the latest technology in order to do my job well.
Now, I don’t work there anymore but it got me thinking: are we really so materialistic as a society? Do people actually care more about what brand or model of phone you have rather than your personality or skillset? Apparently yes! According to research conducted by YouGov America Inc., 46% of Americans believe that owning expensive items is important for success in life – including their cell phones! This means that almost half of us place value on things like iPhones over other aspects such as education and hard work when determining our worthiness for success…scary stuff indeed!
I personally think this kind of attitude needs to change if we want true progress towards equality between all classes within our country (and beyond). We should be focusing on developing ourselves through learning new skills instead of trying desperately keep up appearances with costly possessions which will eventually become obsolete anyway due advances made technologically speaking each year . Afterall , no matter how much money someone has spent acquiring certain items , they cannot buy intelligence nor character traits necessary succeeding long term goals set forth by themselves or others around them either professionally socially etcetera.. So let’s start valuing knowledge & experience gained throughout lifetime more highly than physical objects acquired during same period shall we ??? 🙂
– Anecdote from personal experience regarding former employer making fun/judging based upon type smartphone owned
– Research statistic provided showing majority Americans believe owning expensive items key factor achieving success life
...
Presumably, this is because you are using a 4bit model with exllama and a full model with transformers.
4bit models will always have a worse output to some degree as you are essentially not running the model in it's entirety.
This is simply the tradeoff to having faster inference and lower memory requirements.
There is also the matter of exllama simply using different code than transformers.
As a result, you will pretty much always have a different output between the 2 in some way.