Can't load awq model

Question

Can't load awq model

nNote1377 opened this issue 2 months ago · comments

Describe the bug

I installed text generation webui and downloaded the model(TheBloke_Yarn-Mistral-7B-128k-AWQ) and I can't run it. I chose Transofmer as Model loader. I tried installing autoawq but the installed version of torch is not compatible with it, so while installation it downloads an older version but it still doesn’t work.

Is there an existing issue for this?

I have searched the existing issues

Reproduction

Select a model and try to load it

Screenshot

Logs

C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\quantizers\auto.py:182: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.However, loading attributes (e.g. ['use_cuda_fp16', 'use_exllama', 'max_input_length', 'exllama_config', 'disable_exllama']) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
  warnings.warn(warning_msg)
20:30:11-368425 ERROR    Failed to load the model.
Traceback (most recent call last):
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 263, in huggingface_loader
    model = LoaderClass.from_pretrained(path_to_model, **params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 3452, in from_pretrained
    hf_quantizer.validate_environment(
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\quantizers\quantizer_awq.py", line 53, in validate_environment
    raise ImportError("Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)")
ImportError: Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)
20:36:52-274133 INFO     Loading "TheBloke_Yarn-Mistral-7B-128k-AWQ"
20:36:52-281135 INFO     TRANSFORMERS_PARAMS=
{'low_cpu_mem_usage': True, 'torch_dtype': torch.float16}

20:36:52-285138 ERROR    Failed to load the model.
Traceback (most recent call last):
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 172, in huggingface_loader
    model = LoaderClass.from_pretrained(path_to_model, **params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 3452, in from_pretrained
    hf_quantizer.validate_environment(
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\quantizers\quantizer_awq.py", line 53, in validate_environment
    raise ImportError("Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)")
ImportError: Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)

System Info

CPU: AMD Ryzen 7 7800X3D
GPU: NVIDIA GeForce RTX 4070 Ti SUPER

XXR · Answer 1 · Sun Oct 13 2024 03:24:07 GMT+0800 (China Standard Time)

Describe the bug

I installed text generation webui and downloaded the model black-forest-labs_FLUX.1-schnell and some more, I have the same bug.

Is there an existing issue for this?

I have searched the existing issues

Reproduction

Select a model and try to load it

22:13:52-565871 ERROR Failed to load the model. OpenTrain-V2.safetensors: 4%|██▏ | 110M/2.43G [00:10<03:40, 11.3MB/s]Traceback (most recent call last): File "D:\Games\text-generation-webui-1.15\modules\ui_model_menu.py", line 232, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Games\text-generation-webui-1.15\modules\models.py", line 93, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Games\text-generation-webui-1.15\modules\models.py", line 155, in huggingface_loader config = AutoConfig.from_pretrained(path_to_model, trust_remote_code=shared.args.trust_remote_code) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Games\text-generation-webui-1.15\installer_files\env\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1038, in from_pretrained raise ValueError( ValueError: Unrecognized model in models\stable-diffusion-v1-5_stable-diffusion-v1-5. Should have a model_type key in its config.json, or contain one of the following strings in its name: albert, align, altclip, audio-spectrogram-transformer, autoformer, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot-small, blip, blip-2, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, conditional_detr, convbert, convnext, convnextv2, cpmant, ctrl, cvt, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deformable_detr, deit, depth_anything, deta, detr, dinat, dinov2, distilbert, donut-swin, dpr, dpt, efficientformer, efficientnet, electra, encodec, encoder-decoder, ernie, ernie_m, esm, falcon, falcon_mamba, fastspeech2_conformer, flaubert, flava, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, git, glpn, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gptj, gptsan-japanese, granite, granitemoe, graphormer, grounding-dino, groupvit, hiera, hubert, ibert, idefics, idefics2, imagegpt, informer, instructblip, instructblipvideo, jamba, jetmoe, jukebox, kosmos-2, layoutlm, layoutlmv2, layoutlmv3, led, levit, lilt, llama, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, mgp-str, mimi, mistral, mixtral, mllama, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, persimmon, phi, phi3, pix2struct, pixtral, plbart, poolformer, pop2piano, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rwkv, sam, seamless_m4t, seamless_m4t_v2, segformer, seggpt, sew, sew-d, siglip, siglip_vision_model, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, table-transformer, tapas, time_series_transformer, timesformer, timm_backbone, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vits, vivit, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xmod, yolos, yoso, zoedepth

System info

CPU: AMD Ryzen 5 7500
GPU: NVIDIA GeForce RTX 4060 Ti

idhem963 · Answer 2 · Wed Oct 23 2024 18:04:18 GMT+0800 (China Standard Time)

for nNote if you installed the same one as me (from this link https://github.com/oobabooga/text-generation-webui)
you will find cmd_yourOS.bat/.sh (most likely cmd_windows.dat) open it and paste (pip install autoawq) hopefully it will fix your problem cuz auto awq is no longer supported and isnt downloaded automatically (plz refer to this sub this is where i found the solution https://www.reddit.com/r/Oobabooga/comments/1g86grc/i_get_an_error_when_i_choose_a_awq_model_need_help/?tl=fr :)

Phil · Answer 3 · Tue Nov 26 2024 13:33:58 GMT+0800 (China Standard Time)

same problem but also with Auto-GTQ. @idhem963 I added the pip install to the script and this doesn't work either, it tells me that it's already installed.